Planning, Monitoring and Evaluating Programmes and Projects

Transcription

1 Planning, Monitoring and Evaluating Programmes and Projects Introduction to Key Concepts, Approaches and Terms Working Draft Version 1 March 2000 Global Monitoring and Evaluation Initiative Jim Woodhill IUCN M&E Facilitator for East and Southern Africa

2 Introduction to PM&E Concepts, Approaches and Terms IUCN The World Conservation Union Founded in 1948, The World Conservation Union brings together States, government agencies and a diverse range of nongovernmental organizations in a unique world partnership: over 900 members in all, spread across some 138 countries. As a Union, IUCN seeks to influence, encourage and assist societies throughout the world to conserve the integrity and diversity of nature and to ensure that any use of natural resources is equitable and ecologically sustainable. The World Conservation Union builds on the strengths of its members, networks and partners to enhance their capacity and to support global alliances to safeguard natural resources at local, regional and global levels. The IUCN Monitoring and Evaluation (M&E) Initiative Through an approach which fosters questioning and reflection and engages stakeholders at the regional and global levels, the IUCN M&E Initiative aims to: develop a common understanding of M&E within IUCN develop a reflective culture within IUCN improve project/programme design and implementation through the use of methods and tools in project, systems and institutional assessments assess the relevance of the Union s work against the broader picture of ecosystem and human wellbeing improve learning processes and reporting of lessons learned put an overall M&E System in place for the Union. Publications from the M&E Initiative are available on-line on the IUCN website Acknowledgements Written by Jim Woodhill IUCN Monitoring and Evaluation Facilitator for East and Southern Africa. jwoodhill@oz .com.au : (2000) IUCN International Union for Conservation of Nature and Natural Resources. Reproduction of this publication for educational and noncommercial purposes is authorised without prior permission from the copyright holder provided the source is fully acknowledged. Reproduction for resale or other commercial purposes is prohibited without the prior written permission of the copyright holder.

3 Introduction to PM&E Concepts, Approaches and Terms Contents 1 INTRODUCTION APPROACHES TO PLANNING MONITORING AND EVALUATION KEY PLANNING, MONITORING AND EVALUATION CONCEPTS A PARTICIPATORY LEARNING APPROACH THE MANAGEMENT CONTEXT PROGRAMME AND PROJECT CYCLE PROGRAMME AND PROJECT LOGIC OBJECTIVE HIERARCHIES AND ASSUMPTIONS ACCOUNTABILITY AND CONTROL KEY ASPECTS OF EVALUATION DEVELOPING AN OVERALL M&E STRATEGY/PLAN DEVELOPING AND MONITORING EVALUATION QUESTIONS AND INDICATORS OPEN ENDED EVALUATION ACTIVITIES LOGICAL FRAMEWORK APPROACH (LFA) AND ZOPP RESULTS BASED MANAGEMENT CIDA RBM TERMINOLOGY USAID ROA TERMINOLOGY... 32

4

5 Introduction to PM&E Concepts, Approach and Terms 1 1 Introduction This document has been developed to assist IUCN staff and partners navigate their way through the terminology that surrounds the practice of planning, monitoring and evaluation (PM&E) in environment and development programmes and projects. It should be considered a working draft and will be revised and improved on the basis of feedback from those who use it. A separate glossary of terms accompanies this document. There is no question that planning, monitoring and evaluation are fields littered with terminology that is often unclear and which is used with different meanings by different groups. This problem can t be solved by this document, however, what it can do is to help explain the different approaches to PM&E and how terms are used by different organisations. Unfortunately confusion around terminology often makes PM&E seem much more complex and difficult than is actually the case. In fact there are really only a handful of key concepts that need to be understood about PM&E. If these concepts are understood then it is easy to make sense of the different terminology and to translate terms between different approaches to PM&E. Monitoring and Evaluation (M&E) is often considered as a separate function and responsibility from planning. However, if a project is poorly planned it is very difficult to monitor and evaluate. Very often M&E staff and specialists find themselves having to go back to basic planning principles before they can assist with M&E. Hence the attention in this document to planning and M&E. Within IUCN it would simplify the difficulties of confusing terminology if a standard set terminology could be used. The Global M&E Initiatives is working towards such a set of concepts and terms. However, the reality is that IUCN will always be working with different donors who demand the use of their particular approach and terminology. This means that it will always be necessary for staff to understand the underlying concepts of PM&E and be able to translate between the approaches and terminology of different donors and partner organisations. The confusion about PM&E can also be reduced by understanding clearly the way concepts and terms are used by different approaches and donors. Consequently the later part of this document explores a number of approaches in some detail.

6 Introduction to PM&E Concepts, Approach and Terms 2 2 Approaches to Planning Monitoring and Evaluation Broadly there are three main approaches to PM&E in use by the major donor agencies: 1. The logical framework approach (LFA) which is the most common and widely used. 2. The German ZOPP, a close derivative of LFA. The acronym stands for the German equivalent of objective oriented project planning. 3. Results Based Management (RBM) or managing for results, which has become the favoured model of the Canadians and Americans in recent years. However, even within each approach there are often differences in the use of terminology and many adaptations have been made as different groups put the approaches into practice. Further, those within agencies who should understand the approach being used are often not as clear in their understanding as would be ideal. This difficulty is compounded when agencies are in a transition from one approach to another. However, while there are certainly differences between the approaches, the underlying principles of PM&E that they are each trying to promote are remarkably similar. In essence, they are: 1. To develop programmes and projects based on a thorough understanding of the situation in which an intervention is planned. 2. To involve stakeholders in a participatory process of programme or project design and evaluation. 3. To develop a set of clear logical objectives that can realistically be achieved within a particular timeframe and within an allocated budget and which will make a significant and sustained contribution to a higher level development objective. 4. To make explicit the cause and effect (means ends) relationships and external factors that underpin the programme or project and which must hold true if planned activities are going to lead to desired results and impacts. 5. To establish a monitoring and evaluation system, including indicators, which will show if the objectives have been achieved and provide information to support effective management and learning.

7 Introduction to PM&E Concepts, Approach and Terms 3 3 Key Planning, Monitoring and Evaluation Concepts 3.1 A Participatory Learning Approach The approach to PM&E adopted by IUCN is one that emphasises the participation of stakeholders in continually learning how to improve performance. Monitoring and evaluation is seen very much as a learning process and not as an external top down policing function. It is recognised that given the complexity of conservation and natural resource management and an extremely rapidly changing wider environment an adaptive approach to programme and project management is essential. PM&E should be seen as a process of helping people to learn how to do things better. Consequently the theory and practice of adult learning is very important to the monitoring and evaluation approach being developed within IUCN. Given the breadth of this field in itself, the glossary will not attempt to cover it in any detail. However, when using the glossary it will help to keep in mind the idea of a participatory learning approach to monitoring and evaluation. A participatory learning approach also means that there is much more to M&E than just identifying and monitoring quantitative indicators. Learning implies understanding, analysis, questioning, being critical and trying to explain why things have worked or failed. Certainly quantitative indicators are important and can be helpful but very often they provide only a small part of the information needed for learning. Also for higher level objectives or goals, such as improving protected area management or community wellbeing it is just simply not possible to develop simple quantitative indicators that have any real meaning. Unfortunately there is a widespread view that developing an M&E plan for a programme or project is primarily about developing a set of such quantitative indicators. This document explicitly challenges this perception. For example, well-facilitated review meetings with staff, or the use of qualitative inquiry methods with beneficiaries, will often provide much more valuable information for learning.

8 Introduction to PM&E Concepts, Approach and Terms The Management Context PM&E are essential functions of good management and should enable a programme or project to achieve a high level of performance as illustrated in the box below. PM&E - Critical Tools for Management PM&E (learning) should support Management resulting in Performance It is important to see monitoring and evaluation as tools to be integrated into all aspects of programme and project management, as illustrated in the next box. The starting point is to ask; what information is required for effective management and what sort of M&E system is required to provide it? Management Functions and M&E Planning Controlling Monitoring Functions & Evaluation Organising Leading Staffing

9 Introduction to PM&E Concepts, Approach and Terms 5 Unfortunately M&E is often erroneously viewed as an annoying task of simply providing donors with the information they require. Certainly accountability to funding bodies is one function of an effective M&E system but it is certainly not the only or the most important function. A list of purposes for M&E is given below. Purposes of Monitoring and Evaluation Ensuring planned results are achieved Improving and support management Generating shared understanding Generating new knowledge and support learning Building the capacity of those involved Motivating stakeholders Ensuring accountability Fostering public and political support

10 Introduction to PM&E Concepts, Approach and Terms Programme and Project Cycle The diagram below illustrates a generic programme or project cycle. It emphasises the importance of starting with detailed scoping, situation analysis and design stages. It also illustrates the importance of considering M&E at all stages in the cycle. Importantly it also illustrates the need for constant cycles of planning, acting, monitoring and evaluation (in other words learning) during implementation. The Programme/Project Cycle Gov/Donor Goals/Policies Organisation Mission Beneficiary Needs Scoping Final Evaluation Act Formulation / Design M&E Strategy Implementation Monitoring and Evaluation Plan Evaluate Monitor Financing and contracting Mobilisation and Implementation Planning M&E Plan 3.4 Programme and Project Logic There is an overall logic to any programme or project as illustrated below. This logic can be described as follows. First, there is a situation that a group of stakeholders wish to improve the reasons for a programme or project. This situation is defined, to a significant extent, by the problems and visions of the stakeholder groups. There may or may not be common perceptions of what the problems are or what would actually constitute an improvement, which is why participatory approaches to planning are so important. Developing a detailed and holistic situation analysis is a critical aspect of programme or project planning. The understanding of the situation will lead to a programme or project plan. This will usually include: The goal a summary of what in the long term the programme or project is contributing towards related to impact.

11 Introduction to PM&E Concepts, Approach and Terms 7 The Purpose a summary statement of overall what the programme or project should achieve given its timeframe and resources the overall outcome. A set of Results the main things that must be achieved for the programme or project to realise its purpose, there may be several levels of results. Results are also referred to as outputs and outcomes. A set of Activities what must actually be done for the results to be realised. A set of Inputs the resources required for the activities to be undertaken. Project/Programme Make Up and Logic Reasons for Programme Project The Situation to Improve Problems and Visions IMPACT Programme Project and Plan (Goal, Purpose, Results and Activities) Implementatio Process Inputs Activities Actual Results The programme or project is then implemented according to this plan involving a process of inputs being used to undertake activities that lead to actual results. Of course few projects go exactly as planned and there will need to be constant cycles of planning, acting, monitoring, evaluating, replanning and so on. In some cases it may be necessary completely revise the original plan. The actual results should lead to a set of impacts that will improve the original situation. In most projects there will also be unanticipated impacts that may be positive or negative, which are also important to track. Understanding this basic logic of a programme or project is the starting point for understanding PM&E.

12 Introduction to PM&E Concepts, Approach and Terms Objective Hierarchies and Assumptions Any programme or project has an objective hierarchy, as shown below. This maps out the way low level tasks or activities contribute to higher level objectives and how meeting these objectives leads to the achievement of the overall purpose or goal of a project or programme. Such a hierarchy of objectives is also referred to as the intervention logic or the narrative summary of a programme or project. It shows the cause and effect, or means ends relationships of an intervention. Developing a clear logical objective hierarchy is fundamental to good programme and project design and essential for M&E. It is called an objective hierarchy because any level within it can be seen as being an objective. Higher level objectives are, or should be, a consequence of achieving lower level objectives. Alternatively it could be considered a results hierarchy. Although as will be shown later a distinction is sometime made between an objectives hierarchy being what is planned to be achieved and a results hierarchy being what is actually achieved. Objective Hierarchy Goal Purpose Assumptions (hypotheses) Key Result Key Result Key Result Sub Result Sub Result Sub Result Sub Result Sub Result Sub Result Sub Result Sub Result An objective hierarchy can be thought of as like the structure of a tree, with the leaves or twigs being the detailed activities or tasks (low level objectives) and the trunk being the overall goal (highest level objective) and the branches being different intermediate level objectives. In theory there can be many levels to an objective hierarchy, larger and more complex programmes or projects require more levels than do small simple projects. In practice, most planning approaches find four or five levels adequate. For a programme one might imagine the trunk and main branches as being the programme objectives and the sub-branches twigs and leaves as being the projects that contribute to the programme objectives. In a very large project, it may be helpful to think of having sub-projects. For a complex programme

13 Introduction to PM&E Concepts, Approach and Terms 9 with large projects, it is conceivable that it may be necessary to have a programme, project and sub-project level of planning and each of these levels having six levels of an objective hierarchy. In the end, planning needs to be taken down to sufficient detail to enable day to day workplans for activities and tasks. A well-developed objective hierarchy makes it clear what must be done to achieve results and in reverse, along with indicators, what results are achieved from completing activities. An objective hierarchy also has a sideways logic. The outputs, products or results from one strand of the hierarchy will often be critical inputs into another strand. The conventional representation of an objective hierarchy (as shown above) does not illustrate this sideways flow leading to the criticism that such an approach to design is too linear and restrictive. However it is equally possible to map out a project from a systems perspective showing a series of interconnecting systems that have inputs and outputs. It is important that the linear criticism is not used as an excuse for poorly developed intervention logic. Certainly, for complex programmes it will not always be possible to arrive at a simple hierarchical logic, such as the one illustrate above, that adequately expresses all the dimensions that need to be communicated. There may need to be a number of parallel logics or a matrix structure to the programme framework. A detailed discussion of this is beyond the scope of this document. To assist in the planning process different levels within an objective hierarchy are given different terms (goal, purpose, outcomes, outputs, results, specific objectives, activities, etc). It is the lack of consistency in the way terms are used for the different levels in an objective hierarchy that creates much of the PM&E confusion. However, as long as the concept of different levels in a hierarchy is understood and the meaning of different terms is understood by the stakeholders within a particular context, it doesn t really matter what they are called. Some examples of different objective hierarchies that have a different number of levels and which use different terminology are given on the next page.

14 Introduction to PM&E Concepts, Approach and Terms 10 Examples of Different Objective Hierarchies Vision Goal Purpose Key Results Sub Results Activities Goal Purpose Outputs Activities Goal Purpose Outcomes Outputs Activities Goals Mission Strategic Objectives Key Result Areas Results Vision Goal(s) Objectives Activities Activities Assumptions An objective hierarchy should reflect cause and effect relationships between lower and higher level objectives. The lower level activities/objectives are the means for achieving the ends of the higher level objectives. There will always be assumptions about the cause - effect (means ends) relationships in an objective hierarchy and these assumptions should be made explicit in the design of a programme or project. There are two types of assumptions, which are often confused in the logical framework approach to planning. The first type of assumption is that related to the internal logic of the programme or project. For example a project design might be based on an assumption that by reducing poaching in an area an endangered species will be preserved. This may be a correct assumption or it may be incorrect because the main threat to the species might in fact be reduced habitat or disease. The second type of assumption relates to the external factors or external environment that must exist for a project to succeed. For example, for a tree planting project it may be assumed that rainfall will not be significantly below average. For any project it will be assumed that there will be sufficient political stability for the project to operate effectively. For either type of assumption it is often helpful to identify, what are referred to as, killer assumptions. These are the assumptions that if they are wrong the project will fail completely.

15 Introduction to PM&E Concepts, Approach and Terms 11 Good project design looks very carefully at assumptions and makes them explicit. Likewise an important part of M&E is checking on the validity of original assumptions. Sorting Out Objective Hierarchy Terminology Objective hierarchy terminology, as already mentioned, is the source of most of the confusion around PM&E. To help sort this out it will be helpful to discuss some basic ideas about different levels in an objective hierarchy before getting too caught up in the terminology. At the highest level of an objective hierarchy it is helpful to place the programme or project within the context of some larger human endeavour. For example an integrated conservation and development project may be contributing towards the protection of the ecological, economic and cultural values of a particular forest. However the project itself will not be able to fully realise this highest level objective or goal, it can only make a contribution. This level provides clarity about why a programme or project is being undertaken. It helps to provide a sense of vision about the future for those engaged with a programme or project. This level is commonly understood as the goal for a programme or project. Some planning frameworks call this level the vision and reserve the goal for the next level down. Impact is generally used to refer to the extent to which a programme or project in fact does make a contribution towards the goal. However this may not occur during the life of a programme or project and where others are also contributing it may be difficult to desegregate the contributions made by one initiative from those of another. This makes impact evaluation particularly difficult, but nevertheless important. Below this top level of an objective hierarchy is what can be considered the purpose of a programme or project. This is overall what a programme or project should achieve if it is successful. It is generally considered helpful to try and summarise the purpose as a single statement to ensure focus and clarity. For example to develop the institutional frameworks and human and organisational capacity for sustainable forest management. This is what a programme or project should be able to achieve within its available resources and implementation period. If planning for an organisation this level will usually be called the mission of the organisation. Below this level are a series of major outcomes or results that need to be achieved for the purpose to be realised. These should be the actual observable changes in for example behaviour, institutions, economic circumstances or physical conditions. Again these should be achievable within the resources and timeframe of the programme or project. For example, staff within the forest department effectively carrying out their responsibilities. In the LFA this level is subsumed into the purpose level

16 Introduction to PM&E Concepts, Approach and Terms 12 and it is argued that there are outcomes at the purpose level. Other planning frameworks refer to this level as the main objectives. Each of these major outcomes or results arise from the logical consequence of the programme or project delivering a set of products or services outputs which arise directly from programme or project activities. For example forestry department staff trained in the skills required to carry out their responsibilities. Not all planning frameworks make this distinction between outcomes and outputs. It is this middle level of an objective hierarchy where terms and concepts are most problematic. While outcomes generally refer to the higher order changes or effects that arise from delivering outputs, which are generally considered as tangible or concrete products and services, the distinction is not always so clear cut. There is not a neat dividing line between what can be considered outputs and are outcomes. In fact rather than there being just two cause and effect steps there may be many, ie there are also higher and lower level outputs and outcomes. While a slightly artificial, the output and outcome distinction is still helpful in summarising what can be expected to result at different levels within an objective hierarchy. However, it needs to be remembered that it is a shorthand summary and like all categorisations an approximation of reality. This ambiguity around outputs and outcomes is why IUCN finds it less confusing to talk of key-results and sub-results within the objective hierarchy. The relationship between these terms is illustrated below. Objective and Results Hierarchies Objective Hierarchy - what is planned to be achieved This is where M&E terminology can becomes very confusing Goal Purpose Results Hierarchy - what is actually achieved (also called impact or hierarchy and results Impact Outcomes Actual Overall Result Outcomes Outcomes (Planned) Key Results Outcomes Actual Key Results Outcomes Outputs (Planned) Sub Results Outputs Actual Sub Results Outputs Activities Inputs

17 Introduction to PM&E Concepts, Approach and Terms 13 The lowest level of an objective hierarchy is the activity level. For detailed work planning it will often be necessary to break activities down the subactivities and/or tasks that will occur, for example over a particular year or quarter. From an M&E perspective it is very helpful to think in terms of impacts, outcomes and outputs. However it is not necessary to limit impact questions to the goal level of the objective hierarchy or outcome questions to the keyresult/outcome level of the hierarchy. This is easiest explained with an example. Take the sub-result of a large integrated conservation and development project - sustainable livelihood activities adopted in target villages. An activity of this sub result may the promotion of bee keeping. The real reasons for this may be to try and reduce the negative effects of wild honey collection on a forest. The impact of such an activity would relate to the extent to which a reduction of wild honey collection leads to an improvement in the ecological values of the forest. The outcomes may include the reduced level of actual wild honey collection, the level of success adoption of bee keeping and extent to which bee keeping increases household/village income. The outputs would be the number of farmers trained in bee keeping, the formation of a bee keeping group, the provision of hives or the development of a honey processing facility. In other words even down to the activity level of an objective hierarchy it is possible to examine impacts, outcomes and outputs. In development work objective hierarchy thinking has been very much influenced by the logical framework approach which has emphasised the output level of planning and subsumed the outcome level into project purpose. The idea was to focus on the tangible products and services that a particular project management team should be directly responsible for delivering. This has problems that manifest themselves in two ways. One, the higher level results or outcomes that a project needs to achieve are not made explicit and project management becomes very activity/output driven and hence may not adapt its implementation strategy to achieve higher level results when circumstances change. Two, in trying to overcome this problem and attempting to focus on higher level achievements outputs come to be used as essentially equivalent to outcomes. These issues relate to the questions of accountability and control that will be discussed in the next section. The following table summarises the key terms discussed above.

18 Introduction to PM&E Concepts, Approach and Terms 14 Term Goal Impact Purpose Outcome Output Result Objective Activity Sub Activity / Tasks Definition The longer term, high level improved situation that a programme or projects is contributing towards. The goal helps explain why a programme or projects is being undertaken. Generally the goal can only be achieved through the combined efforts of others and a programme or project cannot be solely responsible for the goal being realised. Sometimes the term goal is used to refer to the highest level direct results of a programme or project (ie it is used interchangeably with what in this table is defined as the purpose). Vision is sometimes used in place off goal. Example The ecological, economic and cultural values of XXX forest protected for current and future generations. The extent to which a programme or project, or some part of it, actually makes a contribution towards the goal. Impact is concerned with intended and unintended and positive and negative contributions. Overall what a programme or project, within the timeframe and resources available, should achieve. Example XXX forest being used and managed in a sustainable way. The observable changes in, for example, behaviour, institutions, economic circumstances or physical conditions that need to result from a programme or project in order for it to realise its purpose and make a contribution to the goal. Result or key result is also used interchangeably with outcome. Example Forestry officers carrying out their responsibilities competently and effectively. The direct services or products that must be delivered for the outcomes to realised. Result or sub result is also used interchangeably with output. Example Forestry officers trained to develop collaborative management agreements with local communities. Used to refer generally to both outputs and outcomes. There can be lower (sub) and higher (key) level results. Example as for outcome and output. Used generally to refer to anything that should be achieved by a programme or project. Example as for purpose, outcome output depending on use. Specific actions that need to be undertaken for outputs to be produced or outcomes/results/objectives to be realised Example Training workshop on collaborative management conducted. A detailed breakdown of activities to the level required for yearly/quarterly/weekly/daily workplanning.

19 Introduction to PM&E Concepts, Approach and Terms Accountability and Control As you move up an objective hierarchy it becomes increasingly difficult to directly control external factors that influence the achievement of results or the goal of a programme or project. For example, a goal of a project might be to improve the capacity of a government department responsible for conservation. Part of this project might involve training. The project can directly control the hiring of a training venue, the preparation of materials, the provision of a trainer, and the notification of potential participants. It has less control over whether potential participants will actually attend and considerably less control, if any, over whether the skills the participants learn will actually be used back in the organisational setting. The Limits of Control and Accountability Ends Beyond project control Goal Purpose what the project is contributing towards Within project control Means Key Results (Outcomes) Sub Results (Outputs) Activities Tasks what is within the direct management control of a project what overall the project can reasonably be accountable for achieving Adapted from Materials Developed by ITAD The issue of accountability and control relates very closely to the notion of assumptions. For example assumptions will have been made that the participants would use there skills back in the organisational setting otherwise there would be not rationale for the project (or at least that particular project activity). There are two dimensions of accountability and control that are very important to distinguish. The first relates to what a programme or project management team should be accountable for if they have been given a plan to implement. In this case they can really only be held accountable for carrying out the activities and for the products or services that flow directly from these activities - ie the outputs. The second dimension relates to the overall accountability of a programme or project, which rests with those who design, fund or have overall management responsibility. At this level there should be accountability for the higher level results (outcomes) and the purpose, even though the achievement of these is dependent on actions by others over which there may be no direction control by the programme or

20 Introduction to PM&E Concepts, Approach and Terms 16 project. However, the programme or project has been established on assumptions that these actions by others will occur and the programme and should be held accountable for the assumptions within the design. Traditional logical framework approaches tend to emphasise the former and suggest that the purpose of a project is outside the control of a management team. Results based management approaches tend to emphasise managing to achieve the higher level objectives (results). 3.7 Key Aspects of Evaluation In developing any monitoring and evaluation system there are five aspects of evaluation to consider as illustrated below. If you can provide information on each of these you will be able to judge the overall performance of a programme or project. 1. Relevance Key Aspects of Evaluation The Situation to Improve Problems and Visions Plan (Goal, Purpose, Results and Activities) 5. Sustainability 2. Effectiveness 4. IMPACT Inputs Activities Actual Results 3. Efficiency Relevance - Was/is the programme or project a good idea given the situation to improve? Was the logic of the intervention logic correct? Why or Why Not? Effectiveness - Have the planned results been achieved? Why or Why Not Efficiency - Have resources been used in the best possible way? Why or Why Not? Impact - To what extent has the programme or project contributed towards its longer term goals? Why or Why Not? Have there been any unanticipated positive or negative consequences of the project? Why did they arise? Sustainability - Will there be continued positive impacts as a result of the programme or project once it has finished? Why or Why Not?

21 Introduction to PM&E Concepts, Approach and Terms Developing an Overall M&E Strategy/Plan To effectively monitor and evaluate any programme or project it is necessary to develop an overall M&E strategy or plan. A common failing for many projects is that the only reference to M&E is the list of indicators and monitoring mechanisms in the logical framework matrix table. This just does not provide enough information to guide the actual implementation of a M&E system. The boxes below illustrate the process for developing and the general content for an M&E plan. Steps for Developing an M&E Plan 1. Establish use and scope of M&E system 2. Check project objectives and logic 3. Establish overall evaluation requirements and questions 4. Establish requirements for regular monitoring of implementation and progress towards desired results 5. Test overall M&E strategy with potential users and refine 3 and 4 5. Establish the information and indicators needed for 3 and 4 6. Develop and test regular data gathering / monitoring mechanisms 7. Design open-ended and/pr periodic evaluation activities 8. Design information management system 9. Design a learning and feedback process 10. Decide how to evaluate the evaluation Contents for an Overall Project M&E Plan Purpose and scope Overview of approach (concepts, terminology, methods) General project evaluation activities - eg... Annual internal reviews external reviews M&E details Goal level (impact) Purpose level Results level Appendices - eg... Budget - key evaluation questions - focussing questions for learning lessons - indicators and monitoring mechanisms - open-ended evaluation activities - participation and responsibilities Details on indicators, monitoring mechanism, reporting Gnat chart of key M&E activities over project life

22 Introduction to PM&E Concepts, Approach and Terms 18 A few points are worth emphasising: 1. It is important to be clear about the overall purpose and scope of the M&E system. In particular it should be clear who needs what sort of information for what reasons, how extensive or minimal M&E needs to be, and what resources are available. For example a project that has a learning or action research focus will require a more comprehensive M&E system than a project that is simply implementing a physical works programme. 2. The overall system that will be needed for M&E to be effective must be considered. This includes designing evaluation questions and indicators that are relevant and practical, training staff in monitoring techniques, developing monitoring forms and reporting processes, establishing an information management system and establishing how information will be analysed, reported and used. 3. It is particularly critical to design learning processes in which staff, beneficiaries, partners and donors participate. For example, annual review meeting. Information from the M&E system should stimulate, inform and support this learning process. 4. The whole M&E system must be developed around the use of information. If information can't be used it s collection is a waste of time. Yet, it is remarkable how much fragmented and often unusable data is collected by projects in the name of carrying out M&E. 5. It is important to recognise the difference between regular monitoring of progress vs periodic and in-depth evaluation of some part of or the entire programme or project. Regular progress monitoring will generally focus more on output level indicators and the achievement of established milestones or targets. Periodic in-depth evaluation examines whether outputs are leading to expected outcomes and impacts, explores reasons why and should assesses the effectiveness of the process of the programme or project. 6. Responsibilities for M&E must be very clear and explicit in any terms of reference, in job descriptions and be a core part of any staff performance monitoring and appraisal system. If staff are asked to undertake M&E work but it is not formalised as part of their core responsibilities it will inevitably slip to the bottom of the work pile and never get done. 7. Make the M&E plan visual so everyone is aware about it and of their responsibilities.

23 Introduction to PM&E Concepts, Approach and Terms 19 Visualising an M&E Plan Develop M&E plan with stakeholders Preparation for mid term Mid term review (Illustrative Only) Phase two preparation Q1 Year 1 Q2 Q3 Q4 Q1 Year 2 Q2 Q3 Q4 Q1 Year 3 Q2 Q3 Q4 Training in use of reporting system PRA with participating communities Annual Review and Planning workshop Preparation for annual review (performance and lessons learnt) Annual Review and Planning workshop Key Meetings Report Due 3.9 Developing and Monitoring Evaluation Questions and Indicators A good M&E plan should clearly articulate the key evaluation questions that need to be asked for each level of the objective hierarchy. To answer these evaluation questions it will be necessary to identify information needs. The necessary information may come from specific quantitative or qualitative indicators, general project records, generally available information or from specially designed evaluative or action research activities. Traditionally a lot of emphasis has been placed on the development of quantitative indicators as the key element in developing an M&E plan. Starting at this point tends to narrow down and straitjacket an M&E system and reduce its usefulness particularly in relation to supporting learning. For good reason it is often very difficult or even impossible to develop sensible quantitative indicators for the goal purpose and outcome levels of a programme or project. Very often when quantitative indicators have been developed for these levels they are either impractical to monitor or provide relatively useless information in terms of overall evaluation of the result. There is no question that indicators and in particular quantitative indicators are an important part of an M&E system and wherever practical they should be used. However, an M&E system will be far more useful if it is designed around the broad evaluation questions rather than narrowly focused indicators. In thinking about evaluation questions and indicators it is important to make the distinction between evaluation and monitoring. It will often be necessary and helpful to have some simple indicators that show regular progress towards a result and which are monitored regularly. Output indicators are

24 Introduction to PM&E Concepts, Approach and Terms 20 particularly useful in this regard. This is the ongoing monitoring of progress that is required to manage a programme or project and which should show early warning signs of problems. Evaluation is a more in depth and probing assessment of the whole situation that should explore the reasons for success or failure. This generally occurs less frequently. Different types of information and indicators may be required for regular monitoring vs in depth evaluation. These two different but related aspects should be reflected in the description of an M&E plan for a particular result. To effectively monitor and evaluation progress towards any particular result (objective) in a programme or project the following steps will generally be appropriate: 1. Identify the key evaluation questions for each level and result in the objective hierarchy. 2. For each question identify what information or indicators will be required to answer the question. 3. For each piece of required information or indicator establish: The methods and frequency for gathering the information or monitoring the indicator. The baseline information required for comparison. What preparation and resources are required for the data to be collected, collated and analysed, for example data collection and analysis forms, training of staff, data base design, external expertise. Who is responsible for carrying out each of the above and by when. 4. For each question, or a set of questions, establish what overall analysis is required and how the resulting knowledge will be used and what change processes need to be in place to learn from and respond to the knowledge. 5. Decide on an overall monitoring and evaluation plan for the particular result. For example, how often will an overall evaluation of progress be made and what indicators or information will be used to regularly monitor progress and how often?

25 Introduction to PM&E Concepts, Approach and Terms 21 Detailed Result M&E Plan Evaluation Questions Required Information and Indicators Data Gathering Methods, Frequency and Responsibilities Baseline Information Requirements Status and Responsibilities Required Forms, Planning, Training, Data Management, Expertise, Resources and Responsibilities Analysis, Reporting, Feedback and Change Processes and Responsibilities 3.10 Open Ended Evaluation Activities A good evaluation system should give adequate attention to what shall be termed here open ended evaluation activities. These are all the aspects of evaluation that complement an indicator based approach. Such open ended activities (examples of which are given below) are necessary for the following reasons: 1. There will often be unintended positive or negative results and impacts from a project that will be missed by an evaluation that just focuses on monitoring predetermined indicators. 2. Monitoring indicators alone often not provide an understanding of why objectives have or have not been met. This requires discussion and analysis with project staff and partners. 3. Monitoring indicators alone will not lead to understanding and learning by programme or project staff and partners. 4. For complex or messy objectives it may not be possible to develop a easily measurable indicator and the achievement of the objective may have to be demonstrated through more anecdotal information. 5. Monitoring indicators provide only limited capacity for evaluation of the success or otherwise of the process of the project.

26 Introduction to PM&E Concepts, Approach and Terms 22 Examples of Open-Ended Evaluation Activities Annual Review and Planning Processes Monthly and/or Quarterly Review and Planning Processes Open Ended Impact Assessment PRA s External Reviews Peer Reviews Stakeholder Meetings Regular Staff Meetings Analysing and Documenting Lessons Learnt Conference Presentations and Papers Advisory Committee Functions Independent Assessments Staff Performance Reviews Implementing Partner Performance Reviews

27 Introduction to PM&E Concepts, Approach and Terms 23 4 Logical Framework Approach (LFA) and ZOPP The logical framework approach (LFA) and ZOPP have evolved from the 1960 s as methodologies for improving the systematic planning of development projects. Over time, they have evolved from simply a framework for structuring project objectives to more sophisticated, process orientated, approaches for involving stakeholders in project design and management. LFA and ZOPP are based around the following programme/project development steps: 1) Systemic and participatory analysis of the situation in which some intervention is anticipated 2) Clearly identifying the problem(s) to be addressed and identifying the causes and effects of the problem(s). This is usually done by developing a problem tree. 3) Using the situation analysis and problem identification steps to consider intervention alternatives and to develop a logical hierarchy of activities and objectives that will enable the problems to be overcome. 4) Identifying the assumptions that underlie the logic of the objective hierarchy ie being explicit about why it is assumed that particular lower lever activities or objectives will lead to higher level ones. Associated with this is identifying the external risks that may lead to these assumptions not being realised and hence the project not succeeding. 5) Establishing the indicators that will be used to verify that project objectives have been achieved. 6) Developing the means by which information for the indicators will be collected and analysed. Various groups and facilitators have integrated an extensive range of participatory planning methodologies and tools with the basic LFA/ZOPP framework and quite sophisticated planning workshops have been developed. There are numerous LFA/ZOPP manuals and documents. The objective hierarchy for LFA and ZOPP usually has the following levels and terms: Goal the long term objective, change of state, or improved situation towards which the programme or project is making a contribution.

28 Introduction to PM&E Concepts, Approach and Terms 24 Purpose the immediate project objective ie the observable changes in performance, behaviour or status of resources that should occur as a direct result of the programme or project. Results (Outputs) the products, services or results that must be delivered by the project for the purpose to be achieved. Activities the specific tasks that must be undertaken for the results to be achieved The outcomes from such a planning processes are summarised in a project planning matrix (PPM) or logframe table as illustrated below. It is important to distinguish between the logical framework approach and the project planning matrix. Often poorly planned projects, that in fact do not reflect an LFA approach, are summarised in such a matrix. Project Planning Matrix (PPM) Objective Hierarchy Indicators Monitoring Mechanisms Assumptions and Risks Goal Purpose Results (Outputs) Activities Different terminology is used by different donors and other groups for both the logframe objective hierarchy and the headings for the columns in the project planning matrix. The main terminology used by the key donors is summarised below. It s also worth remembering that the staff of development agencies aren t always themselves familiar with the correct definitions of some of the terms they are using. Different parts of the same organisation may be using the same terms in different ways. Sometimes, the adoption of new terminology within these organisations takes some time to reach all of the employees. Outputs is the most commonly used term for the level between activities and purpose, however the term results is now becoming more widely used, partly reflecting the move towards results based management approaches an partly because there is some confusion within the M&E terminology about the meaning of outputs. IUCN has decided to use the term result rather than

29 Introduction to PM&E Concepts, Approach and Terms 25 output. The project planning matrix is usually only shown with one level of results (outputs) however it is understood that there can be several levels of results (ie key results and sub results) for a large and complex programme or project. Conventionally it has been understood that the inputs, activities and results are within the direct control of a project, while the purpose and goal is beyond direct project control. However this is a blurred rather than clear line and depends on whether one is concerned with overall project accountability (design, funding, eventual impact) or the just the accountability for project implementation. With complex projects that need to be adaptively managed even this distinction becomes blurred. LFA and ZOPP have become widely accepted as useful and necessary tools for project planning, however, they do have their weaknesses that include: focussing too much on problems rather than opportunities and vision; being used too rigidly and leading people into a blueprint approach to project design and implementation; limited attention to problems of uncertainty where a learning and an adaptive approach to project design and management is required; the tendency for poorly thought through sets of activities and objectives to be entered into a PPM table giving the appearance of a logical framework, when in fact the key elements of the analytical process have been skipped; the simple logic of the LFA is often not appropriate to programme level planning where it may be necessary to deal with a number of parallel or cross cutting logics. Despite these limitations and provided due attention is given to the participation of stakeholders, and it is not used to rigidly the LFA/ZOPP approach remains a very valuable tool for project planning and management.