Monitoring Toolkit. Gardens

Transcription

1 EMERGENCY OPERATIONS AND REHABILITATION DIVISION (TCE) Monitoring Toolkit Gardens including Community, School, Backyard and Institutional garden projects September, 2008 by Andy Lyons, RIACSO Stanley Dlamini, Swaziland ECRU REGIONAL INTERAGENCY COORDINATION SUPPORT OFFICE (RIACSO) JOHANNESBURG, SOUTH AFRICA

2 THE FAO PROJECT MONITORING TOOLKIT SERIES This monitoring toolkit is one of a series that has been developed by the FAO Regional Interagency Coordination Support Office (RIACSO) in Other Monitoring Toolkits Developed by FAO/TCEO Johannesburg, South Africa. The goal of these Conservation Agriculture toolkits is to strengthen and harmonize monitoring Input Trade Fairs and evaluation of supported emergency activities Small Stock throughout the southern Africa region. To achieve Transboundary Animal Diseases this goal, these toolkits provide a core set of Small Scale Irrigation indicators and data collection templates that enable common data collection and the aggregation of results across multiple project sites within countries and the region. The audience for this monitoring toolkit is anyone who is involved in implementing garden projects. This includes FAO staff at country and regional levels, as well as implementing partners at specific project sites, government extension officers, NGO staff, and consultants. This is the second version of the gardens monitoring toolkit. The first version was developed in October This version reflects additional feedback from regional reviews and a season of pilot testing in Swaziland. The main changes in this version are the inclusion of supplemental indicators and data collection templates for specific types of garden projects, and some rearranging of the sections. These toolkits are a work in progress. To be successful they must address regional cross-cutting information needs on project effectiveness, while also accommodating a diverse set of working conditions, operational resources, and information requirements of numerous national and local users. This is not an easy task, hence feedback from users is essential. Please channel all feedback through your local FAO Emergency Coordination Unit (ECU). Comments should be sent to the Regional Information Officer at RIACSO. How to use this Toolkit 1. Review the core output and outcome indicators (page 3) 2. Review the sample data collection forms (page 23) 3. Meet with project staff and implementing partners and reach consensus on: a. Indicators and data collection instruments (Annex II, page 23) b. A workplan for data collection, processing, and analysis (page 15) 4. Implement the workplan. 5. Review and present results back to stakeholders 6. Revise workplan as needed. Development of this toolkit has been made possible by the generous support of The Government of the Republic of South Africa under OSRO/RAF/510/SAF. ii

3 TABLE OF CONTENTS An Introduction to Garden Projects...1 Monitoring Garden Projects...1 Categorizing Garden Projects...2 Gardening Core Indicators...3 Data Collection Workplan...6 Pre-activity planning...8 Selection of beneficiaries...8 Baseline household survey...8 Training...10 Planting, production, harvesting, and sales...10 End-of-season review...12 Terminal evaluation...12 Post-project evaluation...15 Making a M&E Workplan...15 References...17 Annex I. FAO s Regional Monitoring Strategy for southern Africa...19 Basic vs. Preferred Standards for Monitoring...20 Measuring Targeting, Household Vulnerability and HIV/AIDS Impact...21 Annex II. Sample Data Collection Forms...23 Adapting the forms...23 Garden Beneficiary Selection Worksheet...24 Garden Diary Section 1: Planting & Harvesting...25 Garden Diary Section 2: Profit and Loss Statement...26 Garden Diary Section 3: Rainfall (Format 1)...27 Garden Diary Section 3: Rainfall (Format 2)...28 Garden Diary Section 4: Tracking Market Prices...29 Production and Sales Summary...30 Instructor's Training Log...31 Garden Seasonal Summary...32 (for all gardens)...32 Household Garden Survey...35 School Garden Survey...39 Institutional Garden Survey...41 TABLE OF TABLES Table 1. Classification of gardens...2 Table 2. Garden core indicators...4 Table 3. Data collection tasks for each phase of implementation...7 iii

4 iv

5 AN INTRODUCTION TO GARDEN PROJECTS Gardens have been one of the mainstays of food security projects for many years, and for good reason. Gardening is a core livelihood strategy for almost all social strata in nearly all parts of Africa, so it is relatively easy to get participation for garden projects in both rural and urban settings. Gardening is also an extremely versatile development activity; projects can be designed to achieve a variety of goals from better nutrition, increased incomes, HIV/AIDS mitigation, to market development. From an implementation standpoint, the logistics of garden projects are somewhat easier than field crops because they tend to be located near settlements and often at a central location such as a school or water point. The relatively low requirements for land and inputs also make gardening a viable livelihood strategy for the poorest households, making these projects attractive to government and donors who want to help this hard-to-reach population. Successful garden projects must be tailored to fit the local agronomic conditions, dietary preferences, cultural traditions, and market context (FAO 2001). Experience has also taught us that traditional garden projects create the greatest impact when combined with one or more complementary activities, for example small livestock production, savings schemes, health and nutrition education, market development, or small-scale irrigation. For additional information on garden projects, see the References section (page 17). What is a garden? Gardens are not easily defined because there is as much variation in the characteristics of gardens as there is in the diverse contexts in which they re found. Even among experts there is no consensus on where the line falls between gardens and field cultivation. Most gardens are smaller and cultivated more intensely than field crops, and have a more diverse crop mixture. Gardens are also more likely to be irrigated, fertilized, fenced, closer to homes, and cultivated year round. However there isn t a single universal standard that can be applied across the board, as exceptions can be found to all of the above criteria 1. Fortunately, for the purposes of monitoring the term garden is just a label or category for a project, and we don t need to worry too much about trying to precisely measure it. The local understanding of a garden should be followed rather than trying to adopt an external or complicated set of criteria. The bottom line: if the project document says it s a garden, then it s a garden. MONITORING GARDEN PROJECTS To strengthen the monitoring of garden projects, we need to: 1. Develop a system for classifying the different types of garden projects 2. Define core indicators 3. Develop criteria for identifying food insecure households and households impacted by AIDS 4. Design a plan for the logistics or nuts and bolts of data collection and analysis 1 see also FAO (2001): Information Sheet 1: Definition and Concept of Home Gardens in Africa [ 1

6 Categorizing Garden Projects From a monitoring standpoint, gardens are not the easiest to monitor due to the great diversity in contexts and project designs. Among the garden projects supported by FAO in southern Africa, there are nutrition gardens as part of home-based care programmes, school gardens, community gardens, drip kit gardens, seed production gardens, hospital gardens, urban gardens, and gardens at Junior Farmer Field and Life Schools (JFFLS). While all of these garden projects share some common characteristics, such as vegetable production, most also have specific objectives or emphases that are tailored to the local context (Figure 1). The general and specific goals of a garden project should be defined in the project document. For example a community garden project might have a heavy emphasis on demonstrating the benefits of a new technology such as treadle pumps, whereas a backyard garden project might place a strong emphasis on improving household nutrition levels. Figure 1. Cross-cutting and project-specific objectives of garden projects school gardens household gardens cross-cutting objectives community gardens institutional gardens project specific objectives We need a way to categorize garden projects so that monitoring tools and protocols can be tailored appropriately, and results disaggregated when needed. For the purposes of this toolkit, gardens will be classified following the scheme used by Machakaire and Hobane (2005), shown in Table 1. Table 1. Classification of gardens Type of Garden Characteristics community garden a large joint garden where multiple individuals keep separate plots but share equipment, infrastructure, training, and maintenance responsibilities. communal garden a garden managed by a group of people who share both responsibilities and production, either equally or according to some formula agreed upon by the group household garden a garden maintained by a single household, also known as a backyard or kitchen garden institutional garden a garden managed primarily for nutrition or income by an institution, such as a hospital, neighborhood care point, orphanage, church, or prison school garden a garden based at a school often with educational objectives in addition to nutrition In addition to defining categories for the different types of gardens, to design data collection we also need to think about the structure of garden projects. This will help us define indicators and decide what needs to be measured and how. Figure 2 below illustrates a generic hierarchical structure that suits many garden projects. 2

7 Figure 2. Elements of garden projects Level Description Measurable characteristics country level the ECU is the main coordinating body at the country level number of project sites project site A geographic area (e.g., district) where garden projects are supported, usually by a single implementing partner number of gardens in the site gardens one or more beneficiaries An area, usually fenced, where gardening takes place. Management can be by a single individual, a community group, or an institution such as a school or hospital. Individuals who belong to a household and participate and benefit from a garden activity size of garden area under irrigation Variables measured at the number of garden level beneficiaries production and income crop diversification cultivation methods & technology production training provided vulnerability level household demography nutritional status time & labour gardening skills income Variables measured at the individual level Monitoring as a Tool for Participant Education Including beneficiaries in monitoring is a good practice in general, and is particularly important for garden projects. Garden projects require a high level of labour input, and if the project isn t perceived to be working in the eyes of the participants then it will be very difficult to sustain. Furthermore monitoring plays an important role in the learning process which is an explicit goal of most garden projects. Many of the suggested tools in this toolkit therefore are built upon a strategy of using monitoring not only for assessment but also as a tool for training. Gardening Core Indicators During an M&E workshop in February 2006, FAO Emergency Coordinators from southern Africa identified a set of core indicators for garden projects (Lyons, 2006). These are presented in Table 2 below. 3

8 The difference between output and outcome indicators Monitoring is often divided into looking at the process of conducting an activity, and its ultimate impact. Process is measured by describing the immediate outputs, such as the number of inputs distributed or people trained. Impact is evaluated by defining and measuring the medium and longterm outcomes of an activity. In order to claim that an activity made a difference, we need evidence of both the process and the final impact. Hence most monitoring systems simultaneously keep track of both by defining separate indicators for outputs and outcomes. Table 2. Garden core indicators Indicator Output Indicators Basic 1. Number of gardens established, by type, size, and age 2. Number crops per garden, disaggregated by type and size of garden 3. Number of beneficiaries, disaggregated by gender and level of vulnerability 4. Number of gardens using promoted technologies, disaggregated by type of technology 5. Number of beneficiaries trained, disaggregated by type of training and gender School 6. Number of students who benefit from the garden, disaggregated by type of benefit (instruction, school feeding, take vegetables home, etc.) 7. Number of hours of instruction time per week, disaggregated by subject Outcome Indicators Basic 8. Quantity of production disaggregated by crop 7. Quality of dietary intake / meals served (Food Consumption Score) 9. Number of months of production per year Interpretation This indicator will tell us the number of gardens supported, how many are new, and distribution of their sizes. This indicator will capture patterns of crop diversification in the supported gardens, and whether certain garden types or sizes are more diverse than others. This indicator captures the overall scope of garden projects in terms of beneficiaries. This information can also be presented in map form. Many garden projects are designed to demonstrate the effects of promoted technologies, such as biocontrol plants or live fencing. This indicator will capture the types and extent of new technologies tested. Training is a key element of most garden projects, this indicator will capture both the types of training offered as well as number of beneficiaries Students can benefit from school gardens in a variety of ways. This indicator will simultaneously highlight the breadth and depth of impact. This indicator reflects the degree to which the garden is used for educational purposes, and the variety of topics integrated into garden activities Production for consumption or income is one of the primary outcomes of most garden projects. The Food Consumption Score is an index of the consumption of major food groups in the past week, weighted by their nutritional content. It is a good outcome indicator for access to food and the ability to transform food into a diet. This indicator captures one of the main goals of garden projects which is to smooth out food production throughout the year, particularly during the hungry season 4

9 Preferred Indicator Interpretation 10. Number of households who ate Although this indicator doesn t capture the type or or sold produce from supported degree of impact, this is a core indicator that cuts gardens, disaggregated by level of across most activities supported by FAO vulnerability 11. Cropping intensity Cropping intensity is the ratio between the total amount of area cultivated summed over all growing cycles in a year, divided by the physical area prepared for cultivation. It is a measure of intensity of the land and captures whether a garden is being fully utilized. 12. Income generated from garden sales 13. Percentage of beneficiary households who practice garden skills for a second year with little or no direct support Income is an important asset for livelihood security and one of the desired outcomes of many household and community garden projects. This is one of the long-term indicators that reflects the level of sustainability of an activity. What about measuring nutritional status? Most people involved in garden projects hope to ultimately see an improvement in the nutritional status of participants and their families. However, in most cases nutritional status is not necessarily a good indicator for a garden project because garden projects by themselves only address some of the factors that affect nutritional status. Garden projects should improve availability, access, and diversity of food, but improved nutritional status also depends on good health and hygiene, maternal and child care, skills in food storage, and the ability to prepare well-balanced nutritious meals. If a garden project has these additional health and nutrition components (e.g., through joint programming with government or UNICEF), then an indictor for nutritional status would make sense. As an alternative, dietary intake is a reasonably good outcome indicator that captures both availability and the ability to transform crops into a diet. Dietary intake can be measured using the Food Consumption Score which is relatively simple to measure and captures both the diversity of a diet as well as its nutritional content. Therefore this indicator gives us a viable means for tracking how we may be contributing to changes in nutritional status before and after the project. 5

10 Data Collection Workplan At a glance: Basic level data collection Pre-activity planning: conduct a baseline assessment of the physical characteristics of a sample of supported garden sites (does not apply to new gardens) Early in the season: conduct a baseline household survey on a sample of individual participants (backyard and community gardens only) Training: trainers maintain a simple training log, recording the numbers of people trained and topics covered Planting, harvesting, and sales: garden owners/managers keep a diary of planting, harvesting, and sales activity End of season: garden managers and extension officers compile a summary of the season, including training provided, crops planted, production, and sales End of project: an end-of-project beneficiary survey is repeated using the same sample as the baseline The most challenging component of many M&E systems is data collection. A data collection workplan is used to plan the amount of time, human resources, transport, and other resources needed to collect data. A good data collection workplan recognizes the tradeoffs between the information needs, available resources for fieldwork, and the project context. The most important characteristic of a data collection workplan is that it is viable, even if that means collecting less data than you would ideally like. Figure 3. Data Collection Workplan info needs & indicators staff & operational resources project context data collection workplan A data collection workplan takes into account the information needs, operational resources, and project context to come up with a plan for data collection, processing, and analysis that is viable. Table 3 below describes the main steps in implementing a garden project and the accompanying data collection tasks at each step of the process. The sections that follow describe the data collection tasks in greater detail. This table can serve as a guide for a data collection workplan, but is 6

11 incomplete because it doesn't specify individual personnel who will be assigned for each task, geographic areas, transport requirements, amount of time needed, budget requirements, etc. Table 3. Data collection tasks for each phase of implementation Implementation Phase Data Collection Tasks Basic Preferred (in addition to basic) pre-activity planning & garden baseline assessment (page 8) selection of beneficiaries (page 8) baseline household assessment (page 8) training (page 10) planting & production (page 10) harvesting and sales (page 10) end-of-season review (page 12) terminal evaluation (page 12) post-project evaluation (page 15) discussions with stakeholders conduct a baseline assessment of the existing garden sites (page 32) with sample size of 20% (min 20, max 30) (excluding new gardens) garden beneficiary selection worksheet (page 24) administer baseline household survey (page 35) random selection of beneficiaries sample size 5% (min 30, max 200) training log maintained (page 31) garden owner/manager records key events in garden diary (page 25) same sample as baseline garden manager fills in monthly production and sales summary (page 30) same sample as baseline garden manager fills in seasonal summary sheet (page 32) administer end-of-project household survey with same sample as baseline (page 35) stakeholder interviews focus group discussions focus group discussions with potential participants baseline sample size for improved garden projects: 40% (min 20, max 60), including ¼ non-supported gardens 2% (maximum 50) of the Garden Beneficiary Selection Worksheets randomly selected for entry into the monitoring database for analysis of the selection process sample size 10% (min 30, max 400) 25-30% of sampled households should be non-beneficiaries GPS coordinates recorded additional knowledge assessments and evaluations GPS coordinates for each garden recorded additional focus group discussions separate focus groups for men/women external consultant contracted for evaluation revisit some of the sampled households for semi-structured interviews stakeholder interviews focus group discussions repeat as needed 7

12 Pre-activity planning During the pre-activity planning phase, the details of the project design are finalized, sites identified, and roles and responsibilities of partners defined. For improved garden projects (as opposed to new gardens), this is also an opportunity to collect some baseline data on the existing garden sites using the Garden Seasonal Summary (page 32). The focus of this assessment is on the physical characteristics of the garden size, infrastructure, water source, etc. If production and sales data for the previous season are available, these can be entered as well. These data will be used as a baseline comparison for physical improvements to the gardens. All Garden Seasonal Summary Sheets should be entered into the monitoring database. While it may not be possible to visit all of the existing gardens for a baseline assessment, the Basic monitoring standard calls for 20% of project sites to be visited (maximum 20), while the Preferred standard encourages at least 40% of the existing gardens to be assessed (maximum 50). These sites should be randomly selected so that they are representative of all gardens in the project area. For small garden projects, it may be feasible to make a baseline assessment of all the supported gardens. In addition, the Preferred level of monitoring encourages baseline measurements for some nonsupported gardens. These control group gardens should be similar to the project sites as much as possible, with the exception of level of external support. Garden projects supported by another programme would generally not be a good choice for the control group, because we are not usually trying to compare projects against each other. The Preferred monitoring standard also calls for some focus group discussions with potential beneficiaries to learn more about gardening practices, skills, markets, water sources, inputs availability, etc. Such discussions normally take place at the beginning of a project anyway, a focus group discussion is merely a more structured format for focusing and recording the discussion. Selection of beneficiaries The selection of beneficiaries is a big process which is frequently community-based. These factors put limits on the amount of data that can be feasibly collected. The information required at this stage is therefore not detailed information on each and every beneficiary, but general information on the process and criteria by which beneficiaries are selected. The Garden Beneficiary Selection Worksheet (page 24) is designed to help the groups doing the selection to be systematic about the criteria they use, but is not intended to be collected and entered into the monitoring database at the Basic level of monitoring. However the Preferred standard calls for a randomly selected sample of beneficiary selection worksheets (2% with a maximum of 50) to be entered so that a profile of the targeting criteria used can be generated. Baseline household survey The Household Garden Survey (page 35) is the first of two household-level surveys used in backyard and community gardens to collect more detailed information on the household and their previous experiences with gardening. Data from the baseline survey is also used to determine the level of vulnerability of households, which will say something about the targeting methods, and the percentage of participants that come from households affected by HIV/AIDS. The baseline survey should be conducted near the beginning of the garden project, for example during the selection of beneficiaries or soon after the planting activities begin. The same households that are selected for the baseline survey will be revisited during the end-of-project survey, hence it is very important that the households for the baseline survey are selected randomly so that they 8

13 represent the entire population of participants. A good way to randomly select households for the baseline survey is to: 1. decide the total number of participants to be surveyed, based on the needs and resources available for fieldwork, data entry, etc. 2. decide which area(s) will be visited, making sure that at least some households are included from each agro-ecological region, market zone, urban proximity, etc. 3. decide how many participants will be visited in each of the sampled areas (either divide the total desired sample size by the number of areas evenly, or make the sample size in each area proportionate to the total number of participants in each area) 4. take the participant list from each of the selected areas, and select the every n th name on the list, where: total number of beneficiaries in the area n = desired number of households for surveying The Basic standard of monitoring calls for 5% of participants to be selected for the baseline survey, with a minimum of 30 and a maximum of 200 for the entire project. If the Preferred standard is used, the total sample size should be 10% of the total number of beneficiaries, with a minimum of 30 and a maximum of 400, of which ¼ to ⅓ should be non-participant households for comparison purposes. Non-participant households should also be selected as randomly as possible, from the same areas as the participant households, and be involved in other gardening activities if possible. The Preferred standard also recommends recording GPS coordinates of sampled households in order to facilitate relocating the same household during the end-of-project household survey and visualizing the spatial patterns in vulnerability and project outcomes. Our office staff is small and our implementing partners don t have the capacity to do survey work. What should we do? In the real world, operational constraints and institutional dynamics can make it difficult to survey an even moderate sample like 5%, particularly when a project has a late start or field staff have no experience or incentives to do survey work. In these cases, you should still survey as many households as resources permit, and concentrate on making the exercise a learning experience for project staff and partners. Although the small sample size may make it difficult to make definite conclusions about the project as a whole, some interesting lessons can be learned even from small surveys. If nothing else, you are guaranteed to learn some lessons on implementing surveys, such as the amount of time and training needed for enumerators, changes needed to the survey forms, human resources needed for the data entry process, etc. This can help you plan better for the following season. In addition to downsizing the sample, other options may also exist for conducting surveys with limited resources. Many organizations seek to share the costs of data collection by partnering with other organizations. For example the Swaziland ECU uses university students as enumerators when they come for attachment. In Zimbabwe, FAO coordinated 27 NGOs to implement a survey to collectively measure the impact of a number of similar input provision schemes. 9

14 Training Most garden projects include a set of training activities in areas such as production skills, food storage and preparation, health and nutrition, or life skills. The Basic level of monitoring requests training staff to maintain a simple log book of training activities (see Instructor's Training Log, page 31). This template is simple enough to be used by all trainers, including community-based trainers. At the end of each growing season, training logs should be collected and summarized and the total number of trainings, participants, and contact hours recorded on the Garden Seasonal Summary (page 32). The wide array of training topics makes it difficult to provide a universal assessment tool for the many types of training, however the Preferred monitoring standard calls for use of training assessments appropriate for the context. Training assessments can focus on measuring changes in awareness, knowledge, values, or behavior. A variety of assessment methods are used in training evaluations, including interviews, observation of behavior, garden inspections, and focus group discussions. We do a lot of training in our garden project, so why isn t there an outcome indicator for the impact of training? Many garden projects include a training component to improve participants skills in areas such as production, nutrition, food processing, irrigation maintenance, or marketing. The number of topics and people trained is one of the core output indicators for garden projects (Table 2, page 4), and a template for recording the outputs of training is included in Annex II (page 31). However there are no core outcome indicators for training, for two reasons. First, measuring the impact of training properly is very intensive and beyond the capacity of many implementing partners. You have to give knowledge assessments and observe behavior over a period of time that often exceeds the duration of the project. Second, it is very difficult to aggregate information on training impact from different projects and sites, especially when the topics of training are so varied. This does not mean that it isn t important to assess the impact of training, because without assessments of training we really have no way of knowing whether the training is relevant to the participants or if the skills are really sinking in. However this type of focused evaluation is better suited for a terminal evaluation or post-project assessment. Some sample questions for a training assessment are provided in the section on terminal evaluations (page 12). In addition, projects with a heavy training emphasis are encouraged to define additional indicators to measure specific training outcomes for specific projects. Planting, production, harvesting, and sales Monitoring planting and production in gardens is difficult, because unlike field crops cultivation and harvesting take place almost continuously. Hence for record keeping to be effective, it must originate at the garden level, preferably updated on a daily basis. Garden owners/managers can keep track of important events such as planting, harvesting, and sales by maintaining a garden diary using a template such as the one shown in Annex II (page 25). The garden manager is the individual with the overall responsibility for a garden. In an institutional garden, this is usually a staff member of the organization. In a backyard garden, this is an individual 10

15 in the household. A community garden may or may not have a manager from the community. If a garden manager is absent, an extension officer can work with the group to identify someone who can take responsibility for record keeping, or individuals within the garden can maintain diaries for their own plots. Experience has demonstrated that community-level monitoring is more likely to be effective if the information collected is relevant to local needs. Maintaining garden records is useful for many purposes other than monitoring. Keeping a garden diary can be used for various training purposes and demonstrating the effectiveness of various production methods. Record keeping however might be a new skill that will require initial training and extension support. Introducing garden-level record keeping may be particularly challenging in backyard garden projects, because vulnerable households tend to lack formal education as well. When garden diaries don t seem viable or the needed resources for training and extension are lacking, the following strategies might be helpful: community-based extension workers, lead gardeners, or local training staff can provide support to local garden owners/managers in record keeping garden-level record keeping systems can be pilot tested at a sample of garden sites NGO or government extension workers can take charge of record keeping, for example through biweekly or monthly visits To help compile garden-level production and sales data, it is also helpful to have intermediate summaries that can then be aggregated into a final seasonal summary. The Production and Sales Summary sheet (page 30) is a template for compiling daily records of production and sales into a monthly summary. Training might also be needed for this form, including how the information can be used to improve planning and management. Best Practices in Implementing a Survey: Training and Field Supervision As much as we try to ensure that a survey form is clear and easy to use, there has never been a survey that didn't cause some confusion in the field. Hence an important component of survey work is providing training for enumerators. This doesn't have to be extensive, however training is essential to ensure that all enumerators understand the questions, translate them into the local language in a similar manner, record answers similarly, measure the size of fields using the same method, etc. It s also helpful to pilot-test the survey beforehand to catch any mistakes or confusion in the way the questions are written. A second best practice for survey work is field supervision. Particularly in the early stages of data collection, it is important for survey forms to be reviewed by a supervisor as they come in, so any problems or confusion can be quickly identified, diagnosed, and corrected before they cause any data to be thrown out. 11

16 End-of-season review The end of every garden cycle warrants a review regardless of whether project support is coming to an end or will continue for another season. The three-page Garden Seasonal Summary (page 32) is designed to assist an end-of-season assessment by summarizing the physical features of the garden as well as training, production, and sales activity. The seasonal summary can be filled out by garden owners/managers, with assistance from extension staff as needed. The seasonal summary sheet relies on the Garden Diary and/or Production and Sales Summary for production and sales data, and training log(s) for the training summary. If production or training data are not available, these sections should be skipped and the rest of the summary sheet filled out. All seasonal summary sheets should be collected and entered into the monitoring database as several of the core indicators are derived from this data. The Preferred standard also recommends conducting some focus group discussions with participants at the end of each season to explore issues that were missed or too complex to capture during the season. Issues such as the impact of training, gender issues, relationship between the stakeholders, and planning for the next season are examples of possible topics for discussion. The other half of the story: Interpreting indicators Indicators are useful in telling us what happened, but that's only one-half of the story. The other half is why the observed results occurred. To enable the reader to interpret our indicators, we must describe the context during the reporting period. For example production and income might be low not because participation or training was poor but because the fencing material was delivered late. Our description of the results should describe the factors which affected the success of the activity both positively and negatively. Terminal evaluation Terminal evaluations are often a contractual obligation, but should really be seen as an opportunity to share results with partners, review experiences, document lessons learned, and make recommendations for future garden projects. If funds are available, an outside consultant Preferred can be helpful by bringing a wider set of evaluation skills and a broader perspective on garden projects. However even when funds are tight, an internal evaluation is highly worthwhile for packaging, interpreting, and communicating the results of the monitoring system. The elements of a final evaluation often include: analysis and interpretation of project monitoring data interviews with major stakeholders focus group discussions with beneficiaries assessment of the broader context 12

17 An important source of information for a terminal evaluation of a backyard or community garden project is the Post-Harvest section of the Household Garden Survey (page 37). This section of the household survey repeats many of the same questions that were asked on the baseline, allowing a comparison of practices. The Preferred monitoring standard calls for some non-participants to be included in the end-of-project survey as a source of comparison data. We haven t got all of the monitoring data from our implementing partners, and the terminal evaluation report is due next week! What should we do? Missing or delayed data is a fact of life in project monitoring, especially when you rely on implementing partners to collect and/or enter data for you. When you re up against a deadline and some data hasn t been collected and/or entered into the monitoring database, you can still produce summaries with what you have. Although more data usually produces more accurate estimates of total impact, preliminary estimates can still be generated in order to meet a reporting deadline. Delayed data should also trigger a lesson learned for future monitoring. Was the data collection workplan too ambitious? Were the expectations for implementing partners clearly articulated and incorporated into the Letter of Agreement? Were there technical problems with the fieldwork or data entry? By diagnosing the problem you can improve the monitoring system for future reviews. Most terminal evaluations also include focus group discussions and interviews with stakeholders including community leaders, partners from government, other NGOs, and donors. The Preferred monitoring standard calls for separate focus groups for men and women to ensure that gender issues come out clearly and without bias. Separate discussion groups may also be needed if there are other sub-groups among the participants, such as youth or individuals living with HIV/AIDS. Questions that should be addressed in a terminal evaluation are usually outlined in a Terms of Reference, and should be based on the issues that emerged during the course of the season. It is generally preferable to articulate 3-5 key issues for a project evaluation to focus on, rather than ask every question in the book and risk getting an evaluation that is extremely general or based on scanty evidence. See also FAO (1998) for guidelines on planning a terminal evaluation. 13

18 Sample Evaluation Questions for a Terminal Evaluation of a Garden Project 2 Some typical issues addressed in terminal evaluations include: Describe the patterns in vulnerability of the beneficiaries, including impact from HIV/AIDS. Summarize the achievements of the garden project (output and outcome indicators). Describe any observed changes in dietary intake of the beneficiaries. What training was provided, and what are the observable outcomes of this training? How did beneficiaries perceive the garden project? How did implementing partners and other stakeholders perceive the garden project? Describe any barriers to implementation. Describe any gender issues that emerged during the project. Discuss any gender patterns in the core indicators. Discuss the roles of women in all phases of the project (planning, implementation, evaluation), and whether the strategies used by the project to include women were effective. How did households affected by HIV/AIDS benefit from the project? What strategies were used to mainstream the needs of HIV/AIDS impacted households into project design? Were they effective? If technology trials were part of the project design, discuss the results of the trial, the project s strategy for information sharing, uptake, etc. Describe any project-specific objectives (e.g., educational achievements in school gardens, health outreach in hospital gardens) and discuss observed outcomes. What impacts did the garden project have on livelihoods? What will be needed to protect any gains in livelihood, and what additional assistance could be needed to maintain the progress made? Describe any significant changes that took place in the broader context, such as climatic patterns, organizational or political changes, inflation, etc., and how these changes might have affected the success of the garden project Describe any unanticipated benefits or spin-offs that emerged from the garden project, such as additional investment attracted, self-initiated group activities, new relationships between the beneficiaries and extension staff from government and private companies, expansion of technologies demonstrated, etc. Describe any unexpected problems created by the garden project, such as intra-community or intra-household conflicts, elite capture, etc. Describe issues concerning sustainability of the garden project Recommendations for future garden projects 2 see also Machakaire and Hobane (2005) 14

19 Post-project evaluation Output and outcome indicators do a reasonably good job in tracking the short and medium term impacts of gardens, however other anticipated results will only emerge a year or more after the project is over. Post-project evaluations are rare in emergency programmes because there is usually no requirement nor funding to conduct them. Nevertheless, the Preferred monitoring standard recommends a post-project evaluation because there is no other way to determine whether the promoted technologies and approaches for gardening are sustainable on their own or depend on continued donor subsidies. One strategy for conducting a post-project evaluation is to make it a joint exercise with other organizations that have also been promoting gardens, and to include evaluations of past gardens in the M&E workplan of current garden projects. A post-project evaluation is similar in approach to a terminal evaluation, with particular emphasis on changes in the broader context, unanticipated outcomes, creation of dependency, and sustainability of project impact. The best way to prepare for a post-project evaluation is to ensure that the terminal evaluation is well-documented, and data are archived in an appropriate format for re-analysis. MAKING A M&E WORKPLAN Indicators and data collection templates are necessary, but won't collect any data by themselves or produce any analyses. For that you need a workplan which operationalizes the details of monitoring. Each country office needs to take the lead on developing its own workplan for garden M&E as part of its overall M&E work plan in line with local resources, information needs, and operational constraints. A monitoring workplan specifies: what data is going to be collected, when and by who (i.e., data collection workplan, page 6) how, when, and where paper forms will be entered or captured in a database or spreadsheet who is going to do the analysis and write-up estimated costs and time requirements associated with the above tasks This toolkit is not a workplan, but the tools and recommendations included provide many of the core elements of a M&E workplan. To produce a monitoring workplan, the following additional steps are recommended: 15

20 Steps in Developing an Monitoring Workplan 1. Read through this toolkit to get an overview of what is needed and ideas for your specific monitoring system. 2. In addition to the information needs which have already been identified, make a list of the most important national and local level information needs and questions that you would like to address through monitoring. 3. Look at the list of Basic and Preferred indicators on page 3, and decide which ones will be adopted for your country. 4. Think about the specific factors that make households vulnerable to food insecurity in the local context, and come up with a vulnerability index ranging from 1 to 3, based upon the responses to questions in the baseline household survey (see discussion on page 21). 5. Think about the indicators of HIV/AIDS impact, and how to determine whether a household is classified as impacted by AIDS based on the responses to questions in the baseline household survey (page 35). 6. Discuss the indicators and monitoring needs with implementing partners, outlining information needs and options for data collection. 7. Look at the sample data collection forms in Annex II, and based on the chosen indicators decide which ones will be used and whether they need to be modified. 8. Modify the data collection forms as needed. Contact RIACSO and other users of the garden monitoring toolkit if changes to the forms or database are needed. 9. Develop a workplan which outlines all of the needed M&E tasks, including the sampling methodology, field work, data entry, analysis, and writing. The workplan should specify the dates and staff involved in each task. 10. Integrate monitoring responsibilities into job descriptions, individual workplans, and Letters of Agreement. 16

21 REFERENCES Bilinsky, P., and Swindale, A Months of inadequate household food provisioning (MIHFP) for measurement of household food access: Indicator guide. Food and Nutrition Technical Assistance Project (FANTA). Washington, DC. [ FAO Evaluation Mission Brief. FAO Evaluation Service, Rome. [ FAO Improving nutrition through home gardening: A training package for preparing field workers in Africa. Food and Agriculture Organization of the United Nations, Rome. [ FAO School gardens concept note: Improving child nutrition and education through the promotion of school garden programmes. FAO interdepartmental working group, Rome. [ FAO Setting up and running a school garden. A manual for teachers, parents and communities. FAO, Rome. [ FAO A garden for everyone self-tutorial manual. FAO Regional Office for Latin America and the Caribbean. [ Landon-Lane, C Livelihoods grow in gardens: Diversifying rural incomes through home gardens. FAO Diversification Booklet 2. FAO Agricultural Support Systems Division, Rome. [ftp://ftp.fao.org/docrep/fao/006/y5112e/y5112e00.pdf] Lyons, A monitoring and evaluation framework for FAO emergency and rehabilitation activities in southern Africa. FAO RIACSO, Johannesburg, South Africa Machakaire, V., and Hobane, A.P.P Review of garden based production activities for food security in Zimbabwe. FAO Zimbabwe Working Group on Gardens. Harare. Marsh, R Building on traditional gardening to improve household food security. Food, Nutrition and Agriculture. No. 22, pp [ftp://ftp.fao.org/docrep/fao/x0051t/x0051t02.pdf] Mitchell, R. and Hanstad, T Small homegarden plots and sustainable livelihoods for the poor. LSP Working Paper 11. FAO Livelihood Support Programme, Rome. [ Swindale, A., and Bilinsky, P Household dietary diversity score (HDDS) for measurement of household food access: Indicator guide. Food and Nutrition Technical Assistance Project (FANTA). Washington, DC. [ 17

22 18

23 ANNEX I. FAO S REGIONAL MONITORING STRATEGY FOR SOUTHERN AFRICA Monitoring versus Evaluation monitoring: the process of collecting information in a structured manner, based on a specific set of goals and objectives evaluation: the process of summarizing and analyzing information to answer specific questions about a project This toolkit is one of a series developed by FAO Emergency Operations Division in southern Africa to strengthen and harmonize project monitoring and evaluation throughout the region. Project monitoring in general serves three purposes: accountability monitoring allows funders, partners, and beneficiaries to see whether a project is being managed responsibly, is on course, and achieving the desired impacts. implementation monitoring facilitates implementation of an activity. For example, project managers need to know the problems faced by farmers raising small stock in order to plan training activities and procure appropriate inputs. improving project design monitoring allows project staff to make improvements in both the design and implementation of projects, for example targeting, the sequencing and timing of tasks, design of training, complementary services, etc. FAO TCEO and other humanitarian relief agencies are trying to strengthen monitoring for all of the above reasons. To strengthen activity monitoring in seven countries in southern Africa, FAO/TCEO is using a three-pronged approach (Lyons, 2006): a conceptual framework outlining a regional M&E system based on the information needs and operational resources of FAO and its partners a set of core outcome indicators which cut across multiple activities and project sites, in order to show aggregated impact at national and regional levels a series of monitoring toolkits like this one providing guidelines and tools for monitoring different types of activities Why all this new fuss over monitoring? What's wrong with the way we've been reporting? Monitoring in emergency interventions is frequently a challenge due to time constraints, limited staff resources, and the short timeframe of most projects. M&E is therefore often delayed until the project is nearly over, or focuses only on the immediate outputs of an activity. A recent review of project terminal reports has shown that most project reporting has not met the basic information needs of project planners and funding agencies, who have a strong interest in project impact, lessons learned, and the linkages between activities. Without evidence to demonstrate that activities are producing results, the credibility of the entire programme is questionable. This toolkit represents one mechanism to fill in some of these gaps by providing guidelines and templates for a more systematic approach toward data collection and analysis. 19