External Evaluation of the Bomi Social Cash Transfer Pilot Final Report

Transcription

1 External Evaluation of the Bomi Social Cash Transfer Pilot Final Report Dr. Candace Miller Zione Themba Liberia Based Research Team Francis Wreh Joseph Kassellie Boakai Sandimanie Tuwuyor Belleh Moses Dayen Famatta Innis Ezekiel Nyenpan Yusuff Sarnoh Joseph Kwiwalazu Anthony Dymacole Gertrude Weah Numamene Bakolay Kerper Kei Albert Gebeh Center for Global Health and Development (CGHD) BostonUniversitySchool of Public Health Boston, Massachusetts and Liberia Institute of Statistics and Geo-Information Services (LISGIS) Monrovia, Liberia Understanding the Liberian CASH TRANSFER June 2012 This report was prepared by the Center for Global Health and Development (CGHD) at Boston University for the Government of Liberia and UNICEF Liberia. The opinions expressed herein are those of the authors and do not necessarily reflect the views of the funding agency or the Government of Liberia. i

2 STATEMENT OF GRATITUDE To the people of Bomi County, we deeply appreciate the time you spent with us and the honesty with which you told your stories. To the entire research team, thank you for your incredible efforts, despite challenges and setbacks. This report is only possible because of your hard work and dedication to the people of Liberia. You know that poverty is a disease, when it strikes someone, everything that a person does is with poverty in mind. He does not even think about anything else. Male, Bomi County i

3 TABLE OF CONTENTS STATEMENT OF GRATITUDE... I TABLE OF CONTENTS... 1 MAPS... 4 EXECUTIVE SUMMARY... 6 THE BOMI SOCIAL CASH TRANSFER PILOT... 6 EVALUATION OBJECTIVES AND INTENDED AUDIENCE... 6 METHODS... 7 RESULTS... 7 RECOMMENDATIONS... 9 INTRODUCTION...13 BACKGROUND TO THE BOMI SOCIAL CASH TRANSFER PILOT PROGRAM POVERTY, HEALTH AND HUMAN DEVELOPMENT IN LIBERIA CASH TRANSFER IMPACTS OUTSIDE OF LIBERIA THE BOMI SOCIAL CASH TRANSFER PILOT TARGETING APPROACH CASH TRANSFERS (SIZE AND SCALING UP) Table 1. Size of Cash Transfer OBJECTIVES OF THE IMPACT EVALUATION REPORT METHODS...15 STUDY DESIGN SAMPLE SELECTION Figure 1. Intervention and comparison groups SAMPLE SIZE Table 2. Sample size estimation QUALITATIVE RESEARCH In-depth Interviews (IDIs) Focus Group Discussions (FGDs) with Children Focus Group Discussions with Community Members Key Informant Interviews (KIIs) DATA SOURCES AND TRAINING QUANTITATIVE ANALYSIS QUALITATIVE ANALYSIS: INTERVIEWS, FOCUS GROUPS RESPONSE RATE Table 3. Quantitative Survey Sample Size RESULTS...21 DEMOGRAPHICS Bomi County Characteristics of Household Heads Table 4. Demographics - Household Heads Household Composition Table 5. Household size HEALTH AND HYGIENE Table 6. Changes in Adult Health Table 7. Health status among all adults Table 8. Household Expenditure on Health for households that reported health expenditures in previous year Children s Health Final Report 1

4 Table9. Health Status and Changes in Child Health Table10. Chronic Illnesses and Disabilities in Children Table 11. Healthcare according to Household Head Hygiene Practices Table 12. Hygiene Practices of Household Heads EDUCATION School Enrollment and Attendance Table 13. School Enrolment for year olds Table 14. Household Head Reports of Children s Schooling School Expenses Table 15. Annual Household Expenditures on Education (in households that spent money on education) Child Work Table 16. Household Reports of Child Work NUTRITION AND FOOD CONSUMPTION Adequacy of Food Table 17. Adequacy of food consumption Food Diversity Table 18. Household Reported Eating Type of Food Product During One-week Period Food Stores Figure 3. Length of time food stores projected to last among all households reporting at least some stores HOUSEHOLD ASSETS Table 19. Household Assets, Farming Assets, and Livestock Ownership Table 20. Sleeping Conditions HOUSEHOLD EXPENDITURES Table 21. Household Expenditures (LD70=US $1) Table 22. Household Economic Situation Table 23. Social Safety Nets USE OF CASH TRANSFER BY BENEFICIARY HOUSEHOLDS Table 24. Reported Use of the Cash Transfer by Beneficiary Households in the Last Month QUALITY OF HOUSING Table 25. Quality of Housing DECISION MAKING AND BENEFICIARIES USE OF THE CASH TRANSFER Table 26. Decisions About How Money is Used IMPACT OF CASH TRANSFER ON LIKELIHOOD OF EVENTS Table 27. Beneficiary Household Head Reported Likelihood of Events PSYCHOLOGICAL WELL BEING AMONG HOUSEHOLD HEADS Table 28. Psychosocial Well-being of Household QUALITATIVE Table 29. Qualitative Activity, Sample Size, and Themes IMPACTS ON CHILDREN, ADULTS AND HOUSEHOLDS Variation in impacts Priorities for Using the Cash Transfer Cash Transfer Businesses Community Perceptions of Impacts Minimal Iimpacts and Undesirable Uses Community Level Impacts Local Economy THE SITUATION OF NON-RECIPIENT HOUSEHOLDS Coping in Non-recipient Households CHALLENGES FOR THE SOCIAL CASH TRANSFER PROGRAM TARGETING Concerns and Problems with the Targeting EMOTIONAL BLOW OF NOT BEING CHOSEN JEALOUSY CONCLUSION...57 Final Report 2

5 STRENGTHS AND WEAKNESSES SUMMARY OF FINDINGS Impacts Targeting RECOMMENDATIONS WORKS CITED...65 Final Report 3

6 MAPS Liberia in West Africa Bomi County in Liberia Bomi County Final Report 4

7 ACRONYMS CSPC Community Social Protection Committee FGD Focus Group Discussion GDP Gross Domestic Product HHH Household Head IDI In-depth Interview KII Key Informant Interview ME Monitoring and Evaluation PI Principal Investigator RA Research Assistant SCT Social Cash Transfer SCTP Social Cash Transfer Program SCTS Social Cash Transfer Secretariat UNICEF United Nations Children s Fund Final Report 5

8 EXECUTIVE SUMMARY The Bomi Social Cash Transfer Pilot The Liberian Social Cash Transfer (SCT) Pilot is a Social Protection program of the Government of Liberia. UNICEF Liberia provides technical support and together with the European Union and the Government of Japan, provided financial support. The Social Cash Transfer Program (SCTP) was designed to alleviate poverty and improve livelihoods, reduce hunger and malnutrition, and improve school enrolment among the poorest households that are also labor constrained. The SCTP delivers regular and reliable cash transfers to extremely poor households that are also labor constrained. Households are extremely poor when they have minimal assets and income and are labor constrained when there is no person aged able to work either because of age, disability status, or high dependency ratios. Cash transfer programs have yielded positive impacts in African countries where poverty is pervasive and infrastructure is weak. However, the SCTP in Liberia is one of the first programs in a post-conflict society in West Africa. Given the important contextual differences between Liberia and countries with functional cash programs, it is important to examine the Liberian SCTP in an external evaluation when moving from a pilot project to a Liberian program. Thus, the Government of Liberia and its development partners and other stakeholders required an external evaluation in order to examine the SCTP s impact on beneficiary households and communities, as well as the SCTP s targeting procedures. The Government of Liberia and UNICEF Liberia have joined the growing number of countries and organizations committed to evidence-based decision making and using systematic evaluation data to develop social protection policies and guide policy and programmatic improvements. Evaluation Objectives and Intended Audience The purpose of the external evaluation was to assess the situation of cash transfer beneficiary households compared to non-beneficiaries and to better understand the SCTP s targeting outcomes. This report has two main objectives. The first objective is to use qualitative and quantitative methods to examine the differences between cash transfer recipient households (intervention) and non-recipient households (comparison), in the areas of food security, education, health and healthseeking behavior, asset ownership, housing quality, household expenditures, and the well-being of household members. The second objective is to assess the SCTP s targeting process to better understand whether targeting procedures are transparent, fair, and implemented as envisioned, whether cash transfer recipients meet the eligibility criteria of being ultra poor and labor constrained and whether eligible households were omitted from the program. (An economic evaluation and a comparison of targeting strategies were beyond the scope of the evaluation plan.) This evaluation is intended to provide high quality evidence on the Bomi Social Cash Transfer Pilot Program to the Government of Liberia, UNICEF Liberia and other stakeholders throughout Liberia and beyond, who are interested in the impacts and targeting of a social protection program for the extremely poor and labor constrained in a post-conflict society. The outcomes of the evaluation will Final Report 6

9 inform the ongoing development of the national social protection policy and strategy as well as help guide decision making around the scale up of the cash transfer project. Methods This is a quasi-experimental, mixed-method, cross sectional study. The study consisted of the following evaluation and research activities: 1a Quantitative Household Survey with modules on food security, poverty, child health, child development, health and other issues (222 households in total) 1b In-depth Interviews with 27 intervention and 27 comparison household heads 1c Focus Group Discussions with children (20 groups in total, ages years) 1d: Focus Group Discussions with community members (12 groups in total) 1e: Key Informant Interviews with informants from education (n=8), health (n=8), agriculture (n=4), business owners (n=20), and religious leaders (n=4) (44 in total) This study does not yield impact estimates or quantify the expected change over time due to the cash transfer. The stage of implementation, data limitations, and budget constraints precluded a randomized control trial. Still, while not a longitudinal evaluation, we were able to examine how households change when they become cash transfer recipients compared to similar households that were not selected for the SCTP. We present differences between households and attempt to determine whether differences can be attributed to the cash transfer. Additionally, the study design combined quantitative and qualitative activities in order to yield data on differences between an intervention and control group, insights into complex phenomena, obtain a wide range of perspectives, and confirm and verify findings. We included respondents of all ages, economic levels, and occupations in the qualitative evaluation activities. The range of respondents allowed us to gather data on different experiences across households and at different levels, such as among children, adults, households, communities and in local economies. CASH TRANSFERS: (Statistically Significant Differences) Health Health Improvements Adults: 56% of intervention vs. 29% of comparison respondents reported that health improved in the last year. Health Improvements Children: Caregivers reported that 74% of children in intervention vs. 37% of children in comparison households had improved health over the past year. Healthcare Adults: 57% of intervention vs. 27% of comparison respondents reported that they were more likely to seek health care in the last year. Results Bomi County is among the poorest counties in Liberia and among the worst affected by the war, as various factions fought in Bomi in an attempt to take over Monrovia. Infrastructure is inadequate such that there are insufficient schools, health centers, and skilled health professionals. Healthcare Children: Caregivers reported seeking care for 97% of children in intervention households and 77% of children in comparison households for the child s most recent illness. There are poor water and sanitation systems, roads and bridges, and limited access to technology. The war left many women widowed and children orphaned. In this context, we found important differences between the intervention and comparison groups in the quantitative survey. Final Report 7

10 Positive outcomes: In qualitative interviews and focus group discussions, the positive outcomes in intervention households were consistently attributed to the cash transfers. Intervention and comparison household heads, children, key informants and community members all described positive impacts within recipient households. For example, intervention household heads reported better health-seeking behaviors and health statuses among adults and children than comparison household heads. The health impacts were confirmed by the healthcare workers and other key informants, who reported observing important improvements among cash recipients. Intervention respondents reported significantly higher rates of school enrolment than comparison respondents, as well as reduced absences and positive changes in performance. Further, according to respondents there was a significant reduction in child work among children in intervention versus comparison households in the previous year (70% of children in intervention households and 52% of children in comparison households were less likely to work in the previous year). Intervention households reported better food security than comparison households. They were more likely than comparison households to report consuming at least two meals per day and feeling satisfied after meals. The intervention group also had food stores for longer than the comparison group. Household expenditures for food and non-food items were statistically different and higher in intervention versus comparison households. The intervention group reported important housing improvements, including reduced leaks and better quality roofing, walls and floors. Intervention households were significantly more likely than comparison households to share money with people outside their households (19% vs. 8%). They were also significantly more likely to start businesses than comparison households in the previous year (38% vs. 13%). CASH TRANSFERS: (Statistically Significant Differences) Schooling Respondents reported that 92% of intervention children aged 6-17 versus 85% of comparison children were enrolled in school. Further, school enrollment was 94% for intervention girls and 89% for boys versus comparison households where 85% of girls and 85% of boys were enrolled. Intervention children had fewer absentees per month than comparison children (0.60 days versus 1.07 days). Again, girls in intervention households had the fewest absentees. Household heads reported annual school expenditures of LD1,831 in intervention vs. LD1,342 in comparison households (US26 vs. US$19 per year). Food Security and Assets 90% of intervention respondents and 26% of comparison respondents reported that household food intake had improved over the past year. An estimated 60% of the intervention group consumed two meals or more per day versus 28% of comparison households. Intervention households were more likely than comparison households to own a mattress, a radio, a house, and livestock such as chickens and ducks. An estimated 93% of the cash transfer was spent locally within Bomi County in the previous month. Differential impacts: Despite the positive study findings, there were important differences within the intervention group, such that households receiving small transfers experienced much fewer impacts. Intervention adults and children from intervention households expressed frustration with small transfers (i.e. such as LD700 or LD1000), particularly when they felt that they did not receive enough money to send all children in the household to school. Key informants confirmed that some households were not able to send all children to school and that this was a serious issue. In these situations, there may Final Report 8

11 be three explanations: First, the base transfer may be insufficient to sustain the family. Second, the education bonus was not recalculated after children were enrolled in school, so parents who enrolled children subsequent to becoming recipients may not receive the bonus. Third, the program has difficulty capturing household composition changes, in part by design (to keep families from shifting children around in order to qualify) and because of infrequent retargeting. Undesirable uses: In the qualitative interviews and discussions, respondents provided several anecdotes of recipients using transfers for socially undesirable expenses. The frequency with which this occurs appears low, but complaints and reports of misuse emerged in the data in all study clans and should be considered a threat to the program if not addressed. Community level impacts and impacts on the local economy: We estimate that approximately 93% of the cash transfer is spent locally within Bomi County. We also found evidence of a myriad of ways in which the cash transfer generates positive impacts. Business owners reported that they were coming to rely on the cash households as important to their monthly sales and estimated that the transfer led to growth of between 20% to 50%. Some recipients shared money outside the household (19% of intervention versus 8% of control households). We also heard reports that recipients helped out non-recipient families when they had serious healthcare needs. Recipients also created employment opportunities with the transfer and made goods easier to obtain when they started local businesses in their communities. The situation of the control group: The interviews with comparison households and key informants and discussions with children and community members revealed that many non-beneficiary households in Bomi are extremely poor and destitute such that families go hungry and eat poor quality foods, lack adequate housing, and are unable to send children to school. Targeting: The targeting procedures must continue to be reviewed and revised in order to reduce fraud, errors, and corruption. (Fraud occurs when recipients misrepresent their situation in order to receive the transfer. Errors are mistakes made by the SCTP either in targeting, accounting, or other aspects of program management. Corruption is the deliberate cheating by implementers for their own gain or to influence the program.) We heard reports that SCT enumerators demanded bribes, coached households to respond to questions in order to boost chances of selection; and that on occasion, households were not selected or removed from the SCTP for arbitrary reasons (i.e. hair not grey). Furthermore, despite the revised targeting approach, implemented in 2011, that relied on SCT enumerators rather than community members, some eligible households were left out of the initial listing. (The approach has subsequently been revised from utilizing ad-hoc SCT enumerators to full-time, regularly paid, government employed SCT enumerators. This revised approach should be monitored and examined to determine its effectiveness.) Finally, there were also situations in which grievances were not heard or rectified. Recommendations The study findings provide ample evidence that for the majority of cash transfer recipients the impacts are transformative. Previously destitute households are able to make significant improvements in their lives and improve health, food security, children s education, asset ownership, and housing. These impacts spill over into communities and local economies. Still, we documented important potential threats to the program, such as misuse of the transfer, variation in impacts and problems with the targeting procedures. We offer recommendations for programmatic Final Report 9

12 improvements so that the program yields the intended impacts as it expands to and throughout additional counties. Recommendations are to improve 1) operations 2) targeting procedures and 3) impacts. (According to UNICEF, some of the recommended revisions are already being developed.) To improve SCTP Operations, we suggest the following: Develop and implement a full monitoring system designed with checks and balances to regularly monitor all program activities. The system must have feedback loops for information to stimulate continued programmatic improvements. Regular but unannounced monitoring of SCT enumerators can reduce instances of bribery. Community outreach activities should be implemented to identify households that may have been overlooked during the targeting activities. Monthly SCT reports should be submitted and scrutinized to understand field challenges so that feasible solutions can be devised and implemented. Use multiple communication media to publicize aspects of the program, including the targeting procedures, eligibility criteria, payment levels, and expected uses of the cash. Empower community members to observe the process and report fraud or targeting errors. Spot check and monitor SCT staff that list households to reduce errors and corruption. There is an increased opportunity for fraud and corruption when communities do not fully understand the targeting criteria and program. While there is a risk that SCT enumerators or monitors could coach households or that households may commit fraud in order to obtain the cash, this could be reduced through better monitoring. Although the costs of improved monitoring have not been estimated in this report, the current approach, when further scaled throughout the country, risks serious inclusion and exclusion errors that would threaten the program s long-term sustainability. While attaching conditions to cash transfer programs has generally not been welcomed in many African countries because of low capacity to monitor conditions and inadequate infrastructure to deliver services, some countries (e.g. Kenya and Uganda and others) are empirically testing and comparing the costs and burden of conditions within cash transfer programs. It may be worth considering some conditions in Liberia to reduce the use of transfers on socially undesirable expenses. Establish and publicize grievance procedures so that there is a process for citizens to report perceived problems with the program. Problems should be documented and responded to appropriately. Grievance systems are not difficult to design, however, they do require staffing to ensure that problems and grievances can be submitted and that there is a process to respond to them. To improve the program s targeting procedures and outcomes, we suggest the following: Add additional checks and balances to the current targeting process in order to reduce the ability of SCT enumerators or monitors to require and accept bribes. The SCT Secretariat could arrange spot checks and visit households to ensure that eligible households are chosen and to ensure that excluded households do not meet the program s eligibility criteria. Ensure that there is adequate staff in place at all levels in order to ensure quality and fairness as the program expands. The SCT Secretariat and staff must allocate more time to monitoring activities, ensuring that the right households are chosen, and that grievances are documented and rectified. To their credit, the SCT Secretariat office manages a wide range of activities. However, the work burden may be too great to manage the range of challenges that are inevitable in a country where the majority of people are poor, and where civil servants are not well paid and are often paid late. By ensuring adequate staffing levels, the Final Report 10

13 SCT office can allocate more time to monitoring activities, ensuring that the right households are chosen, and responding to grievances. Develop additional community outreach methods to ensure that households are not overlooked and thus excluded from the transfer, particularly in hard to reach locations. We frequently heard of households that were not included because they were not listed or were not home when the SCT enumerators did their listing. If households are overlooked, they should be able to still be included in the program as long as they meet the criteria. We suggest using radio and other methods to build awareness of the targeting procedures as well as encouraging local leaders to be more aware of households that might meet the criteria but are difficult to find or visit. Transparency within communities has helped reduce targeting errors in other countries. We strongly encourage program implementers to examine whether the SCTP can be expanded to additional households within Bomi because we encountered a number of families that met the eligibility criteria but were excluded, despite their destitute circumstances. In some cases these families were not home when the SCT enumerators listed households. There were also corroborated cases where households were intentionally excluded despite meeting the eligibility criteria. In order to maximize program impacts, we recommend the following actions: Provide business training skills to interested cash transfer households. Some recipients established businesses on their own and were successful while others encountered problems or took on strenuous businesses that were burdensome for children and adults. Partnering organizations could provide business training and assist with a market analysis to determine which businesses would be the most successful. Households should be sensitized to the problems associated with child labor, particularly in Liberia where children burn coal, mine and do other dangerous work in order to earn money for food and schooling. While child work in poor households without cash transfers may be inevitable in the absence of additional social protection programs, in cash transfer households, families should be sensitized to the dangers of child work. One condition of cash may be that children should not do harmful or dangerous work. Program implementers should consider launching a social norm campaign to build the culture of using transfers appropriately. Publicizing the positive uses of cash transfers can generate healthy competition, and inspire or shame recipients into using the transfer for food, health, improved housing and education. Program implementers should further link to health services, education and agricultural projects and other sectors to increase the likelihood of a range of positive impacts. While the SCT office is at capacity, there are community based organizations and non-governmental organizations that could provide additional services to cash recipients who are an active audience during monthly transfer disbursement gatherings at pay points. We suggest assessing how to further assist one person households that receive LD700 per month. For example, if the Secretariat or a partnering organization was able to link multiple one-person households together through a savings group (susu) these individuals might be able to pool money for greater impacts. Reassessing the transfer amount may also be necessary. The trade-off between fewer households with larger benefits versus more households with smaller benefits is a serious question for debate. This study does not have an adequate sample size or design for a full analysis of the transfer size versus impacts. Final Report 11

14 Finally, given the widespread poverty in Bomi County and throughout Liberia and the number of households in a destitute situation, continuing to develop a responsive, reliable social protection system is essential to the human and economic development of Liberia. We found many households without labor that likely should have been included in the cash transfer. Based on the SCTP data, qualitative key informant interviews and focus group discussions, we conclude that these households were excluded because of targeting errors, rather than not fulfilling the eligibility criteria. We also encountered many extremely poor households with labor that require appropriate social protection solutions such as cash for work programs. As the cash transfer pilot program is scaled up throughout Liberia and the social protection system is further developed, we strongly recommend that stakeholders throughout Liberia continue to prioritize evaluation research to determine whether programs and policies are effective, to obtain impacts estimates, to reduce threats to program scale up and sustainability, to test and assess new and improved approaches, to determine what impacts cost, and to compare the costs of different approaches through a cost effectiveness analysis. Final Report 12

15 INTRODUCTION Background to the Bomi Social Cash Transfer Pilot Program The Liberian Social Cash Transfer (SCT) Pilot Program is a Social Protection program of the Government of Liberia. The SCTP was developed with technical support from UNICEF and financial contributions from UNICEF, the European Union, and the Government of Japan. The SCT was designed to alleviate poverty and improve livelihoods, reduce hunger and malnutrition, and improve school enrolment among the poorest households that are also labor constrained. The Social Cash Transfer Program delivers regular and reliable cash transfers to extremely poor households that are also labor constrained. Households are extremely poor when they have minimal assets and income and are labor constrained when there is no person aged able to work either because of age, disability status, or high dependency ratios. The Liberian SCT pilot was implemented by the Government of Liberia, through the Ministry of Gender and Development. The design and the financing of the program were facilitated by UNICEF. In Bomi County, the Social Cash Transfer Secretariat is responsible for all operational aspects of targeting and delivering transfers to recipient households. i,ii Before decisions are made to scale up the cash transfer program, the Government of Liberia, UNICEF and other stakeholders required an external evaluation in order to examine the SCTP s impact on beneficiary households and communities and to examine the program s targeting procedures. The Government of Liberia and UNICEF have joined the growing number of countries and organizations committed to evidence-based decision making and using rigorous evidence to guide policy and programmatic improvements. Poverty, Health and Human Development in Liberia Poverty is pervasive in Liberia due to more than a decade of war and a history of poor governance. In 2010, per capita GDP was estimated at US$247, which ranks Liberia among the poorest countries in the world. iii In 2007, the poverty headcount, or the ratio at the national poverty line, was 63.8%. iv Poverty is greatest in rural areas and particularly in Liberia s remote counties. As of 2008, 79% of urban and 51% of the rural population had access to improved water sources and 25% of urban and 4% of the rural population had access to improved sanitation facilities. v In 2011, life expectancy at birth was 58 years. vi The maternal mortality rate was 990 per 100,000 births (2008) vii and the infant mortality rate was 88 per 1000 live births (2011). viii The adult literacy rate was 58% in ix Education expenditures as a percentage of GDP were 2.8% in 2008, x which ranks Liberia at 144 among 163 countries in the world. The school-life expectancy rate was 11 years in 2000, when the latest data was available. xi In 2007, the percentage of children under five years old that were underweight was between 15 and 20.4%. xii Cash Transfer Impacts Outside of Liberia Cash transfer programs have yielded positive impacts in other African countries where poverty is pervasive and infrastructure is weak. xiii In fact, in a similarly designed program in Malawi, an external evaluation revealed impacts in poverty reduction xiv, food security and diversity xv, child education xvi, reductions in child poverty xvii, improved ability for persons with chronic illnesses to access healthcare, and to provide for family members. xviii However, the SCTP in Liberia is one of Final Report 13

16 the first SCT pilot programs in a post-conflict society in West Africa. Given the important contextual differences between Liberia and countries with functional programs that are yielding positive impacts xix, xx, it is important to examine the Liberian SCT in an external evaluation. The Bomi Social Cash Transfer Pilot Bomi was selected as the pilot county for the cash transfer program because of its exceptionally high level of food insecurity, which approached 75% in the 2007 Comprehensive Food Security and Nutrition Survey. The county s proximity to Monrovia also allowed better oversight and monitoring from the capital over the course of the pilot. Photo 1. Paying the cash transfer The SCT program was launched in February 2010 and scaled up throughout Bomi County by September 2011 to reach 1,900 families (7,123 people), approximately 10% of the households in Bomi County. Targeting Approach Households are selected in a multi-stage process, led by enumerators or monitors from the Social Cash Transfer Secretariat (SCTS). The enumerators conduct two rounds of interviews in communities to determine whether households meet the eligibility criteria. Once households are identified by enumerators or monitors, town chiefs and the county social protection committees review the list, make corrections or approve households. The targeting approach was revised in 2010 and 2011 in order to reduce inclusion and exclusion errors. Prior to revisions, the targeting was completed by community volunteers. However, the original procedures led to serious inclusion and exclusion errors. Thus, the approach was modified to be less dependent on community processes given that solidarity and volunteerism is weak in Bomi County. xxi Households must meet the eligibility criteria to receive cash transfers. One criterion is extreme poverty which is determined by the household s food security, asset ownership, and whether there is any support for the household elsewhere. Second, households must be labor constrained, which is defined as having no adult between the ages of 19 and 64, or the adult is not able to work because of chronic illness, disability, or needing to care for more than three children, disabled or elderly people. Cash Transfers (Size and Scaling Up) Once households are approved as beneficiaries, they receive monthly transfers, which are paid at a local meeting point, where staff from a commercial bank and SCT officials manage the process (Photo 1). The value of the transfer depends upon the size of the household (Table 1). The basic grant is unconditional, though additional sums are added for children enrolled in school. Final Report 14

17 Table 1. Size of Cash Transfer Number of household members L$ per month US$ per month $ $ $ $25 Additional amount for each child in primary school 150 $2 Additional amount for each child in secondary school 300 $4 Average payment size 1750 $25 According to UNICEF, there is donor commitment to scale up the SCTP to 5,000 households in Bomi and Maryland Counties through In line with the original selection of Bomi County, the order of counties for future inclusion is based on the level of food insecurity in each county. If scaled up nationally, it is estimated that approximately 50,000 families would meet the current program criteria. xxii Objectives of the Impact Evaluation Report The purpose of the external evaluation was to assess the situation of cash transfer beneficiary households compared to non-beneficiaries and to better understand the SCTP s targeting outcomes. Thus this report has two main objectives. The first objective is to examine the differences between cash transfer recipient households (intervention) and non-recipient households (comparison) in the following areas: Nutrition and Food Consumption (food security, satisfaction with consumption levels, food diversity, consumption of complex proteins, food stores) Education (enrolment, attendance, perceptions of performance) Health and Hygiene (perceptions of health status, recent illnesses, health-seeking behavior) Child Labor Asset Ownership (durable goods, productive assets, livestock) Housing Quality Household Expenditures and use of cash transfer (food, assets, health, education, and other items) Well-being of Household Members The second objective is to assess the targeting process to better understand whether procedures are transparent, fair, and implemented as envisioned; whether cash transfer recipients meet the eligibility criteria of being ultra poor and labor constrained; and whether eligible households were omitted from the SCTP. METHODS The Boston University Institutional Review Board and the University of Liberia Institutional Review Board approved the study protocols submitted for the quantitative and qualitative portions of the evaluation. Study Design Final Report 15

18 This is a quasi-experimental, mixed-method, cross sectional study. The study consisted of the following evaluation and research activities: 1a Quantitative Household Survey with modules on food security, poverty, child health, child development, health and other issues (222 households in total) 1b In-depth Interviews with 27 intervention and 27 comparison household heads 1c Focus Group Discussions with children (20 groups in total, ages years) 1d Focus Group Discussions with community members (12 groups in total) 1e Key Informant Interviews with informants from education (n=8), health (n=8), agriculture (n=4), business owners (n=20), and religious leaders (n=4) (44 in total) The sampling frame for the intervention group was a roster of cash transfer households that were included in the SCTP because they were identified as being extremely poor and labor constrained. The sampling frame for the comparison group was a roster of households that were listed but were not selected for the SCTP because they did not meet the criteria of being labor constrained. In theory, the comparison group was an approximate counterfactual of the intervention group. The group met the poverty criteria, but according to the SCT office, did not meet the labor constrained criteria. This study does not yield impact estimates because it is not a longitudinal randomized control trial. The stage of implementation and budget constraints precluded undertaking a randomized control trial. Rather than presenting impact estimates, we compare households that were approved to receive the cash transfer with households that do not receive the transfer. We aim to determine whether intervention households have similar or better indicators than the comparison group and to determine whether differences can be attributed to the cash transfer. While attribution is difficult in many evaluations, in this case, we know which households received cash, we can observe what they bought with cash, and we know there were no other interventions that could have caused economic changes in households. Thus, we can attribute changes in these households to the cash transfer with some confidence. The quantitative portion of the study was conducted in clans where the cash transfer program had been operational for at least one year, while the qualitative portion of the study was conducted in clans where the SCTP had been operational for 18 months to two years. Still, we asked respondents about their current perceptions of the program with the revised targeting approach. Sample Selection We used the targeting data from the Social Cash Transfer Secretariat office in Bomi and randomly selected cash transfer households to form the intervention group. These were Category D households that should have met the program s eligibility criteria when the targeting activities took place in 2010 (Figure 1). We also randomly selected households for the comparison group that met the extreme poverty criteria, but did not meet the criteria of being labor constrained when the community level targeting process took place in In other words, the comparison group was comprised of households (in Category C) that just missed the eligibility criteria because they had a dependency ratio of three or better, rather than worse than three when the targeting exercises occurred. This non-beneficiary group provides a comparison to help show the change in households based on the cash transfer. Note that the quantitative data collection only occurred in clans where the revised targeting Final Report 16

19 approach was implemented so that we did not include households from clans that used the earlier more problematic approach. We can thus generalize findings from the quantitative study to other clans that utilize the newer, revised approach. Figure 1. Intervention and comparison groups Labor Constrained: No able bodied adult aged or a Dependency Ratio worse than three (dependents may be children, elderly, chronically ill or disabled) Poverty: 48% below the extreme poverty line A Not poor, not labor constrained C Poor, not labor constrained B Not poor, labor constrained D Poor, labor constrained Sample Size Intervention group from Category D. Comparison group from Category C should be among poorest & near meeting labor constrained criteria. criteria. The quantitative evaluation study was powered to examine the differences between households based on receipt of cash transfers. We hypothesized that the cash transfer would yield a 15 percentage point difference in food security between cash recipients and non-recipients (e.g. the proportion of households that consume two or more meals per day over one year); increase the percentage of households that own 3 or more valuable assets by 30 percentage points; and increase the percentage change in expenditures for all consumption categories of spending (Table 2). Table 2. Sample size estimation Domain Outcome Expected outcome in comparison and intervention households (respectively) Expected Change Over Time Food security Percent consuming 2+ meals per day 70%, 85% +15 percentage points Assets Percent owning valuable assets 50%, 80% +30 percentage points Poverty Difference between expenditures for consumption categories 20% difference in expenditures We planned to measure differences between intervention and comparison households in the domains of nutrition and food security, asset ownership, expenditure patterns, schooling, child labor, health and health seeking behaviors, well-being, and coping behaviors. Using the Decision and Support System s online Power and Sample Size program, we calculated the necessary sample size for a study to determine differences in the above outcomes given the following parameters and assumptions: Alpha =.04, the conventional estimates for a Type I error for a two sided test and Power =.80, or the probability of correctly rejecting the null hypothesis of equal proportions in the population. We estimated a sample size of approximately 210 households (105 intervention and 105 comparison households in Bomi County). In order to reach our desired sample size for the quantitative portion of the study, we oversampled households because we knew that some households would not be available for the interview for a variety of reasons. The sampling rosters contained 136 randomly selected intervention and 136 Final Report 17

20 comparison households proportional to the size of the clans. The field team systematically enrolled households to maintain a sample proportional to the size of the clan. Again, this is neither a longitudinal study nor a randomized control trial. While differences between groups may be statistically significant, we cannot determine whether the differences are solely due to the cash transfer or isolate the effect size. We do not have baseline data to determine whether there were statistically significant differences between the intervention and comparison groups prior to the cash transfer intervention. We did however use limited demographic and economic data collected by the SCT Secretariat on each group, which indicated that all households were extremely poor during the targeting activities. Qualitative Research In the qualitative phase of the impact assessment, we conducted 1) in-depth interviews with intervention and control household heads; 2) focus group discussions (FGDs) with children aged 10 to 18 years from intervention and comparison households; 3) focus group discussions with community members; and 4) key informant interviews with local civil servants and community leaders, including health workers, agricultural workers, teachers, religious leaders, and business owners. In-depth Interviews (IDIs) The purpose of the qualitative in-depth interviews (IDIs) with intervention and comparison households was to generate more detailed data and information on sensitive topics related to the economic situation of the household, depending upon whether the household received the transfer. For the IDIs, we identified households through a simple random sampling of all households that had been listed by the cash transfer office. xxiii Households receiving the transfer were intervention households and households not selected as recipients were comparison households. We selected a representative sample and interviewed the household head. Focus Group Discussions (FGDs) with Children The purpose of the FGDs with children (ages years) was to generate more detailed data and information on the situation of children in both intervention and comparison households and their perceptions of other children and families. We asked the young respondents about the experiences of children depending upon whether their families received cash transfers. Again, we used the rosters from the SCT office to identify intervention and comparison households and we invited children from these households to participate in separate FGDs. We held the discussions at a safe and private local meeting area. Focus Group Discussions with Community Members The purpose of the FGDs with community members was to learn about community perceptions of cash transfer impacts on recipient households and on the overall community. We examined both positive and negative perceptions and concerns about the cash transfer program; perceptions of recipient behaviors regarding livelihoods, health seeking behaviors, and use of the cash transfer; the community s perceptions of the fairness of targeting; the existence of jealousy among nonbeneficiary households; and perceptions of possible economic multiplier effects. We conducted FGDs with community members from villages where the program had been operational for 18 Final Report 18

21 months to 2 years. We asked local community leaders to help us invite community members, who had not been a part of any related evaluation activities. Key Informant Interviews (KIIs) The purpose of these interviews was to assess perceptions of the impact of cash transfers on households and the community from the perspective of a variety of stakeholders. All key informants, including health care workers, teachers and headmasters, religious leaders, business people, and agricultural workers were familiar with the cash transfer program. Again, we examined both positive and negative perceptions of the cash transfer program. We asked what respondents observed in regards to recipients use of the transfer, health seeking behaviors, and livelihood activities; the community s perceptions of the fairness of targeting; the existence of jealousy among non-beneficiary households; and perceptions of possible economic multiplier effects. During field work, we identified key informants and conducted interviews with local leaders from healthcare facilities and schools, agricultural extension workers, religious leaders, and business owners. Data Sources and Training First, we developed a structured quantitative questionnaire in English to obtain data on household food security, expenditures, safety nets, asset ownership, and household members health and activities, as well as child education. The survey was circulated for comments and feedback. In Bomi County, we trained the team of research assistants (RAs) over one week prior to each round of data collection in order for RAs to understand the objectives of the study, learn the instrument and the purpose of every question, revise the survey instruments to ensure they were appropriate to the local context, conduct mock interviews, and pilot the instrument. Following the pilot, we debriefed extensively and provided ongoing training to ensure high quality interviews. We developed semi-structured guides for in depth interviews (IDI), key informant interviews (KII) and focus group discussions (FGD) in order to understand the positive and negative impacts of the SCTP on beneficiary households and in communities. In addition to the training described for the quantitative portion of the study, we also conducted training in qualitative methods over one week and provided ongoing instruction as interviews and focus groups were completed. The RAs participated in mock and pilot interviews, KIIs, IDIs, and FGDs. These activities were followed by reflective discussions and training and instruments were revised as needed. RAs conducted IDIs and FGDs in pairs or individually, depending upon their level of experience. They took handwritten notes of all interviews and focus group discussions were taped using digital voice recorders. The transcripts contain all direct quotes from each respondent. Transcripts were reviewed and interviewers were asked for clarification or required to make call-backs if necessary. Training consisted of the following modules: Introduction to the study Interview methods, including managing the interview and training respondents, probing, being aware of non-plausible or contradictory data, dealing with difficult interviews and challenges, and problem solving Qualitative versus quantitative methods (classroom instruction and discussion, mock interviews, pilot interviews) Data management (including collection, entry, cleaning, security) Roles and responsibilities of team members Final Report 19