Evaluation Brief 2013 Information & Communications Technology for Maternal, Newborn and Child Health in Malawi: Evaluation Methodology This brief presents an overview of the methods used to evaluate outcomes of the Innovations for Maternal, Newborn & Child Health pilot project in Balaka District of Malawi, implemented by VillageReach. The pilot, called Chipatala cha pa foni ( CCPF ) or Health Center by Phone, used information and communications technology interventions to provide pregnant women and caregivers of young children new and reliable means of accessing maternal, newborn and ( MNCH ) information. In Malawi underlying causes for poor health outcomes among women and children include the lack of timely, reliable health information, limited access to health facilities, and scarce human resources. Women and children frequently travel long distances across difficult terrain to see a health professional, often to face long wait times upon their arrival. Implemented in Balaka District in the southern region of Malawi, an area with some of the lowest maternal and indicators in Malawi, CCPF helps bridge the divide between communities and life-saving health and information services. Photo: Doreen Namasala works as one of the hotline operators for CCPF at the project call center in Balaka District Hospital. She has been with the project since September 2011. Her colleague Lawrence Mtambo (background) has been a hotline operator since May of 2011. Kieran McConville/Concern Worldwide Newborn & Child Health, visit: www.innovationsformnch.org
The Innovation CCPF set out to provide women and families with a reliable means of accessing health information and care. The services offered included: A toll-free case management hotline providing protocolbased health information, advice and referrals. Callers who may not be able to access an in-person consultation at a health center are connected to trained personnel at Balaka District hospital. An automated and personalized tips and reminders service for pregnant women, guardians of children under one year of age and women of child-bearing age. Users of the hotline are offered the opportunity to receive weekly calls on appropriate care seeking and health practices. These can be received as SMS or voice messages. Village volunteers, trained and provided with phones, mobilized users in the catchment areas of four health centers and ensured access to services to those without phones. with the formal health system, improving the perceived quality of MNCH services measured by satisfaction. The following outcome-related hypotheses were tested: Coverage-related outcomes Increased knowledge of home-based and facilitybased practices for maternal and Increased behavioral use of home-based practices for Increased behavioral use of home-based practices for Increased behavioral use of facility-based services for Increased behavioral use of facility-based services for Quality-related outcomes Increased client satisfaction with facility services for Increased client satisfaction with facility services for Evaluation Design & Methods The project s implementing partners developed a mixed method approach to understand and examine exposure to CCPF and change in MNCH knowledge and behavior. For the quantitative component a quasi experimental, pretest and post-test design was used, consisting of a crosssectional household survey in the four catchment areas of the intervention and in two comparison health centers. This component was supplemented by focus group discussions, in-depth interviews client exit interviews, key informant interviews and rapid health facility assessments. An ethnographic method to record information called Hearsay was also used. Photo: Elesyia Mulinde, who is six months pregnant and lives in the village of Chiyendausiku, Balaka, listens to messaging from CCPF on a cellphone provided by Community Volunteer Jenna Kachipewa. Kieran McConville/Concern Worldwide Hypotheses The main project hypothesis was that exposure to the intervention will result in improved coverage and quality of MNCH services. In terms of coverage, CCPF was expected to increase care-seeking among pregnant, lactating, and postpartum women, and of care-givers of children under five by spreading timely information. In terms of quality, CCPF was expected to foster more positive interactions Quantitative methods Household Survey The core of the assessment were baseline (carried out between June and July 2011), endline (carried out between May June 2013), cross-sectional population-based surveys of mothers aged 15-49 who had children under 5 years of age, as well as pregnant women and caregivers of children under 5 years of age. Data on household characteristics was also collected. Questions were based on the Multi-Indicator Cluster Survey and included questions on MNCH knowledge and behavior. The end line survey in the intervention sites included questions on exposure to and use of CCPF. 2
Sampling & Data Collection The primary sampling units were the villages; the sampling methodology was a one-stage cluster sample with stratification at the health center level. At baseline, GIS information and maps from the health centers were used to define the catchment areas of the health facilities and to create a comprehensive list of villages with information on each village s total population, estimated number of women of child bearing age and estimated total of children less than five years of age. Photo: Patuma Grey and her 7 month old daughter Shalia who live in the village of Sikawandeu, Balaka district, Malawi. Patuma used the CCPF service during her pregnancy and after Shalia was born. Kieran McConville/ Concern Worldwide Using a systematic sampling approach, villages were randomly selected to be included in the study such that villages in each health center catchment area had the same probability of being selected into the sample regardless of their population size. The rationale for this approach is that data from women in smaller villages, where arguably there is less knowledge about MNCH, would have the same weight as data from women in larger villages. Every household in the selected villages was listed and every household that had a woman in the age range 15-49 years, and/or had a caretaker of a child under 5 years of age was eligible for the study. At baseline questionnaires were administered until predetermined numbers had been reached. At end line IKI returned to the same villages and again listed all households. Questionnaires were administered for all eligible households except where members of the household were not available during the period of the study field work. The number of survey respondents is shown in Table 1. Table 1. Number of survey respondents by population group at baseline and at endline Baseline survey End-line survey Intervention Control Total Intervention Control Total Households 1600 1210 2810 2398 1281 3669 Women 1622 1217 2839 2508 1344 3852 Under-five children 2137 1483 3620 2221 1089 3310 Qualitative methods Several qualitative methods were employed, with some methods only used in either baseline or end line (see Table 2 for details). Sampling for these methods was purposive, with specific considerations prioritized depending on the method. Focus Group Discussions Focus group discussions were used to inform the quantitative data gathered by the household survey. At baseline, twelve focus group discussions were conducted with women. Two focus group discussions were conducted in each of the four treatment areas (eight 3
randomly selected villages) and two FGDs were conducted in each of the two control areas (four randomly selected villages). Three focus group discussions were conducted with men. At endline, questions on exposure to CCPF were included. Participants for 12 focus group discussions in the intervention areas were randomly selected from the household survey. Focus group discussions were conducted with women who had used CCPF, women who had heard but not used CCPF, and women who had not heard of CCPF. In-depth interviews These were conducted at endline to understand and obtain further details from specific individuals who actively participated in focus group discussions. Interviews were carried out with women who had used CCPF and those who had heard of the project but not used it. Key Informant Interviews Conducted at end line only, these were held with key project stakeholders to explore aspects of implementation with those familiar with CCPF. Staff members of the health facilities (the in-charge and a nurse-midwife), and Health Surveillance Assistants, were interviewed. At district level, four District Health Management Team members were interviewed. Three members of the VillageReach project implementation team were also interviewed. Client Exit Interviews These were carried out at baseline at the health centers. Exit interviews seek to understand client perceptions of the quality of service at the health center and the time spent waiting for the service, as logged through a wait time register. They were not carried out at end line; instead women s perception of satisfaction with services was evaluated using the quantitative questionnaires. Rapid Health Facility Assessments Health facility assessments were conducted at the four health centers in Balaka District and the two comparison health centers in Ntcheu during the baseline survey to provide information on availability of services, supplies, staffing, etc. The assessments were repeated at endline. Hearsay Ethnographic Method Hearsay ethnography is a method for studying conversations and social interactions in their natural social settings. The method provides a way to capture informal talk about project interventions and, more generally, MNCH in the Balaka health center catchment areas. Two community members in each of the four Balaka health center catchment areas were selected and trained to pay attention to public conversations about the project interventions and write about them in a field journal. Data from the journals was used to supplement other qualitative methods. Table 2. Type and number of qualitative methods used at baseline and end line Baseline Endline Focus Group Discussions, number of groups 15 12 In-depth Interviews, number of respondents -- 16 Client Exit Interviews, number of respondents 664 -- Key Informant Interviews, number of respondents -- 47 Rapid Health Facility Assessment, number of facilities 6 6 Hearsay, journals from 8 journalists reviewed ^ ^ 4
Photo: The CCPF logo is painted on the back wall of the call center at Balaka District Hospital. Kieran McConville/Concern Worldwide Intention to Treat: This approach estimates differences in intervention and control groups, regardless of CCPF uptake. Heterogeneous Treatment Effects: This refers to the potential for different types of individuals to respond differently to the intervention. Identifying these potential treatment effects prior to data analysis guards against data mining for sub-groups where effects are statistically significant due to chance. The effects were anticipated based on distance to health center, presence of Health Surveillance Assistants, women s parity, access to a mobile phone, literacy, age, marital status, and ethnicity. Analysis The analyses included a descriptive use, and estimates of the effects of the intervention. The descriptive use of the interventions included who and how many potential users had heard of CCFP and of those, how many had called the CCPF hotline or subscribed to messages. Those who used were further analyzed against a number of variables: age, marital status, education, number of children under five, ethnic group, ownership of a mobile phone, wealth (defined by status of walls, roof and floor of house). The overall effects of CCPF were assessed through various methods applying the difference-in-differences techniques, with outcome variables from the treatment and comparison groups analyzed based on a set of assumptions. A major assumption is that the difference between the intervention and control areas prior to the interventions remains constant in the absence of treatment. The evaluation also estimated the effectiveness of the interventions within a multivariate regression framework, allowing inclusion of the other predictors of interest (as described earlier) to provide a more precise estimate of the effect. In order to reduce the number of tests to be conducted, a mean effects index was calculated for each hypothesis, allowing collation of multiple indicators of the same outcome into a single index. This permitted capturing the average relationship between CCPF project interventions and a group of indicators for the same outcome. The following difference-in-differences estimates are modeled: Treatment on the Treated: These estimates are meant to determine the effects of the interventions on the individuals who used the services compared with those who did not, in both intervention and control areas. The Treatment on the Treated effects are estimated using assignment to the intervention area as an exogenous predictor for actual use of CCPF, which essentially determines how big the Treatment on the Treated effect would need to be in order to account for the Intention to Treat effect, given the rate of using CCPF in intervention areas. Limitations of the Evaluation During implementation of CCPF some adjustments were made to the pilot, namely dropping the booking system component. This had implications for the evaluation design since it reduced the number of outcomes associated with the quality objective. The hotline and tips and reminders were offered in all four intervention health centers. As the two services were often not clearly distinguished by users it was difficult for the evaluation to distinguish between the effects of the tips and reminders and the effects of the hotline. About Innovations for Maternal, Newborn & Child Health Innovations for Maternal, Newborn & Child Health, an initiative of Concern Worldwide U.S., accelerates the discovery and testing of creative solutions to understand and overcome barriers that prevent essential health services from reaching women and children. Innovations is supported by a multi-year grant from the Bill & Melinda Gates Foundation. For For more information about Innovations for for Maternal, Newborn and & Child Health, visit: www.innovationsformnch.org 5