End-of-Project Technical Report. 13 October 2014

Transcription

1 Measuring the impacts of health facility reinforcement and EID and EPI service integration on HIV testing and immunization services in Southern Province, Zambia End-of-Project Technical Report 13 October 2014 Prepared by: Zambia Ministry of Health Zambia Ministry of Community Development, Mother and Child Health IDinsight Clinton Health Access Initiative Zambia Center for Applied Health Research and Development

2 CONTENTS Acronyms... iii Executive Summary... i Background... i i Evaluation Design... i Results... i Recommendations... ii Background... 1 Study Design and Methods... 3 Study Area... 3 Study Sample and Randomization... 3 Description of Interventions... 3 Data Collection... 5 Pre-Evaluation Activities... 5 Monthly Facility Visits and Administrative Data Sets... 6 Exit Interviews, Focus Groups, and Structured Interviews... 6 Ethics... 7 Data Quality and Monitoring... 7 DBS Data... 7 Maternal Retest Data... 8 Immunization Data... 8 Data Analysis... 9 Exploratory Analysis... 9 Regression Analysis... 9 Secondary Analysis Exit Interviews and Qualitative Analysis Results Infant DBS Testing Differences between Baseline and Intervention Periods Regression Results Time Trends Maternal HIV Retests Differences between Baseline and Intervention Periods Regression Results Time Trends Immunization Differences between Baseline and Intervention Period Regression Results... 19

3 Time Trends Secondary Analyses Age at First DBS Test Time for DBS Sample Processing Exit Surveys Staff Interviews Focus Group Discussions Discussion Interventions Increase HIV Maternal Retesting No Detectable Impact on Immunizations Stock Outs Integration of Services Evaluation Strengths Evaluation Limitations Recommendations Sources Appendix A Timeline of Activities Appendix B Randomization Approach Appendix C Simple Intervention CE model Appendix D Power Calculations Appendix E Additional DBS Regression Results Appendix F Additional Retest Regression Results Appendix G Additional DPT1 Regression Results ii

4 ACRONYMS 3DE Demand Driven Evaluations for Decisions AIDS Acquired Immunodeficiency Syndrome ANC Antenatal Care ART Antiretroviral Treatment ARVs Antiretroviral Drugs BUPIP Boston University PMTCT Integration Programme CHAI Clinton Health Access Initiative CHW Community Health Worker DBS Dried Blood Spot DHO District Health Office DMO District Medical Office DPT1 First dose of Diphtheria, Pertussis, and Tetanus Vaccine EHT Environmental Health Technicians EID Early Infant Diagnosis EPI Expanded Programme on Immunization GEE Generalized Estimating Equation GNI Gross National Income HAHC - Hospital Affiliated Health Center HIA2 Health Information Aggregation Form 2 HIV Human Immunodeficiency Virus IRB Institutional Review Board ITNs Insecticide-treated bednets LGH Livingstone General Hospital MCDMCH Ministry of Community Development, Mother and Child Health MoH Ministry of Health ODK Open Data Kit OPV Oral Polio Vaccine PCR Polymerase Chain Reaction PMTCT Prevention of Mother-to-Child Transmission PMO Provincial Medical Office iii

5 RCT Randomized Controlled Trial PCV Pneumococcal Conjugate Vaccine PNC Postnatal Care SMAG Safe Motherhood Action Group U-5 Under-five UTH University Teaching Hospital WHO World Health Organization ZCAHRD Zambia Center for Applied Health Research and Development iv

6 EXECUTIVE SUMMARY Background Zambia has made significant progress expanding services for mothers living with HIV. However, gaps still remain in the identification of HIV-exposed and HIV-infected infants. Zambia s Ministry of Health (MoH) guidelines dictate that known HIV-exposed babies should be tested twice for HIV status after birth. Although an estimated 81,000 HIV-positive women gave birth in Zambia in 2012, only 48,000 infants received an HIV test by eight weeks of age in this same year. 1 Thus, improving maternal and early infant diagnosis (EID) testing rates is critical to advancing Zambia s performance along the mother-child HIV testing and treatment cascade. Zambia MoH guidelines state that mothers with unknown HIV status or a previously negative status should continue to be retested every three months until their infant reaches 18 months of age. Yet, these retesting rates are similarly low, estimated at less than 20% for facilities with available data. i For the purposes of this technical report, these postpartum maternal HIV rapid tests are referred to as retests. ii Zambia s MoH and Ministry of Community Development, Mother and Child Health (MCDMCH) requested an evaluation to assess two interventions intended to improve maternal HIV retest and EID dried blood spot (DBS) testing rates: The Simple Intervention 1) provided a guaranteed stock of HIV testing commodities and 2) emphasized current MoH guidelines around EID and HIV-testing for mothers. The Comprehensive Intervention included the components of the Simple Intervention and added 1) hands-on operational optimization to integrate EID testing into routine six-week immunization visits and 2) opt-out HIV testing for all mothers with previously negative or unknown HIV status at the six-week under-five (U-5) visit. The goal of the evaluation was to inform Zambia s HIV testing guidelines and assess the potential for integration of HIV services with other health services. Evaluation Design A three-arm cluster randomized evaluation involving 60 health facilities in Southern Province was used to measure the impact of the two interventions on the number of DBS tests, HIV retests and first doses of diphtheria, pertussis, and tetanus immunizations (DPT1) administered. Quantitative findings were complemented by qualitative inquiries to assess client and health staff perceptions toward HIV testing, U-5 activities, and the two interventions. Results This evaluation revealed that the Simple Intervention significantly improved HIV retesting for mothers but had no detectable effect on DBS testing rates for infants. The Comprehensive Intervention also positively impacted maternal HIV retesting rates but did not have a clear effect on DBS testing rates outside of two outlying facilities (one in the Control group, one in the i These data come from ZCAHRD s 2013 BUPIP programme data and is only for Southern Province. National figures are not available for this outcome. ii This group could contain some mothers that are tested for the first time postnatally, although the wide majority of mothers in this group are being retested after having been tested at ANC i

7 Comprehensive Intervention group). Intensification of HIV testing at the six week U-5 visit via the Simple and Comprehensive Interventions did not negatively impact overall immunization rates. Exit surveys and focus groups revealed that mothers viewed additional HIV testing for themselves and their infants positively with few mothers expressing reservations. Interviews with Comprehensive Intervention facility staff revealed that staff viewed the intervention positively. Specific findings include: No statistically significant effects on DBS uptake were detected for either intervention. The monthly number of DBS tests per facility was 1.01 (P-value = 0.14) tests greater in Simple Intervention facilities versus Control facilities, and 1.54 tests (P-value = 0.12) greater in the Comprehensive Intervention facilities versus Control facilities. However, these results were predominantly driven by two large facilities, and subsequent regressions that diminished the influence of outliers estimated small, non-significant effects for both interventions on DBS uptake. Statistically significant impacts on retests were detected for both interventions. The change in monthly number of retests per facility was tests (P-value < 0.01) greater in Simple Intervention facilities versus Control facilities, and tests (P-value = 0.06) greater in Comprehensive Intervention facilities versus Control facilities. No statistically significant impacts on DPT1 doses were detected for either intervention. The change in monthly number of DPT1 doses per facility was 0.86 (P-value = 0.53) doses smaller in Simple Intervention facilities versus Control facilities, and 0.51 (P-value = 0.78) doses smaller in Simple and Comprehensive Intervention facilities versus Control facilities. Recommendations Based on the findings and observations from this evaluation, the study team offers the following recommendations: This evaluation provides evidence that improvements to Zambia s HIV test kit supply chain can yield important health benefits and should be prioritized. Integrating EID with EPI services can be explored as a feasible strategy for improving HIV maternal retest rates without negatively affecting immunization uptake. However, integration should only be pursued if sustained monitoring and operational support is possible to support implementation. Simple, low-cost steps to improve coordination between EPI and HIV policy counterparts to streamline health services to facilitate integration can be explored. These include: o Updating HIV fields on U-5 cards to match existing guidelines. Currently there is no place to record postpartum retest results or the second DBS test result. o Integrating HIV services into other U-5 data collection tools such as tally sheets and increase size of facility monitoring tools to improve legibility. o Aligning postpartum retesting time periods with the immunization schedule rather than time since last test. Opportunities include: 6 weeks (DPT, OPV, PCV, RV), 9 months (Measles 1), and 18 months (Measles 2). o Joint messaging and sensitization between PMTCT and immunization programming. ii

8 BACKGROUND Significant progress has been made over the past decade in reducing the global burden of HIV/AIDS. Improved access to life-saving treatments and coordinated prevention efforts have contributed to a 29% decrease in AIDS-related mortality since 2005 and a 33% decrease in the number of new HIV infections since Gains have been greatest among children, who primarily acquire HIV through vertical (mother-to-child) transmission, with a 52% decrease in the number of new HIV infections among children between 2001 and Advances in prevention of mother-to-child transmission of HIV (PMTCT) have dramatically shifted the outlook for HIV-infected mothers and their newborns. Currently available prophylactic antiretroviral drugs (ARVs) can reduce mother-to-child transmission to less than 5%. 4 As a result, vertical HIV transmission and HIV-related mortality among pregnant women and children has virtually been eliminated in high-income countries. 4, 5 Despite these gains, children are still vulnerable to HIV, particularly in low- and middle-income countries. Roughly 230,000 of the estimated 260,000 new HIV infections among children each year occur in Sub-Saharan Africa. 2 An integral part of expanding PMTCT is improving early infant diagnosis (EID) of HIV. Early identification of HIV-positive infants is critical to improving chances of survival. It is estimated that over 50% of HIV-positive infants will not survive through their second year. 6 Up to 76% of these deaths can be avoided with early identification and initiation of treatment before 12 weeks of age. 7 However, only 39% of children in low- and middle-income countries have access to HIV testing services within the first two months of life. 5 Zambia is one of 22 countries in the world with the highest estimated numbers of pregnant women living with HIV. Recently, it has made strong progress in expanding PMTCT services, with 94% of women undergoing HIV testing at antenatal care (ANC) and 95% of HIV-positive women receiving antiretroviral therapy (ART) for PMTCT in Unfortunately, these successes have not carried over into the postnatal period. Current MoH guidelines stipulate that HIV-exposed iii infants should be tested twice using the dried blood spot test (DBS) test, first at six weeks after birth and again at six months. With an estimated 80,000-90,000 HIV-positive women giving birth in Zambia each year, an equal number of newborns should receive a DNA polymerase chain reaction (PCR) test within the first two months of life. 8 However, in 2012, only 45,000 infants were tested by eight weeks of age around 50% of the national target. 1 Similarly, MoH guidelines stipulate that mothers with unknown HIV status should be retested every three months while their infants are still breastfeeding (and, therefore, still at risk of transmission). The Zambia Centre for Applied Health Research and Development (ZCAHRD), an affiliate of Boston University in the United States, has worked with the Southern Province Provincial Medical Office (PMO) to test postpartum mothers at six weeks, six months, nine months, 12 months, and 18 months. Despite this support, fewer than 20% of mothers who should be tested actually receive a postpartum retest. iv Many low-income countries have achieved impressive childhood immunization rates. As a result, the World Health Organization (WHO) and UNICEF recommend routine childhood immunization services iii An infant born to a mother living with HIV. iv Calculated using the BUPIP data from January to August 2013; Mothers who should receive tests are defined as those attending immunizations who are not known to be HIV positive. 1

9 to be a platform for delivery of other maternal and child primary healthcare interventions. 9 Studies have demonstrated that services such as provision of vitamin A supplementation, deworming medications, hygiene activities, growth monitoring, and insecticide-treated bed nets (ITNs) can be successfully integrated with immunization services Zambia currently integrates many of these activities at U-5 clinic days rather than running separate immunization days. At U-5, facilities regularly provide immunizations, growth monitoring, and nutrition counseling. Occasionally, they provide supplemental services such as distribution of ITNs, postnatal care (PNC), and some HIV services. The WHO Expanded Programme on Immunizations (EPI) guidelines schedule routine immunizations at birth, six weeks, ten weeks, 14 weeks and nine months. 13 The six week visit aligns with the first recommended EID testing point and the Zambia MoH s postpartum retesting goals. The coverage of first dose Diphtheria, Pertussis, and Tetanus (DPT1) vaccinations in Zambia was estimated to be 83% in , suggesting that the six-week visit could be a particularly useful way of reaching a high percentage of mother-infant pairs. This has prompted Zambian health officials to explore the possibility of utilizing routine U-5 services as an opportunity to boost Zambia s EID and maternal retesting rates. Recent qualitative and observational studies suggest that integrating universal opt-out maternal and infant HIV testing with routine immunizations can be an acceptable and effective approach to improving maternal and infant diagnosis However, there is little rigorous evidence that aims to quantify the impact of integration of HIV and U-5 services on HIV testing and immunization rates. This evaluation examined the impact of two interventions on levels of HIV testing and immunization uptake. The Simple Intervention focused on preventing supply stock outs of HIV testing commodities and reinforcing current MoH guidelines around EID and HIV-testing for mothers. The Comprehensive Intervention coupled the activities in the Simple Intervention with 1) an intentional operational optimization and integration of EID testing into routine six-week U-5 visits, and 2) opt-out rapid HIV testing for all mothers with previously negative or unknown HIV status at the six-week U-5 visit. The specific study aims were to: 2 1. Estimate the impact of both the Simple Intervention and Comprehensive Intervention on monthly averages of the following HIV-related outcomes: a. The total number of DBS tests administered i. The number of first DBS tests administered ii. The number of second DBS tests administered b. The total number of postpartum retests administered for HIV negative or status unknown mothers at all U-5 visits i. The number of postpartum retests administered at the six-week immunization visit 2. Estimate the impact of the Comprehensive Intervention on the monthly average number of DPT1 doses administered compared to the Simple Intervention and Control, and the joint impact of the Simple Intervention and Comprehensive Intervention on the monthly average number of DPT1 doses compared to Control. 3. Examine attitudes and perceptions of mothers and facility staff on HIV-testing and integration of services. This evaluation was launched under Demand Driven Evaluations for Decisions (3DE), an initiative designed to use impact evaluations in a demand-driven and efficient manner to generate evidence for catalyzing at-scale implementation of cost-effective and impactful policies.

10 STUDY DESIGN AND METHODS This evaluation was a cluster randomized controlled trial (RCT) randomized at the health facility level in the Southern Province of Zambia. The Comprehensive Intervention was piloted in assigned facilities from August 12 th to September 30 th, 2013, with full implementation continuing through the intervention period of October 1, 2013 to March 31, Appendix A contains the full evaluation timeline. Study Area This evaluation took place in Livingstone, Monze, and Choma districts in the Southern Province of Zambia. Located in southern Africa, Zambia has a population of more than 13 million people and one of the highest population growth rates in Sub-Saharan Africa. 19 Zambia s gross national income (GNI) per capita is $1,350, with over 60% of its population living below the national poverty line. Zambians life expectancy at birth is 57 years. 20 Zambia s Southern Province is comprised of thirteen districts. v It is home to 1.5 million people with 75% of its population living in rural settings and 48% of its population below the age of In 2007, it was estimated that 14.5% of the women and men age who received an HIV test in Southern Province were found to be positive. 21 The selection of Southern Province was made in conjunction with MoH officials. The evaluation districts of Choma, Livingstone, and Monze were selected based on geographic dispersion, urban / rural characteristics, current HIV prevalence rates, and the absence of conflicting research projects. Study Sample and Randomization The evaluation took place in 60 government-run health facilities providing PMTCT and EID services, with 20 facilities in each evaluation arm. These facilities are supported by ZCAHRD, who provide training for HIV service provision and staff support, including the training and hiring of one PMTCT lay counselor per facility. ZCAHRD also served as a partner for this evaluation, providing guidance in the development of the research protocol, access to baseline data, and facilitating meetings with provincial, district, and facility staff. The final sample was achieved using a stratified and constrained randomization approach (see Appendix B). Statistical power calculations were based on the immunization outcome variable, as this was the outcome where the policy relevant detectable effect size was smallest. A description of power calculations is found in Appendix D. Description of Interventions The evaluation had three study arms: 1. Control facilities continued status quo activities and provided HIV and immunization services as they typically would. 2. The Simple Intervention targeted two potential causes of low EID rates: (1) testing supply stock outs and (2) poor understanding of existing MoH testing requirements and guidelines. The two components of this intervention were: Reinforcement of HIV testing supplies The research team worked with facility staff to ensure that orders for necessary HIV testing supplies were placed on time and in sufficient quantity. In the event of a generalized stock out of HIV testing supplies at the provincial or v Prior to 2013, Southern Province had 10 districts; three districts were added in

11 district level, the evaluation team provided facilities with an additional outside supply. This support was limited to HIV tests and was not extended to vaccines. Reinforcement of current MoH guidelines The research team facilitated one-time visits to health facilities by District Health Office (DHO) officials to meet with relevant health staff at the beginning of the evaluation. The purpose of these meetings was to remind providers of existing MoH HIV testing guidelines and to reinforce that improving EID and postpartum testing is a MoH priority. Additionally, an HIV Activity Sheet created by the evaluation team was introduced for use at all U-5 sessions to help facility staff track postpartum retesting, DBS testing, and distribution of medication. This sheet was designed to centralize HIV services data collection for the facility at U-5 and provide a reminder of what services to offer. 3. The Comprehensive Intervention included all components in the Simple Intervention, but additionally operationalized the integration of universal, opt-out HIV screening and testing for mothers and infants with in-facility (static) U-5 services. These components were not included in U-5 services that are delivered outside of the facility in the community (outreach services). The unique components of this intervention were: Communication of the intervention: Health facility staff were instructed to communicate all aspects of the Comprehensive Intervention during ANC appointments, in-facility deliveries, and PNC visits. Additionally, at six week immunization visits, mothers and caregivers received group counseling on opt-out HIV screening service and the importance of regular HIV screening for mother and child health. Finally, the research team engaged Safe Motherhood Action Groups (SMAGs), community health workers (CHWs), and active neighborhood health committee members to further increase awareness in facility catchment areas. These community members completed low-touch community sensitization over changes that would be made to U-5 services. Operations optimization and integration of testing services: This component optimized the patient flow during U-5 services to include new testing procedures and to more closely integrate EID and EPI services, while mitigating increases in staff workload and patient waiting times (Figure 1). The evaluation team worked with facility staff to identify efficient allocations of staff and tailor the order of services. A key component of the operational optimization was a new patient triaging approach that sorted patient U-5 cards into bins and used separate queues for three types of patients: 1) first visit infants (six weeks), 2) second visit or later infants who required immunizations and 3) infants who were scheduled to only receive growth monitoring. The six week visit was emphasized as a key patient touch point, as this is when both the first round of childhood immunizations (DPT, oral polio vaccine (OPV), pneumococcal conjugate vaccine (PCV), and rotavirus vaccine) and first maternal postpartum retest should take place. Additionally, DBS testing at six weeks allows facility staff to use the ten week or fourteen week immunization visit to deliver DBS test results. Implementation of universal, opt-out screening of HIV for both mothers and infants: Facility staff were instructed to examine the maternal HIV status on the U-5 or ANC card for all infants attending their first U-5 visit and do the following: o If marked Confirmed Exposed, the health care worker conducted the DBS test on the infant only. o If marked, Mother Status Unknown or Confirmed Not Exposed the mother or caregiver was asked if the mother had ever tested HIV-positive. If yes, a DBS was done on the infant only. If no, the mother was offered an HIV antibody test in an opt-out manner. 4

12 Figure 1: Overview of Comprehensive Integration Approach In all Simple and Comprehensive Intervention facilities, normal antibody testing procedures were followed using the Determine TM Antibody Tests and Unigold TM confirmatory test along with all standard counseling messaging per existing practice. Infants of mothers who tested HIV-positive should have then received a DNA PCR test the same day. These procedures reinforce standard HIV testing policies set by the Zambian MoH and WHO. Data Collection Pre-Evaluation Activities All facilities included in the evaluation were provided with larger U-5 tally sheets to improve data quality. vi A short training was held with representatives from all facilities to introduce the research team, to explain what data would be collected, and, for all intervention facilities, to deliver the guidelines reinforcement portion of the evaluation. The DHO was represented at each meeting to complete the guidelines reinforcement. Separate meetings were held for each intervention group in each district. All Comprehensive Intervention facilities received an initial onsite workshop to introduce the components of the intervention and do a full walk-through of the new patient flow procedures. The intervention was piloted for six weeks between August 12th and September 30 th, During this period, each Comprehensive Intervention facility was visited at least twice with additional supervisory visits as needed to troubleshoot difficulties before the start of data collection. vi Previous tally sheets had extremely small printing, making them difficult to use. All data collection fields on these forms and layout remained the same. 5

13 Monthly Facility Visits and Administrative Data Sets Research team members conducted monthly visits to all 60 sample facilities to collect data from facility registers and evaluation team-supplied data sheets ( HIV Activity Sheets and larger U-5 tally sheets). All data were either used for data analysis or for validation of the main source of data. Since facility register data could not leave the facility, data were collected by evaluation staff using Open Data Kit (ODK) surveys programmed on Android mobile phones. The data from evaluation teamsupplied sheets were collected and transported to the IDinsight office for data entry and secure storage. During these monthly data collection visits, the evaluation team also briefly interviewed facility staff to discuss any staffing changes and to take stock of HIV testing and immunization supplies. Evaluation staff also collected facility-level data from several additional administrative data sources, including historical data on immunizations and ANC data from DHOs and the MoCDMCH, DNA PCR laboratories for DBS at both University Teaching Hospital (UTH) in Lusaka and Livingstone General Hospital (LGH) in Livingstone, and baseline HIV testing records from ZCAHRD. Exit Interviews, Focus Groups, and Structured Interviews Quantitative and qualitative methods were used to assess and to compare perceptions of HIV testing as well as of the interventions among mothers and facility staff. Exit surveys were conducted with mothers and other caregivers attending under-five services at 51 out of 60 facilities using convenience sampling. vii Evaluation staff identified the first visit mothers and other caregivers and invited them to complete the exit interview. While facility staff helped direct mothers and other caregivers to the evaluation staff after delivery of U-5 services, facility staff members were not informed on the specific topic of the survey, and interviews were not conducted in front of facility staff. All mothers and other caregivers who were invited to participate in the exit survey were informed that the survey was separate from the services that they received. Informed consent procedures were followed with each participant. Mothers and other caregivers were asked about U- 5 services, as well as any potential impact of prior knowledge of HIV testing services on the likelihood of attending U-5 services. Focus groups were conducted in catchment area communities of eight study facilities in the Simple or Comprehensive Intervention arms to better understand why mothers do or do not attend U-5 services and their perceptions of the changes at U-5 clinic regarding HIV testing. These facilities were purposively sampled across intervention arms to achieve a mix of urban and rural settings and large and small facilities. In each location, CHWs identified women who had successfully delivered a baby between September 1 st, 2013 and February 1 st, 2014, since these babies would have been eligible for six week vaccination within the intervention period. Mothers who lived in an area that provided both static and outreach U-5 services were divided into groups for the type of U-5 sessions they attended. viii All focus groups were conducted in the local language of Tonga. vii The nine facilities that were not included for exit interviews typically only had U-5 once per month and had fewer mothers per U-5 session. Some of these facilities were visited during a scheduled U-5 session, but there were no first visit mothers to be interviewed. viii Focus group discussions were also planned to include groups of women who had not attended U-5 services at all. However, these women could not be identified in the communities. Evaluation staff took the following steps to try to identify them: 1) CHWs were asked to identify anyone within their communities who had not attended U-5 services, including inquiring with other community leaders and groups; 2) Evaluation staff asked mothers in each focus group if they knew of other women in the community who had not attended U-5 services. 6

14 Finally, structured interviews were conducted with health facility staff from ten Simple and ten Comprehensive Intervention facilities during their final data collection visit to gain more insight into staff perspectives of the intervention, including perceived benefits and challenges. Ethics This evaluation was approved by the ERES Converge Ethics Review Board in Lusaka, Zambia and the Boston University Institutional Review Board (IRB) in Boston, Massachusetts, United States. Additionally, the evaluation received formal approvals from Zambia s MoH, MoCDMCH, and the District Health Offices of Choma, Livingstone, and Monze. Data Collection, Quality, and Monitoring The primary data sources used in this evaluation were administrative and generated by non-research staff members who worked at health facilities, districts, and hospitals in low resource settings as part of normal day-to-day operations (Table 1). To mitigate potential inaccuracies in these data sources, the evaluation team cross-verified data whenever possible. Table 1. Summary of Data Sources Used for Analysis and Quality Control Outcome Primary Data Quality Control Sources Source DBS Testing UTH and LGH PCR Lab Databases DBS Tracking Register DBS Lab Requisition Book Interviews at clinic of mothers of infants receiving 2 nd / 3 rd series of immunizations Interviews of mothers in communities Postpartum Retesting BUPIP Monthly Report HIV Activity Sheets (Simple and Comprehensive Intervention facilities) VCT / General Counseling Register Immunizations Under-5 Tally Sheets HIA2 Monthly Report to Districts Interviews at clinic of mothers of infants receiving 2 nd / 3 rd series of immunizations Interviews of mothers in communities The three primary outcomes included in the analysis were 1) The average monthly number of DBS tests administered per facility per month; 2) the average monthly number of maternal postpartum retests administered per facility per month; and 3) the average monthly number of DPT1 doses administered per facility per month. These outcomes were measured as follows: Average Monthly Number of DBS Tests This outcome was measured using the number of DBS tests administered for each facility for each month from January 2012 through March The primary data source for this outcome was the DBS lab database that received the samples collected by each facility. A DBS test was defined as administered if the sample arrived at the lab, regardless of the result of the test. Therefore, samples with indeterminate results were included, but samples that were taken but did not make it to the lab were not. These data were compared to DBS test figures from the facility. At the facility, the numbers of first DBS tests and second DBS tests were collected from the DBS tracking register and the total 7

15 aggregate number of DBS tests was collected from the lab requisition register. All three data sources were highly correlated with an average difference of less than 5%. Additionally, DBS records from HIV-exposed infants were checked between U-5 cards inspected during data verification interviews and the DBS tracking register at the facility. Out of 325 verification interviews, 30 U-5 cards indicated that a DBS test had been completed. Of these 30 tests, 87% (26 tests) were verified by the research team with three of the four unverified tests coming from a single facility. Average Monthly Number of Maternal Retests The average monthly number of maternal retests conducted was measured using data from the Boston University PMTCT Integration Programme (BUPIP) monthly report. This outcome included all retests that were recorded, regardless of the test result. These data were verified with the evaluation HIV activity sheets. Verification was only possible for intervention facilities, as HIV activity sheets were not introduced in control facilities. Overall, the two sources had a correlation of 0.75 for the total number of tests done per month per facility. The average total number of tests per month per facility in intervention sources was very similar with the BU data source averaging 26.0 tests per month and the HIV activity sheet data source averaging 24.4 tests per month. ix Average Monthly Number of DPT1 Doses This outcome was measured using monthly figures of the number of DPT1 doses for each month from January 2012 to March 2014 that the facility reports to the DHO each month. These estimates included both outreach and static figures, since the facility does not disaggregate these. Figures were compared to the number of DPT1 doses that evaluation enumerators collected from each facility by counting the number of DPT1 immunizations recorded on each U-5 tally sheet for a given month. The research team compared these two figures and recollected tally sheet data if the two data sources differed by 10% or more. This process occurred in approximately 52% of cases. The research team then reconciled the resurvey data using pre-established data cleaning rules. DPT1 data were also verified using a data verification interview conducted with mothers or other caregivers who brought their infant to the first under-5 visit during the evaluation period. Of the 341 interviews conducted, 77% were conducted at facilities with mothers or other caregivers who had returned for 2 nd and 3 rd immunizations with their infants, while 23% were conducted in communities with mothers who were identified by CHWs. During each interview, the enumerator recorded the date of the DPT1 immunization, infant HIV status, and date of DBS test, if applicable, from the infant s U-5 card. These data collected were then compared to facility registers to check if the record existed at the facility and if the dates matched. Overall, DPT1 data for 3% of mothers (325 out of 10,435) were double-checked through data verification interviews, achieving 88% agreement. x ix There are several reasons for possible discrepancies. First, the activity sheets were a newly introduced tool and were inconsistently filled out during the beginning of the evaluation. Second, postpartum retests done at times other than U-5, although likely uncommon, may not have been recorded on HIV activity sheets. Overall, the BUPIP source was favored for its breadth in including all facilities (including control) as well as being the source of the baseline data. x There were direct matches for 62% of the data (202 out of 325 cases). Additionally, 26% of the data contained matched dates between the U-5 cards at the U-5 tally sheets but did not have a matching U-5 number. Often there are cases where the U-5 number is recorded as a dash on the tally sheet for first visits if the card number has not yet been assigned (as can be the case during first immunization visits). 8

16 Covariates Facility-level characteristics were primarily collected during the baseline period. A facility was characterized as rural if it was more than ten kilometers along the road away from the DHO, based on administrative reports. The average baseline number of mothers attending ANC was collected from monthly reports that facilities provide to the DHO. Additionally, the number of mothers appearing for 1 st ANC visit each month was also collected from these reports. The number of HIV positive mothers anticipated at U-5 was calculated by multiplying the average baseline number of mothers attending first ANC visit by the % of women attending ANC that are living with HIV, as supplied by the BUPIP database. Finally, the actual distance between the facility and the DHO was measured using GPS coordinates. STATISTICAL METHODS All major analyses were conducted using facility-level comparisons between Control, Simple Intervention, and Comprehensive Intervention facilities. Exploratory Analysis The main outcomes and covariates were examined visually to understand data structures, distributions, and trends over time. Line graphs of the three main outcomes (number of DBS tests, number of maternal retests, and number of DPT1 doses) over time were constructed using three month running averages for smoothing. Additionally, facility-specific baseline and intervention period data were compared for all outcome variables to examine trends and to identify potential outliers. Regression Analysis Data for the main outcomes consisted of repeated monthly measures for each facility from both the baseline and the intervention period. Multivariate linear regression models accounting for clustering at the facility level were used. Standard errors were estimated using bootstrapping methods. The isolated effect of each intervention on each of the primary outcomes was estimated by including interaction terms between a factor variable for the intervention groups and the time period (baseline versus intervention period). Additional covariates were included in the regression models to control for confounders and to increase precision. Covariates were chosen using Akaike Information Criterion (AIC). The regression model for the average number of DBS tests administered also adjusted for district and urban/rural characteristics since these are the variables on which the initial sample was stratified, as well as time (using splines to account for linear trends), a binary indicator for a period of a national stockout, the number of DPT1 doses provided in the same month, and the number of mothers living with HIV anticipated in that month. Because the DBS testing results were particularly affected by outliers, the primary regression for this outcome used a log transformation of the outcome. The regression model for the average number of maternal retests administered adjusted for district and urban/rural characteristics, time, the monthly average number of mothers going to first ANC, the distance between the facility and the DHO, and the number of DPT1 doses provided in the same month. Finally, the regression model for the average number of DPT1 doses provided each month adjusted for the average number of mothers going to first ANC, time, and the average distance from the DHO. Since the Comprehensive intervention alone was operationally integrated with immunization services, the analysis first assessed if the Comprehensive Intervention resulted in a difference in the 9

17 change between baseline and the intervention period compared to the combined change experienced in both the Control and Simple Intervention facilities. It then assessed if the combined intervention arms experienced a statistically significantly different change than the Control facilities. Since these data were collected as repeated measures from the same facility and the outcomes were measured as count data, we conducted the same regression analyses on the outcomes using generalized estimating equations (GEEs) with a negative binomial family, log link function, and exchangeable working correlation matrix as a sensitivity analysis. The multi-level approach that this model uses accounts for the high correlation between outcome measures from the same facility. The negative binomial distribution was used to account for the non-normal distribution of the data and the fact that the value of the outcome cannot be negative. GEE results are presented in appendices. Secondary Analysis Secondary analyses were conducted for DBS outcomes at the patient level, rather than the facility level, to test for additional evidence of intervention effects. We used linear regression to test for differences in the age of an infant at their first test between intervention arms. We also created several graphs that demonstrate bottlenecks across intervention arms, in the delivery of samples to labs and the processing once they arrive. Exit Interviews and Qualitative Analysis Exit interview data used Pearson s chi-squared test to test for differences in proportions across the three evaluation arms. Facility staff interviews and focus group discussions were analyzed using a thematic analysis approach. Each focus group discussion was coded according to common themes by two separate evaluation staff members. xi All statistical analyses were done using Stata version 12 (Stata Corp LP, College Station, Texas, US). RESULTS Using baseline data that were directly collected from the facilities, the monthly averages of DBS tests, DPT1 doses administered, first ANC visits, six week retests, and total retests were similar across the evaluation groups. xi Inter-rater agreement was calculated using the kappa statistic with an expected percent agreement of 25%. The overall kappa was 0.63, meaning there was 63% agreement beyond what is expected by chance alone. The kappa for themes that were specifically related to HIV testing and the integration of services was The majority of disagreement was due to the fact that one coder did not include as much as information as the other, as opposed to the two coders reaching dissonant conclusions. Therefore, by using both sets of codes for analysis, a fuller representation of the focus group discussions was achieved. 10

18 Table 2. Summary of Sample by Intervention Arm Control (N=20) Simple (N=20) Comprehensive (N=20) N (%) N (%) N (%) Total N Facility Characteristics No ART 14 (70%) 15 (75%) 15 (75%) 44 Static ART Site 2 (10%) 3 (15%) 4 (20%) 9 Mobile ART Site 4 (20%) 2 (10%) 1 (5%) 7 Strata Choma Rural 8 (40%) 8 (40%) 8 (40%) 24 Choma Urban 2 (10%) 2 (10%) 2 (10%) 6 Livingstone Urban 3 (15%) 3 (15%) 3 (15%) 9 Monze Rural 6 (30%) 6 (30%) 6 (30%) 18 Monze Urban 1 (5%) 1 (5%) 1 (5%) 3 Mean (SD) Mean (SD) Mean (SD) P-value * Baseline Monthly Averages DBS tests 3.97 (7.05) 3.74 (6.98) 4.30 (8.08) 0.89 DPT1 doses (23.43) (17.42) (26.75) 0.99 First ANC Visits (25.04) (21.44) 29.9 (21.44) Week Retests 3.29 (3.73) 1.18 (1.81) 3.04 (5.31) 0.14 Total Retests (15.64) 6.74 (9.38) (27.16) 0.19 * The intervention arms were regressed as indicator variables on each outcome. After the regression, an F-test was used to test for equality between the three evaluation arms. At the time of sampling, all samples with p-values of less than.90 for DBS, DPT, and ANC averages were removed from consideration. Since that time, more complete data is available, explaining the 0.89 p-value for the DBS tests at the final sample. Infant DBS Testing The change in the average number of infant DBS tests administered per month from baseline to the intervention period was examined and compared between evaluation arms. The baseline period for the DBS analysis started January 1 st, 2012 (month = 0) and ended July 31 st, 2013 (month = 19). During this period, a total of 4,529 DBS samples were collected. Data from August and September 2013 were excluded from the analysis as this was when the interventions were being piloted. Differences between Baseline and Intervention Periods The average number of tests in the study sample at baseline was 4.0 tests per month. The average number of DBS tests administered per month per facility increased by 0.43 tests over baseline in Simple Intervention facilities and by 1.19 tests over baseline in Comprehensive Intervention facilities. The average number of DBS tests administered per month decreased by 0.49 tests in Control facilities (Table 3). While the Control group s monthly DBS testing numbers decreased between the baseline and the intervention period (-0.49), the Simple Intervention group s monthly testing numbers increased between baseline and the intervention period (0.43), with a total difference of 0.92 more tests per month over time as compared to the Control group. Similarly, the difference in the average monthly DBS testing numbers between the baseline and the intervention periods among facilities in the Comprehensive Intervention group was 1.78 tests greater than the difference experienced among facilities in the Control group. 11

19 Table 3. Average Number of DBS Tests Administered per Month per Facility, by Group Control Simple Comprehensive N=20 N=20 N=20 Mean (SD) Mean (SD) Mean (SD) Baseline Average 3.96 (5.31) 3.75 (6.19) 4.30 (6.51) Intervention Period Average 3.48 (2.75) 4.18 (6.18) 5.49 (8.27) Difference (3.44) 0.43 (2.14) 1.19 (3.69) The average number of DBS tests during baseline was plotted against the average number of DBS tests during the intervention period for each facility, with the size of the circle weighted by the expected number of HIV-positive mothers to appear for U-5 visits (Figure ). Figure suggests that a few large Comprehensive Intervention and Control facilities are driving a large proportion of the overall changes observed in the aggregate baseline and intervention periods. Figure 2. Baseline versus Intervention Period Monthly Averages of DBS Tests for All Facilities Regression Results The linear regression using a log-transformed outcome of the average monthly number of DBS tests revealed that Simple Intervention resulted in a 17% [90% CI: -7%, 46%, P-value = 0.26] greater change in average number of monthly DBS tests over time as compared to the change experienced in Control facilities (Table 4). The Comprehensive Intervention experienced a 10% [90% CI: -10%, 12

20 36%, P-value = 0.43] greater change in the average monthly number of DBS tests between baseline and the intervention period as compared to the change experienced in Control facilities. Neither of these differences was statistically significant. Table 4. Linear Regression Results of Intervention Arm on Logged Outcome (Number of DBS Tests per Facility per Month) Covariates e B P-value [90% CI] Intervention Arm Control Ref Simple [0.72,1.08] Comprehensive [0.83,1.21] Time Period Baseline Ref Intervention period [0.59,1.00] Intervention Impacts: Simple v Control [0.93,1.46] Comprehensive v Control [0.90,1.36] # of Monthly DPT1 Immunizations [1.00,1.01] District & Urban/Rural Stratum Urban Choma Ref Rural Choma [0.69,1.49] Urban Livingstone [1.12,3.07] Urban Monze [0.88,3.17] Rural Monze [0.97,2.08] Time [1.01,1.12] Time [0.90,1.01] Time [0.89,1.03] Time [0.99,1.29] Time [0.98,1.07] Time (Intervention Pd) [0.97,1.05] Known National Stock out of DBS Kits [0.79,1.10] Number of HIV+ Mothers Anticipated < 0.01 [1.06,1.19] Constant [0.73,1.64] Additional linear regressions using a log transformation were run to assess intervention impacts on first DBS tests and second DBS tests. No significant intervention effects were detected results are presented in Appendix E. Two additional post hoc linear regressions without a log transformation were run (Table 5). One regression included all of the facilities, and the other excluded two large outlier facilities (one Comprehensive Intervention facility and one Control facility). The regression that included all of the facilities estimated that the Simple Intervention resulted in an additional 1.01 tests per facility per month (90% CI: -0.11, 2.13, P-value = 0.14) over the change experienced in the Control group, and the Comprehensive Intervention resulted in an additional 1.54 tests (90% CI: -0.08, 3.16, P-value = 0.12) over the change experienced in the Control group. The regression that excluded the outliers 13

21 estimated a much more modest impact for both interventions, indicating that results seen in the aggregate difference-in-difference measures are largely driven by the two outlier facilities. Table 5: Multivariate Linear Regression Results With and Without Outliers for Number of DBS Tests per Facility per Month Covariates Full Model Excludes Outliers Coeff P-value [90% CI] Coeff P-value [90% CI] Intervention Arm Control Ref Ref Simple [-1.19,0.88] [-1.11,0.29] Comprehensive [-0.38,1.80] [0.10,1.88] Time Period Baseline Ref Ref Intervention period [-2.89,-0.12] [-2.95,0.18] Intervention Impacts: Simple v Control [-0.11,2.13] [-0.72,1.33] Comprehensive v Control [-0.08,3.16] [-0.49,1.01] # of Monthly DPT1 Immunizations < 0.01 [0.02,0.06] < 0.01 [0.01,0.05] District & Urban/Rural Stratum Urban Choma Ref Ref Rural Choma [-0.72,2.21] [-1.50,0.87] Urban Livingstone [0.63,5.14] [-1.06,2.37] Urban Monze [1.00,9.10] [-1.81,4.74] Rural Monze [0.44,3.11] [-0.39,1.78] Time [-0.01,0.45] [-0.10,0.39] Time [-0.36,0.21] [-0.39,0.28] Time [-0.86,0.10] [-0.41,0.17] Time [-0.02,1.78] [-0.24,0.84] Time [-0.46,0.27] [-0.24,0.35] Time (Intervention Pd) [-0.13,0.24] [0.05,0.25] Known National Stock out of DBS Kits [-1.53,0.72] [-1.82,-0.04] Number of HIV+ Mothers Anticipated < 0.01 [0.81,1.32] < 0.01 [1.09,1.66] Constant [-4.61,-0.76] [-3.03,-0.03] Finally, a GEE model is presented in Appendix E. The GEE results are similar to those obtained from the linear regression with the log transformation. Time Trends A line graph using a three month running average of the number of DBS test kits administered reflects the trends over the baseline and the intervention periods (Figure 3). There is an observable drop in DBS testing between time = 9 (September 2012) and time = 13 (January 2013) as the result of a national shortage of DBS test kits. This temporary drop is controlled for in the analysis. As the regression results also demonstrate, the Simple and Comprehensive Intervention arms experienced an increase in the number of DBS tests administered during the intervention period, though the trend observed in the Comprehensive Intervention line was largely a result of one facility. Similarly, the decrease that is seen in the Control arm is also heavily influenced by one facility. 14