Supplementary webappendix This webappendix formed part of the original submission and has been peer reviewed. We post it as supplied by the authors. Supplement to: Basinga P, Gertler PJ, Binagwaho A, Soucat ALB, Sturdy J, Vermeersch CMJ. Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation. Lancet 2011; 377: 1421 28.
Web Appendix 1: Outcome Measures, Covariates and Statistical Power Outcome Measures Maternal health services utilization: The outcome measures include an indicator for any prenatal care utilization, an indicator for completing 4+ prenatal care visits, and an indicator for institutional delivery for women who gave birth in the 18 months prior to the survey. Quality of prenatal care: We use two measures of prenatal care quality. First, we measure whether women reported receiving the tetanus vaccine during a prenatal care visit. Note that this corresponds to one of the payment indicators listed in Box 1. Second, we construct an index of the process quality of prenatal care by computing the share of actual clinical content items delivered during a prenatal care consultation to the items that should compose a typical prenatal consultation as recommended in the Rwandan clinical practice guidelines. i This measure has been used extensively in the literature to measure quality. ii,iii,iv,v,vi,vii,viii The items in the measure cover medical history questions, physical examinations, lab tests and follow-up procedures (Box 3). The Cronbach alpha scale of reliability for the 38 item score is 0.78, indicating satisfactory internal consistency. In the impact analysis, we standardize the score by subtracting out the baseline mean and dividing by the baseline standard deviation. The unit of measurement is interpreted as standard deviations in quality. The prenatal quality score was computed at the individual patient level using two samples. First, enumerators conducted exit interviews with women who visited the facility for prenatal care on the day of the facility survey. Second, the same information was collected in the household survey from women who gave birth in the last 18 months and received prenatal care from the facility. In the analysis, we combined the exit interview and household survey data for first prenatal care visits to assess the impact of P4P on prenatal care quality. Child Preventive Care: Following common practice and recommendations in the Living Standards Measurement Surveys, ix we use an indicator of whether the mother reported taking her child for a preventive care visit in the four weeks prior to the household survey to measure preventive child utilization. Child preventive services cover immunization, vitamin A, distribution of mosquito nets and child growth monitoring with referral of malnourished children to higher levels of care for treatment. We analyze preventive visits separately for children 0-23 and 24-59 months old as the younger group is expected to have more visits than the older group. We also examine the impact of P4P on whether full immunization was achieved on time, measured by an indicator variable for 12-23 month olds, coded 1 if the child received all vaccines required by the national protocol and 0 otherwise. Vaccination status was assessed based on the vaccination card. Less than 4 percent of the mothers could not produce the child s vaccination card and those observations were dropped from the analysis. Covariates In the maternal care utilization models, we control for individual-level characteristics (age, education, partner lives in household, number of pregnancies) and household characteristics (health insurance, number of household members, distance from the health facility, land ownership and assets value quartile). Assets are measured as the value of owned houses, durables in the house, farm animals, farm equipment and microenterprise equipment. In the prenatal care quality analysis, we control for patient-level characteristics (age, education, partner lives in household, and insurance enrollment) and include a variable indicating whether the observation was from the facility exit interview or household survey in order to control for potential recall bias from the household sample. In the children s utilization analyses, we include controls for individual-level characteristics (age, gender), parentallevel characteristics (height, age and education of mother, father lives in household) and household characteristics (health insurance, number of household members, number of household members under age 6, land ownership, assets value quartile). Statistical Power and Methods Using data from the baseline survey, we conducted ex-post power calculations to determine the minimum effect sizes that we would be able to observe with the obtained sample sizes. Using a significance level of 5% and power of 0.9, observed correlations at baseline within the district clusters, and observed sub-sample sizes (women and children), we determined that the available sample would allow us to detect the following: an increase in the likelihood of making 4 or more prenatal care visits from 0.18 to 0.318; an increase in the likelihood of institutional delivery from 0.35 to 0.473; an increase in the likelihood of a preventive care visit by a 0-23 month old child from 1
0.21 to 0.319; and an increase in the likelihood of a preventive care visit by a 24-59 month old child from 0.08 to 0.1770. As a result of the reassignment of districts between the treatment and comparison groups, we view the evaluation design as quasi-experimental and use difference-in-differences to estimate program impact. This method compares the change in outcomes in the treatment group to the change in outcomes in the comparison group. By comparing changes, we control for observed and unobserved time-invariant characteristics as well as for timevarying factors that are common to the treatment and comparison groups. The change in the comparison group is an estimate of the true counterfactual i.e. what would have happened to the treatment group if the additional financing was not tied to performance. 2
Web appendix Table 1: Prenatal care utilization and quality : full regression results (1) (2) (3) (4) (5) Dependent variable Any prenatal care (=1) 4 + prenatal care visits (=1) Institutional Delivery (=1) Tetanus Vaccine (=1) Standardized Quality Score Treatment (=1) 0.002 0.008 0.081** 0.051* 0.157** (0.011) (0.035) (0.032) (0.026) (0.065) 2008 (=1) 0.021** 0.105*** 0.138*** 0.033 0.164*** (0.010) (0.021) (0.023) (0.030) (0.057) Age < 20 years (=1) -0.034-0.044 0.042 0.106*** -0.049 (0.043) (0.052) (0.092) (0.028) (0.049) Age > 35 years (=1) -0.011-0.040* -0.044-0.246*** -0.026 (0.009) (0.022) (0.030) (0.026) (0.023) Primary education or more (=1) 0.017 0.020 0.068* 0.026 0.078* (0.013) (0.031) (0.040) (0.034) (0.044) Partner lives in household (=1) 0.039* 0.020-0.004 0.072** 0.019 (0.022) (0.031) (0.055) (0.031) (0.038) Number of pregnancies -0.002 0.002-0.017** (0.003) (0.005) (0.007) Health insurance (=1) 0.010 0.039* 0.068** 0.003 0.047* (0.007) (0.020) (0.028) (0.022) (0.025) Number of household members 0.004-0.002-0.011 (0.004) (0.006) (0.009) Household-Facility distance (in Km) -0.001-0.007-0.020** (0.002) (0.005) (0.008) Household owns land (=1) -0.010 0.015 0.032 (0.010) (0.031) (0.038) Assets value quartile 2 (=1) 0.032** 0.012 0.027 (0.016) (0.022) (0.032) Assets value quartile 3 (=1) 0.032** -0.006-0.006 (0.015) (0.022) (0.027) Assets value quartile 4 (=1) 0.032** 0.027 0.075** (0.015) (0.021) (0.031) Exit interview (=1) -0.134* -0.157*** (0.071) (0.053) Observations 2309 2223 2108 2856 3826 Facility fixed effects Yes Yes Yes Yes Yes Number of health facilities 164 164 164 148 148 Notes: The coefficient on Treatment (=1) is the estimated treatment effect controlling for the listed covariates. We use a linear model with fixed effects at the facility level. Standard errors were adjusted for clustering at the district-year level. In columns (1), (2) and (3), the sample consists of women who gave birth within 18 months prior to the household survey. In columns (4) and (5), the sample consists of pregnant women who exited the health facility at the time of the exit interview and had obtained prenatal care, as well as women who gave birth within 18 months prior to the household survey; the corresponding models include a variable indicating whether the observation was from the facility exit interview or household survey. Standard errors are reported in parentheses. The number of observations in the tetanus model is less than in the quality score model because tetanus is only given to women with 5 pregnancies or less. *** p<0.01, ** p<0.05, * p<0.1 3
Web appendix Table 2: Child preventive care and immunization: full regression results (1) (2) (3) Dependent variable Preventive visit in last 4 weeks Fully immunized Sample 0-23 months 24-59 months 12-23 months Treatment (=1) 0.119*** 0.111*** -0.055 (0.039) (0.025) (0.064) 2008 (=1) 0.020-0.047** 0.005 (0.033) (0.020) (0.028) Female (=1) -0.034* -0.011-0.016 (0.020) (0.009) (0.034) Maternal height (cms) -0.000 0.001-0.003 (0.001) (0.001) (0.003) Mother's age -0.002 0.001 0.000 (0.002) (0.001) (0.003) Mother completed primary education (=1) 0.027 0.027 0.041 (0.038) (0.024) (0.071) Father lives in household (=1) -0.003 0.023 0.115* (0.037) (0.026) (0.058) Health insurance (=1) 0.010-0.001 0.029 (0.023) (0.013) (0.039) Nr of household members -0.009-0.006 0.009 (0.007) (0.004) (0.010) Nr of household members 0-5 years old 0.007-0.003-0.053* (0.014) (0.011) (0.027) Household owns land (=1) 0.051 0.030 0.038 (0.030) (0.032) (0.074) Assets value quartile 2 (=1) 0.017-0.012 0.136*** (0.028) (0.018) (0.033) Assets value quartile 3 (=1) 0.033-0.007 0.080 (0.026) (0.019) (0.049) Assets value quartile 4 (=1) 0.014 0.002 0.134*** (0.031) (0.020) (0.044) 3-5 months old (=1) -0.162*** (0.047) 6-8 months old (=1) -0.188*** (0.066) 9-11 months old (=1) -0.245*** (0.051) 12-14 months old (=1) -0.356*** (0.052) 15-17 months old (=1) -0.343*** -0.012 4
(0.054) (0.047) 18-20 months old (=1) -0.368*** -0.054 (0.059) (0.054) 21-23 months old (=1) -0.379*** 0.006 (0.053) (0.056) 27-29 months old (=1) -0.036 (0.026) 30-32 months old (=1) -0.088*** (0.030) 33-35 months old (=1) -0.050 (0.036) 36-38 months old (=1) -0.027 (0.024) 39-41 months old (=1) -0.080*** (0.027) 42-44 months old (=1) -0.091*** (0.028) 45-47 months old (=1) -0.085*** (0.031) 48-50 months old (=1) -0.094*** (0.028) 51-53 months old (=1) -0.081*** (0.030) 54-56 months old (=1) -0.091*** (0.029) 57-59 months old (=1) -0.077** (0.037) Observations 1971 2902 872 Facility fixed effects Yes Yes Yes Number of health facilities 164 164 162 Notes: The coefficient on Treatment (=1) is the estimated treatment effect controlling for the listed covariates. The omitted age category is 0-2 months (column 1), 24-26 months (column 2), and 12-14 months (column 3). In each of the columns, the sample consists of children that belonged to households interviewed in the household survey and who were within the indicated age range. We use a linear model with fixed effects at the facility level. Standard errors were adjusted for clustering at the district-year level. Standard errors are reported in parentheses. *** p<0.01, ** p<0.05, * p<0.1 5
References i Ministry of Health Contractual Approach Unit, Rwanda. Quarterly quality assessment grid for health centers. Kigali: Ministry of Health, 2008. ii Barber S. Does the quality of prenatal care matter in promoting skilled institutional delivery? A study in rural Mexico. Matern.Child Health J. 2006;10(5):419-425. iii Barber SL, Bertozzi SM, Gertler PJ. Variations in prenatal care quality for the rural poor in Mexico. Health.Aff.(Millwood) 2007;26(3):w310-23. iv Barber SL, Gertler PJ. Empowering women to obtain high quality care: evidence from an evaluation of Mexico's conditional cash transfer programme. Health Policy Plan. 2009;24(1):18-25. v Peabody JW, Gertler PJ. Are clinical criteria just proxies for socioeconomic status? A study of low birth weight in Jamaica. J.Epidemiol.Community Health 1997;51(1):90-95. vi Peabody JW, Luck J, Glassman P, Jain S, Hansen J, Spell M, et al. Measuring the quality of physician practice by using clinical vignettes: a prospective validation study. Ann.Intern.Med. 2004;141(10):771-780. vii Peabody JW, Rahman O, Fox K, Gertler P. Quality of care in public and private primary health care facilities: structural comparisons in Jamaica. Bull.Pan Am.Health Organ. 1994;28(2):122-141. viii Das J, Hammer J. Which Doctor? Combining vignettes and item response to measure doctor quality. Policy Research Working Paper Series 2004; Nr. 3301. Washington DC: The World Bank. ix Grosch, Margaret and Paul Glewwe, Eds. Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 years of the Living Standards Measurement Surveys. Washington DC: World Bank, 2000. 6