EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING IN HEALTHCARE IN SIERRA LEONE

Transcription

1 CORDAID NAAM BU» REPORT EXTERNAL VERIFICATION JUNI 2014 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING IN HEALTHCARE IN SIERRA LEONE

2

3 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 1 TABLE OF CONTENTS Table of Contents... 1 List of Tables... 3 List of Figures... 3 Acronyms... 5 Acknowledgement... 7 Executive Summary Introduction Terms of Reference of the External Verification The External Verification team Set up of the report Background Health Sector in Sierra Leone Health Status Health system Free Health Care Initiative PBF in the Sierra Leone context Trends in service delivery Methodology of the external verification Introduction Organisation Project team and verification teams Standard working procedure Learning approach Timeframe Sources of information Quantitative data Cross cutting issues Qualitative data Sampling Facility sampling Sampling for Patient Tracing and Patient Satisfaction survey Quality assurance Prior to data collection During the verification After data collection Reliability and significance Reliability of data Equal distribution of facilities Significance of PHU data Significance of patient satisfaction data Completeness of data Data analysis Systems used Primary Facility Output Data Data from Patient and key informant interviews External verification in the hospitals External Verification Findings Introduction Indicators Hospital PBF Output indicators in PHUs... 36

4 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE Variation between sources of information Influence of missing data on calculations Differences in data in facilities Differences per level of facility Differences per geographical area Satisfactory and unsatisfactory entries Crosscutting Issues General Specific indicators Hospital external verification Ola During Children Hospital Princess Christian Maternity Hospital Non-PBF hospital Patient Tracing and Satisfaction Survey Patient tracing Patient satisfaction Satisfaction scores Satisfaction scores per level of facility Payment for services Patient tracing by Councils and DHMTs Systems Assessment Introduction Accessibility and equity Autonomy and accountability PHUs Capacities Planning and management of small projects Financial management in practice Delays in payment Community involvement Separation of functions in the PBF programme Councils DHMTs MOHS MOFED Definition of indicators Contracts Hospital PBF Contracts Implementation Indicators Expenditure Non-PBF hospitals Discussion Quality Data quality Missing data Data consistency Case definitions Triangulation PBF light Financial management Conclusions and Recommendations Conclusions The general and specific objectives of the PBF programme The Terms of Reference Recommendations Validation workshop Short-term recommendations Clarification of the operational manual... 77

5 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE Longer-term recommendations Future developments of PBF Hospital PBF LIST OF TABLES Table 1 Summary of activities and outputs Table 2 Sources of information and tools used to capture this information Table 3 Methods used for assessment of cross cutting issues Table 4 Data sources and tools for qualitative information Table 5 Number of PHUs in the PBF project in 2012 per district Table 6 Number of PHUs selected per Council (urban areas) Table 7 Geographical areas selected in each rural Council Table 8 Distribution of PHU type per district Table 9 Domains for assessment of hospital performance Table 10 Totals per output indicator sampling 4 months Table 11 Extrapolation of all service attendance based on missing data Table 12 Differences between data sources above 25% in PHUs Table 13 Differences between data sources FP per level PHU Table 14 Percentage satisfactory entries per Council per indicator Table 15 Differences between average satisfactory scores (IV and EV) per district Table 16 Percentage satisfactory records per level health facility Table 17 Percentage of persons who could not be traced in EV Table 18 Reasons for not interviewing persons in EV Table 19 Patient satisfaction scores and contributing factors Table 20 Patient satisfaction per level of facility Table 21 Persons interviewed who were asked to pay for services Table 22 Average, minimum and maximum amounts paid Table 23 Involvement in action panning of PHUs Table 24 PBF payments to PHUs requested by MOHS to MOFED Table 25 Percentage of quarterly payments, traced in PHUs during the EV Table 26 Roles of HMC according to chairs HMC Table 27 Registers reported out of stock by PHUs LIST OF FIGURES Figure 1 Family planning Figure 2 ANC Figure 3 PHU deliveries Figure 4 Children fully immunised Figure 5 Selection process for patient sampling Figure 6 Number of PHU selected per district Figure 7 Formula for calculating sample size Figure 8 Percentage completeness of family planning records per districts Figure 9 Reasons for unavailability of records per information source Figure 10 Comparison Output indicators per source of information Figure 11 Comparison sources of information per indicator Figure 12 Extrapolation of service utilisation FP based on missing data Figure 13 Extrapolation of ANC service utilisation figures based on missing data Figure 14 Distribution of facilities by differences Figure 15 Geographical spread of facilities with differences data sources Figure 16 Distribution of Absolute Differences between IV and EV Figure 17 Plot box differences satisfactory entries Figure 18 Crosscutting issues IV and EV Figure 19 Distribution crosscutting issues scores in EV Figure 20 Distribution of crosscutting scores in IV Figure 21 Kono District Comparing Crosscutting Indicators IV and EV... 47

6 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 4 Figure 22 Comparing crosscutting indicator Administration IV and EV Figure 23 Comparison crosscutting indicator Stock Outs IV and EV Figure 24 Ola During Children Hospital IV and EV Figure 25 Trend in IV scores in Ola During Children Hospital Figure 26 Princess Christian Maternity Hospital IV and EV Figure 27 Trend analysis Princess Cristian Maternity Hospital Figure 28 Comparison PBF and non-pbf hospitals in EV Figure 29: Building blocks of RBF Figure 30 Word cloud investments in PHUs Figure 31 Roles in PBF in Sierra Leone Figure 32 Relations in the quality system Figure 33 PHU-F1 form box totals Figure 34 Step by step introduction of sampling IV... 76

7 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 5 ACRONYMS AIDS ANC CHC CHP CSOs DHIS DHMT DHS DMO DPPI EmONC EPI EV FHC GDP HMC HIV HMIS IMNCI IV IVT LC LGFD M&E MCH MCHP MDGs MOFED MOHS NGO NHA OM PAD PBF PHU PMTCT PRSP RCH RCHP SDHSP SDPS SLDHS TBA TOT UN UNICEF WB WHO Acquired Immunodeficiency Syndrome Antenatal Care Community Health Centre Community Health Post Civil Society Organisations District Health Information System District Health Management Team Demographic and Health Survey District Medical Officer Directorate of Policy, Planning and Information Emergency Obstetric and Neonatal Care Expanded Programme on Immunisation External Verification Free Health Care Gross Domestic Product Health Management Committee Human Immunodeficiency Virus Health Management Information System Integrated Management of Neonatal and Child Illnesses Internal Verification Internal Verification Team Local Council Local Government Finance Department Monitoring and Evaluation Maternal and Child Health Maternal and Child Health Post Millennium Development Goals Ministry of Finance and Economic Development Ministry of Health and Sanitation Non-Governmental Organisation National Health Account Operational Manual Project Appraisal Document Performance-Based Financing Peripheral Health Unit Prevention of Mother to Child Transmission Poverty Reduction Strategy Paper Reproductive and Child Health Reproductive and Child Health Project Strengthening District Health Service Project Service Delivery Perception Survey Sierra Leone Demographic and Health Survey Traditional Birth Attendant Training of Trainers United Nation United Nations Children Fund World Bank World Health Organisation

8 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 6

9 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 7 ACKNOWLEDGEMENT To a great extent, this verification consisted of qualitative studies and included inputs from health care staff and the population in service areas of the Peripheral Health Units (PHU). Only part of the data was collected in a quantitative way, by looking at registers in health facilities and by using the existing health information systems. The success of any External Verification exercise relies heavily on the collaboration and openness of key people working in health facilities, policy makers and people seeking health care. Having targeted 235 PHUs, their HMCs and 8 clients per PHU for structured interviews, almost 2000 people were interviewed at local levels. Without exception, we received full collaboration and dedication from all participants, not in the last place from people on the road who guided us to hard- to-reach areas in order to trace patients. Thanks to all these people it was possible to collect a tremendous amount of valuable information for our analysis and recommendations. This information will probably also be used for future comparison and trend analysis. The Councils and District Health Management Teams (DHMTs) play a crucial role in steering, management, monitoring and capacity building in relation to the performance based financing (PBF) programme. For the External Verification we relied on them for planning and facilitation of the fieldwork and as resource for data collection and interviews. All 13 DHMT s and 18 Councils were interviewed and shared valuable information. A special word of thanks goes to all the Monitoring and Evaluation (M&E) officers and Planning Officers at the DHMTs and Councils who assisted us in the collection of the required quantitative PBF data. We spent many hours in the M&E offices and always met collaborative attitudes. Without the patience and willingness of our counterparts at central MOHS and MOFED to share information and additional data, the External Verification exercise would have been impossible. The PBF programme in Sierra Leone does not operate in a vacuum as several international Development Partners contribute to the progress and improvements of the country s healthcare sector. The External Verification team had extensive meetings with several of them. This enabled us to put our findings in a broader perspective. We hope that this report provides useful inputs for further strategising and alignment of all the interventions in the health sector. Cordaid is grateful that this assignment was entrusted to us. Our activities not only focused on performing the technical verification, but also on empowering different actors in the health sector and on increasing knowledge on internal verification. Thus, it is our sincere hope that the insights and knowledge that was built during the exercise will strengthen Sierra Leone s Performance Based Financing programme. We hope it may also set an example for efficient organisation of independent external verification in other countries. The EV team consisted of Dr Jaap Koot, team leader, Mrs Marjan Kruijzen, project manager, Mr Frank van de Looij, PBF-expert, Mr Chenjerai Sisimayi, data analyst, and Mr David Yambasu, field coordinator. PBF External Verification Team

10 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 8

11 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 9 EXECUTIVE SUMMARY Introduction The health sector of Sierra Leone receives support from the World Bank through the Reproductive and Child Health (RCH) Project, which has the objective to increase utilisation of a package of essential health services by pregnant and lactating women and children under the age of five. Performance Based Financing (PBF) is part of the RCH project. The general objective of the PBF system is: to change the behaviour of health providers at facility level for them to deliver more quality services under the free health care policy. Free health care is a collaborative effort of the Government of Sierra Leone and Development Partners to achieve the Millennium development Goals. The PBF programme exists of payment for six output indicators, corrected for payment for crosscutting issues and remoteness of facilities. District health management teams and Council officials perform quarterly internal verification. Part of the programme is a Hospital PBF, piloted in two hospitals in Freetown. The Ministry of Finance and Economic Development - IPAU (Integrated Project Administration Unit) of the Republic of Sierra Leone contracted the Dutch Non-Governmental Organisation Cordaid for the External Verification (EV) of the PBF component of the RCH project in the health sector in Sierra Leone, starting on 28 October 2013 and ending on 28 April The Terms of Reference (TOR) for the external verification were: 1. To review the accuracy of the facility data from the registers and other records; 2. To analyse the data of the first full year of PBF implementation (2012); 3. To review the roles and responsibilities of the different PBF stakeholders and advise on the areas of improvement if necessary; 4. To evaluate the benefits of the performance based financing in term of services delivery, strengthening the health system information (verification of data and timely reporting), the governance of health facilities (management of human resources, environmental health, financing, etc.). Cordaid developed a methodology for the External Verification and fielded teams to visit all 19 Councils, 13 Districts, 47 Community Health Centres (CHCs), 52 Community Health Posts (CHPs) and 130 MCH Posts, sampled. Furthermore four Hospitals were visited for an external verification of the Hospital PBF. Findings of the External Verification of Output Indicators The External Verification of Output Indicators in the PBF system showed: Considerable, sometime significant differences exist between aggregated numbers in internal and in external verification. With exception of deliveries, the aggregated Internal Verification (IV) figures are 12% - 73% higher than the EV. Recorded attendance in the IV is in the majority of indictors also higher than other sources of information (Health Management Information System (HMIS), or F-forms 1 ). In general, the aggregated figures from various sources of information differ, whereby the EV showed most concordance with F-reports. The differences cannot be attributed to missing data. At facility level for all indicators the differences between sources of information are large, often more than 25% higher or lower. The differences in recording are spread over the country and not related to specific districts. Lower-level health facilities show larger error margins than higher-level facilities. There is no statistically significant difference between IV and EV as regards percentage of satisfactory or unsatisfactory entries in the records. There is no significant difference per level of facility, and not per district. Crosscutting issues The external verification for crosscutting indicators showed: The scores for the crosscutting issues in the external verification were consistently lower than in the internal verification in nearly all districts, for nearly all the indicators. 1 F-forms or returns are standard reporting forms filled by PHUs and sent to the District Health Management Team for entry into the automated HMIS.

12 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 10 In the external verification standardised assessment criteria were applied, reducing the chances of personal bias. Those criteria might have been stricter than applied in the internal verification. Due to the time lapse between 2012 and 2014 differences may have been created, e.g. missing registers, leading to lower scores. Worsening supplies of medicines may have caused lower scores in availability of essential medicines in 2014 compared with The high percentage of maximum scores for all indicators in all facilities, as found in the internal verification in some districts, could not be confirmed in the external verification. Hospital PBF The external verification of the Hospital PBF found that: The EV team gave slightly higher scores in the EV to Ole During Child Hospital compared to the latest IV (79% vs. 61%), but lower than in other IVs (85%-95%) The EV team gave slightly lower scores in the EV to Princess Christian Maternal Hospital compared to the latest IV (84% vs. 89%), in the range of other IVs (82%-89%) Hospitals not receiving PBF scored lower than PBF hospitals, but one of those scored only slightly lower, while the score of the other hospital was wide off range. Patient tracing and satisfaction The external verification of the patient tracing and satisfaction found that: 92.6% of the patient/clients earmarked for tracing could indeed be identified either by meeting the clients in person, or by identification by a member of the community. There is no reason to believe that PHUs recorded ghost patients to inflate the numbers of attendance. The average satisfaction score of clients was 7.3 (out of 10), with a variation between 4.1 and 9.8. Client satisfaction was strongly related to short waiting times, friendly treatment, availability of medicines and non-payment for services. 12% of patients interviewed had to pay for services, although they were supposed to benefit from Free Health Care. Systems analysis Free Healthcare was introduced in 2010, and was supported by several donors an agencies, e.g. through human resources management, provision of medicines, etc. Free health care has resulted in considerable increase in service delivery in reproductive and child health services, although recently there has been a levelling off of service utilisation at a higher level than before the introduction of Free Healthcare. The PBF programme works complementary to Free Healthcare, and offers to health facilities a compensation for the loss of income through patient fees. The programme has been successful in this. The programme has succeeded in providing more autonomy to health facilities to manage their own small projects, which contribute to better work environment: more hygiene, better equipped buildings and better supplies have been achieved. Financial management is a weak area, with virtually no systems in place at grass root level. Late payment during the period of review affected continuity of the PBF programme and had high opportunity costs: PHUs were eager to implement health services improvements, but had to wait for over one year to get their due payments. Health workers expressed fear that they would not receive their bonuses after such a long period of delay, e.g. after transfer. Not receiving a performance bonus created frustration, rather than motivation for better performance. The programme has succeeded to some extent in improving community contribution to management of health facilities, although the capacities are still limited. In Sierra Leone a light PBF approach is applied, which means that not all theoretical concepts of PBF with regard to separation of responsibilities (e.g. Health results Innovation Trust Fund 2 ) are fully implemented. The Local Council is officially responsible for the health services, but is at the same time the contracting agency. The DHMT is the technical supervisor and at the same time the internal verifier. In practice the collaboration between Councils and DHMTs often is not as envisaged in Sierra Leone s PBF plan. The DHMTs often operate independently, and Councils do not feel engaged in the programme. 2

13 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 11 Conclusions The general objective of the PBF system is: to change the behaviour of health providers at facility level for them to deliver more quality services under the free health care policy. The EV team concludes that systems have been put in place and are operational to a reasonable extent in a number of health facilities (see Chapter 7.1). Further strengthening of the system is possible within the present design of PBF in Sierra Leone, when a number of implementation issues can be solved. The EV team concludes that PBF provides cash at facility level to cover the local costs of delivering services and removing the need for 'informal' fees. Only 12% of the patient paying for those services, which supposedly are free. Late transfers of PBF funds may have forced PHUs to ask for contributions for patient records, etc. when funds dried up. Payments by patients may reduce further if PBF payment improves. Incidental misbehaviour by health workers cannot be ruled out. The EV team concludes that to some extent PBF provides financial incentives to facilities in order to increase productivity and quality of care, especially for the identified key indicators. There is an increase in service utilisation, although that increase is levelling off. There are signs of improved attention for quality. However, the relation between performance and payments is too weak for health workers. The incentive system is not transparent enough and payments come so late, that they are no longer seen as reward for good performance. Equity of distribution of funds may have taken place using district-based payment formula, but was not visible for grass root workers. The flow of funds in general was not regular enough to hire contract workers (with exception of the two PBF hospitals). Outreach may have benefited from PBF funds, e.g. by repair of motorcycles and purchase of fuel. In general, funds were used for repairs of the building, furniture, equipment and supplies, water and sanitation, etc. These investments have contributed to patient satisfaction and higher scores for crosscutting quality indicators. Recommendations The MOHS district visits planned for the month of April 2014 will offer an opportunity to confirm with the Councils the roles and the responsibilities as laid-down in the PBF operational manual. The roles of the Councils in contracting, in internal verification, and in financial management and reporting have to be renegotiated per Council, as circumstances and conditions may vary. The roles of HMCs have to be clarified. New Memoranda of Understanding can be signed to confirm commitments. During the district visits the MOHS could provide an orientation workshop on quality of internal verification. The quality of internal verification has to improve: uniform case definitions have to be applied, and DHMT members involved should understand their tasks well. On the spot double check of IV report, F-forms and HMIS form (brought from the DHMT s M&E office) should be introduced to identify data inconsistencies and resolve them, or explain them. The validation workshop at the end of the external verification called for simplification of the Internal Verification, while improving the quality. The idea was to introduce sampling, not only months (one month per quarter), but also PHUs (e.g. 25% of PHUs). HMIS data would be guiding in payment for performance, rather than the data from IV. This is possible, but only if certain criteria are met. The first step in this process is to guarantee data quality of registers, F-forms and HMIS. Facilities should have the required registers and forms. HMIS and F-forms should be filled completely and should match. Districts, which cannot meet minimum criteria of HMIS quality, should first bring their house in order. The second step is to select PHUs, which meet criteria of data quality, with matching IV and HMIS. Those with reasonable data quality are admitted to the pool. But they can be removed from the pool if in a control they are found to be missing the quality standards. From there, step-by-step, more facilities are added to the pool. NB: quarterly supervision and assessment crosscutting issues should continue in all health facilities!

14 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 12

15 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 13 1 INTRODUCTION 1.1 TERMS OF REFERENCE OF THE EXTERNAL VERIFICATION PBF project The health sector of Sierra Leone receives support from the World Bank through the Reproductive and Child Health (RCH) Project, which has the objective to increase utilisation of a package of essential health services by pregnant and lactating women and children under the age of five 3. Performance Based Financing (PBF) is part of the RCH project. The general objective of the PBF system is: to change the behaviour of health providers at facility level for them to deliver more quality services under the free health care policy. The specific objectives of the system are 4 : 1. Provide cash at facility level to cover the local costs of delivering services and removing the need for 'informal' fees. 2. Provide financial incentives to facilities in order to increase productivity and quality of care, especially for the identified key indicators. 3. Increase the equity of distribution of resources with funds from PBF allowing facilities to hire contractual workers and finance outreach activities. Terms of Reference The Ministry of Finance and Economic Development - IPAU (Integrated Project Administration Unit) of the Republic of Sierra Leone contracted the Dutch Non-Governmental Organisation Cordaid for the External Verification (EV) of the PBF component of the RCH project in the health sector in Sierra Leone, starting on 28 October 2013 and ending on 28 April The Terms of Reference (TOR) for the external verification are: 1. To review the accuracy of the facility data from the registers and other records; 2. To analyse the data of the first full year of PBF implementation (2012); 3. To review the roles and responsibilities of the different PBF stakeholders and advise on the areas of improvement if necessary; 4. To evaluate the benefits of the performance based financing in term of services delivery, strengthening the health system information (verification of data and timely reporting), the governance of health facilities (management of human resources, environmental health, financing, etc.). The assignment is therefore broader than an external verification per se; it encompasses an assessment of elements of the project design and implementation. The summary of the Terms of Reference is found in annex 1, in Volume II of the report. 1.2 THE EXTERNAL VERIFICATION TEAM The EV team consisted of Dr Jaap Koot, team leader, Mrs Marjan Kruijzen, project manager, Mr Frank van de Looij, PBF-expert, Mr Chenjerai Sisimayi, data analyst, and Mr David Yambasu, field coordinator. The field team for data collection consisted of five teams of local experts from the organisations Christian Brothers, SEND and Njala University, School of Community Health Science. The names of the team coordinators and enumerations are listed in annex 2, of Volume II of the report. 1.3 SET UP OF THE REPORT The following chapters describe the background of the health sector in Sierra Leone (Chapter 2), the methodology of the external verification and the sampling (chapter 3). Chapter 4 describes the analysis of output indicators and crosscutting issues, as well the hospital PBF. Chapter 5 gives 3 RCHP project Grant Agreement, PBF Operational Manual, version edited July 2013

16 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 14 information on the patient satisfaction survey, and chapter 6 analyses the PBF system. Chapter 7 discusses some key issues and Chapter 8 gives conclusions and recommendations. The annexes of the report are in a separate volume, and contain Terms of Reference, Team members, List of Samples Facilities, Specific Council Reports and Case Definitions.

17 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 15 2 BACKGROUND 2.1 HEALTH SECTOR IN SIERRA LEONE HEALTH STATUS Since the end of the civil war in 2002, Sierra Leone has made progress in improving the health status of the population 1. Maternal Mortality Ratio (MMR) and Child Mortality Rate (CMR) decreased to MMR 890 per live births in 2010 and CMR to 174 per 1,000 live births in 2010, but are at still far from the MDG targets of 320 and 92 respectively 5. Fertility rates dropped from 6.5 in 2004 to 5.0 in 2010, but only 17% of the demand for family planning was satisfied in The 2010 document that lays the foundation for the Basic Package of Essential Health Services for Sierra Leone 6, stated the Health Status of the population compared to other sub-saharan countries is critical. The disease burden of children under-five consists mainly of communicable diseases and poor nutrition. Malaria (38%), acute respiratory infection (16.9%) and watery & bloody diarrhoea (9.7%) together, account for about 65% of all diseases. The stunting percentage (36.4% in 2008) also contributes to the high disease burden for under-fives. Inequity in the health system also remains a problem, both in terms disparities between income groups as well as disparities between geographical locations 7. For example, the percentage of births attended by skilled health personnel is around 28% for the poorest and around 75% for the richest quintiles HEALTH SYSTEM Infrastructure Peripheral Health Units (PHUs), i.e. Community Health Centres (CHCs), Community Health Posts (CHPs), and Maternal and Child Health Posts (MCHPs) deliver Primary Health Care. There are 40 hospitals in the country 8. The number of government health facilities has increased to over 1,200 in 2012 compared to 843 in There are now five Basic Emergency Obstetric Care (BEmOC) centres each in all the 13 districts. In total, 13 district medical stores have been constructed to enable storage of medicines and medical products both at national and district levels. Blood Banks have been established in all district hospitals to provide safe blood for transfusion. A school for training midwives was established in Makeni 9. Since the start of the National Health Sector Strategic Plan (NHSSP) , Government has embarked on a series of improvements in the health sector. It has increased the total workforce in the public health sector from 7,164 in 2009 to 8,446 in Incentive allowances are provided to health workers in remote communities. Many PHUs are still heavily understaffed, or do not dispose of adequate infrastructure (electricity, water supply) or equipment 7. Organisation The health system in Sierra Leone is decentralised, with devolution as the mode of operation. The Ministry of Health and Sanitation (MOHS) is responsible for formulating government health policies and for technical guidance to the Councils, who as Local Government Authorities are responsible for the implementation of health services. The MOHS provides technical guidance in the area of health 5 Source : Sierra Leone Health Data 2012 Profile 6 Basic Package of Essential Health Services for Sierra Leone, Government of Sierra Leone, Ministry of Health and Sanitation, March Koyejo Oyerinde, Yvonne Harding, Philip Amara, Rugiatu Kanu, Rumishael Shoo, Kizito Daoh, The status of maternal and newborn care services in Sierra Leone 8 years after ceasefire, International Journal of Gynecology and Obstetrics, 114 (2011) National Health Sector Strategic Plan (NHSSP) , Government of Sierra Leone, Ministry of Health and Sanitation, November MOHS, Health Sector Performance Report, Draft, July 2012

18 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 16 and has an important task in development of human resources for health and in logistics and supplies of medicines and equipment for the health facilities. The Ministry of Finance and Economic Development (MOFED) finances most of the health services through the Councils and finances the human resources in health through the Human Resources Management Office (HRMO). In the districts the Local Councils are responsible for most of the service provision to the community, including health. The District Health Management Teams (DHMTs) manage the primary healthcare services on behalf of the Local Councils. The NHSSP identifies the following major challenges in relation to health service delivery: (1) weak M&E capacity, (2) weak mechanisms for public accountability, (3) minimal involvement of communities, (4) low motivation of health workers, (5) high attrition rate, (6) weak health information systems and (7) inadequate budgetary allocations. Healthcare financing Both MOHS and Development Partners have increased the funding for the health sector considerably over the last years and have invested in human resources, supply of medicines, improvement of the infrastructure, etc. The total health expenditure is estimated at US$ Funding for the health sector was estimated around US$ 85 million in The MOHS liaises with multilateral organisations such as WHO, UNICEF, UNAIDS, Global Fund, GAVI, AfDB, EU, and World Bank and with bilateral organisations such as DFID, Ireland. The Government of Sierra Leone (GOSL) agreed a Health Compact with most of the Development Partners and NGOs aiming to make faster progress to achieve the Agenda for Change in health and the Millennium Development Goals FREE HEALTH CARE INITIATIVE The President of Sierra Leone launched the Free Health Care Initiative in April 2010 with the aim to increase access to health services by pregnant women and children. Removing patient fees would take away the barriers for the poorest in society to seek institutional health care. Measures undertaken to support this initiative included: Undertaking a payroll cleaning exercise that removed around 1000 ghost workers, freeing up resources, which enabled the Government of Sierra Leone (GoSL) to recruit 1000 legitimate health workers. This resulted in a 15% increase in the health workforce which was cost neutral; Increasing all health workers salaries by at least 90% with some highly skilled staff receiving fivefold rises in their salaries; Procuring over $10 million of pharmaceuticals and strengthening drug storage and supply systems; Initiating a new financial mechanism to provide cash grants to all health facilities for them to purchase essential supplies; Accelerating essential infrastructure repairs in hospitals, health centres and district drugs stores; Conducting a mass communications campaign across the country to inform the target population of their rights to free care. Indeed, after the introduction the number of under-fives outpatient consultations increased with 250% compared to the period before the launch of the Free Health Care Initiative, and this trend is continuing. Immunisation coverage for children increased from 67% in 2006 to 82% in Until today inputs from donors like DFID and UNICEF contribute to the Free Health Care Initiative for e.g. medicines and salary costs. This collaborative effort is the backbone for improving healthcare in Sierra Leone. 10 MOHS, National Health Accounts, draft MOHS, Health Sector Performance Report 2011, draft July GOSL, Health Compact, December 2011

19 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE PBF IN THE SIERRA LEONE CONTEXT Performance Based Financing was launched in Sierra Leone in April 2011, to strengthen the Free Health Care Initiative. PBF is a systematic approach to health reforms, which provides incentives for health worker performance to improve staff motivation and funds for additional investments at grass root level. It leverages major paradigm shifts in terms of accountability, governance, information systems, planning and the inclusion of communities in verification and providing feedback. This approach is expected to have impact on performance of the healthcare system and to have a multiplier effect on efforts of all partners involved in the Free Healthcare initiative. Sierra Leone has chosen to implement a light-pbf, with a limited set of indicators and a highly simplified, but well prioritised quality component. The PBF project team has not created new structures for the different functions within PBF, but utilises existing institutions for contracting and internal verification TRENDS IN SERVICE DELIVERY The Free Health Care initiative has resulted in increase of utilisation of health services. In general, a positive trend in service delivery figures is visible from the HMIS statistics. Family planning is still increasing, while Antenatal Care (ANC), institutional deliveries and children fully vaccinated are levelling off at a substantial higher level than before the start of the Free Health Care initiative. Unfortunately, during this external verification no quarterly statistics were available from before the start of Free Health Care in 2010 to quantify the increase. PBF is a countrywide system, to strengthen the impact of Free Health Care. There is no way to disaggregate the contribution from PBF to improvement of healthcare services and the contribution from other support activities, e.g. the support to salary increases, or the provision of essential medicines. All districts were benefiting from PBF. As mentioned before, the increase of service utilisation should be considered as the result of a collaborative effort of the MOHS, other Ministries, Agencies and Departments and all Development Partners. Figure 1 Family planning Source: MOHS, HMIS 13 MOHS, Performance Based Financing, Operational Manual, Revised Version, October 2013

20 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 18 Figure 2 ANC Source: MOHS, HMIS Figure 3 PHU deliveries Source: MOHS, HMIS Figure 4 Children fully immunised Source: MOHS, HMIS

21 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 19 3 METHODOLOGY OF THE EXTERNAL VERIFICATION 3.1 INTRODUCTION Because service providers are paid according to their performance, verification of reported performance is a crucial element in any performance-based financing (PBF) programme. Apart from frequent internal verification, it is common practice to externally verify the program. Reasons for this include fiduciary stipulations from donor organisations, limited capacities of organisations, which perform internal verification and/or limited separation of functions. External verification primarily answers the question whether payments in the program were indeed valid and legitimate. As indicated in the terms of reference, this assignment also targets to review the effectiveness of the programme. Therefore, the methodology used for external verification included several stakeholder interviews, validation workshops and an extensive patient satisfaction survey. This enabled the consultants to formulate clear recommendations for continuation of the programme. This chapter explains the methodology that was used in more detail. It explains how data collection was organized, which timeframe was applicable and which sources of information were used. An important element of the methodology is the sampling technique that was used to select the PHUs and patients that were visited to gather information. This technique is explained in a separate paragraph. Thereafter the methods for quality control and data analysis are explained. The chapter ends with an explanation the approach, which was used for external verification of the Hospital PBF component. 3.2 ORGANISATION PROJECT TEAM AND VERIFICATION TEAMS The final responsibility for the external verification lied with the international project team. This team was responsible for overall planning, creating instruments and tools, development of training material, organisation of workshops, quality assurance, data analysis and report writing. Data collection was done by five verification teams each consisting of one coordinator and three enumerators. The coordinators and enumerators were all employed by three partner organisations of Cordaid: Christian Brothers, School of Community Health Science and SEND. Christian Brothers and the School of Community Health Science provided each two coordinators and six enumerators. SEND provided one coordinator and four enumerators. Each verification team moved from Council to Council, covering a total of 18 councils, 235 PHU and 1196 patients in less than two months time. While the enumerators visited an average of two PHUs and 16 patients per week, the coordinator mainly concentrated on the quality control, Local Council and DHMT verification, data entry in the computer and a limited number of PHU verifications (one per week). Both the international and national supervisors closely monitored the data collection by moving from Council to Council, working alongside verification teams and executing spot checks STANDARD WORKING PROCEDURE The verification in the districts encompassed meetings with the Local Councils, the DHMTs, the PHUs, (traced) patients and community representatives (Health Management Committee, HMC). Each visit followed a structure of introduction, interview, verification and preliminary feedback. The figure below shows the approach.

22 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 20 Figure 1 Standard Working Procedure verification visits Local Council The coordinator of the external verification team introduced the exercise to the Council in a courtesy call to high-level officials. Thereafter the coordinator conducted meetings with relevant officers, especially the M&E Officer and the Finance Officer. Topics for discussion were the involvement of the Local Council in the internal verification, the financial management procedures and reporting. More in general the place of PBF in decentralisation-by-devolution was discussed. The coordinator verified the PBF financial reports, if available. All councils received a summary-sheet capturing the main findings of the verification in the respective Council. These summary reports can also be found in annex 4, in Volume II of this report. DHMT The District Coordinator introduced the external verification to the DHMT and interviewed the District Medical Officer (DMO) and other relevant officers, specifically the M&E Officer and the Finance Officer. Topics for discussion were the overall progress in district healthcare, implementation of Free Health Care and PBF systems, the internal verification process (e.g. role of the Councils), reporting and financial management systems. The coordinator thereafter collected information from the HMIS system (computers at district level) and verification reports. After completing the assignment the coordinator presented major findings, completeness of the verification, availability of information and progress made in the health facilities. The DMO also received the Council summary sheets. PHU The enumerator introduced the assignment and interviewed the PHU staff. Important topics during introduction were patient confidentiality and non-disclosure of medical records. Interview questions focused on how PBF has helped the PHU and what has been done with the money received. Pictures were taken of purchased equipment or improved infrastructure. Finally the method for patient tracing will be explained. Subsequently, the external verification took place and 8 patients were randomly selected from the registers. At the end of the visit, the enumerator provided feedback on reporting, data quality, overall progress, cross cutting issues and patient satisfaction. All information was captured in a PHU summary sheet. Health Management Committee The enumerator interviewed the Health Management Committee (HMC) and asked their views on the performance of the PHU and the health system in general. Participation in the decision-making and spending of PBF payments was an important topic for the discussion. HMC also assisted in tracing patients.

23 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 21 Patients/clients The enumerator visited 8 randomly selected patients within the service area of the health facility (maximum 10 KM). Opinion leaders, health staff and the HMC assisted in tracing patients. The enumerators first sought permission from chiefs or opinion leaders in the Chiefdoms. Clients receiving family planning were not traced, because of privacy issues. Children of less than five years old, which visited the PHU for consultation, were also not selected because the recall time between external verification and actual consultation in 2012 (between one and two years) is too long for relatives to remember the treatment exactly. Other indicators like antenatal visits, delivery, postnatal visits and vaccinations were for recall (often with evidence of ANC-cards or under-five cards). The enumerator assessed whether the patient actually visited the PHU and received the service indicated in the register. In addition, the enumerator also conducted a brief satisfaction survey (waiting time, staff attitudes, etc.) LEARNING APPROACH During all verification visits and interviews, emphasis was put on mutual learning. The coordinators and enumerators were trained in such a way that they were able to help all actors understand the weaknesses and challenges in reporting and verification. The main purpose of feedback was to explore possible interventions to improve the PBF system. Good examples of this approach were experienced during both the inception workshop and the validation workshop. During the inception workshop knowledge on indicator definition and sampling increased. The validation workshop led to increased insight in the functioning of the free health care system, drug availability and challenges in information systems. During the external verification exercise 22 people from three independent organisations were trained in external verification techniques. Also, several data collection tools were designed and the software package EPI-info (already used in the DPPI) was introduced for PBF data entry and simple analysis. The MoHS or other organisations can use these tools for future verification exercises. 3.3 TIMEFRAME The external verification took place from November 2013 to April 2014, covering 6 months in a sequence of activities, summarised as inception, verification, analysis and reporting. The table below shows the output per phase in the verification. Table 1 Summary of activities and outputs Work package Main activities Timeframe Output Scoping Mission Verification of facility performance Study documentation Field visit two districts (DHMT, council and facilities) Assess HMIS reliability Stakeholder interviews (MOFED, MOHS, UNICEF, DFID, World Bank, civil society) Stakeholder workshop Training of Council Coordinators and enumerators. Verify entry in registers Consistency between registers and HMIS Verify score on quality indicators Trace patients. Measure patient satisfaction Interviews with facility staff, DHMT and Councils November / December 2013 January / February / March 2014 Adjusted methodology Facility level verification sheets. Tool for patient tracing Interview questionnaires Planning for interviews Manual for enumerators Sampling of facilities Inception report Filled out verification sheets per facility. Filled out structured questionnaires. Brief verification reports at council level.

24 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 22 Data Analysis Validation of findings and reporting Data entry (EPI info) Final database cleaning and cross checks on data validity Standard sequential analysis of HMIS data Specific comparative analysis of indicators and cross sections Regression analysis Interviews at national and council level One day workshop with stakeholders and main decision makers Report writing March 2014 April 2014 Data analysis plan Preliminary data analysis and summary of interview findings as part of the issues paper and workshop presentations. Final report on verification assignment During the inception phase 14 the scope and methodology of the external verification were determined in detail. It allowed the MOHS to send out official letters to Local Councils and DHMTs, who in turn communicated with the PHUs and opinion leaders in Chiefdoms about the process of external verification and patient tracing. A crucial element in the inception phase was a workshop on 19 and 20 November 2013, in which the methodology was discussed, facilities were sampled and instruments for external verification were aligned with existing instruments. An important example is the elaboration of definitions used for the cross cutting issues as part of the quality checklist. A six-day training of verification teams marked the start of the verification phase in the second week of January This training focused on acquaintance with general healthcare knowledge, thorough training in auditing facility registers, interview techniques and data entry. Immediately after the training, the teams started the verification. The team started in the districts with meetings with the DMO to plan the exact dates of visits to PHUs, Community Health Committees, DHMT and Council, thus assuring the availability of staff and required records. The team did the sampling of months and patients to be traced in the district together with the DMO. While the verification teams worked through districts in batches and transferred data to the national supervisors and senior consultants, the analysis of district and PHU data already started. This allowed for corrections and further investigations when questions arose during the analysis. In some cases the verification teams and the supervisors returned to the districts for further data collection. The analysis phase 15 covered the entire month of March. During this month, the international consultants also performed the external verification in the two PBF hospitals. The preliminary results and conclusions were discussed in a validation workshop on the 20 th of March This resulted in additional interpretations and recommendations that provided input for this final report. 3.4 SOURCES OF INFORMATION QUANTITATIVE DATA As indicated before, the core of any internal or external verification is to investigate the validity and accuracy of reported patient numbers. In other words: is the information on which the payment was based consistent with the actual patient numbers that visited the facility? If this is not the case, it is necessary to understand the different stages of data processing and the challenges that exist during each of these stages. If the entries in the registers are incorrect, then the facility staff should be assisted to improve data entry. If differences are the result of data processing at district level, then capacities of M&E officers or data entry clerks should be improved or errors in information systems should be corrected. In general, five stages can be distinguished (see figure 2 below). The external verification in Sierra Leone looked at facility registers as well as actual patient visits, by means of random sampling of patients and asking for a confirmation of the reported visit. 14 The findings of the inception mission are reported in the PBF External Verification Inception Report, Cordaid, 18 Dec The preliminary findings of the analysis are reported in the PBF External Verification Issues Paper, Cordaid, 20 March 2014

25 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 23 Figure 2 Information Flow in the National PBF programme in Sierra Leone Patient visit For a specific service, eg. OPD consultation or delivery Facility registers For instance OPD under five or maternal and neonate register PHU-F report Summary sheets that are filled out by PHU staff and sent to DHMT HMIS Data from PHU-F reports entered in HMIS at DHMT level Payment Request Based on HMIS information Internal / External verification At the start of the PBF program in Sierra Leone, facilities were to be paid based on the attendance as recorded in the HMIS. However, later due to challenges in the completeness and accuracy of HMIS information, it was decided to base payments on the internal verification data. For the purposes of this external verification we distinguish five sources of data and checked their consistency: (1) patient tracing, (2) facility registers, (3) PHU-F reports, (4) HMIS system and (5) Internal verification reports. Table 2 summarises sources of information and tools used to collect the information during external verification. Table 2 Sources of information and tools used to capture this information Source Detailed description Tools used Patient tracing Interviews with 8 randomly sampled patients Excel sampling tool Structured questionnaire Data entered in EPI-Info Facility registers (referred to as external verification ) PHUF report HMIS Internal verification Payment Re-counting patients from the following registers: - Under five clinic register - Immunisation register - Family planning register - Maternal and neonate health register (ANC, PNC and delivery) Information on all six indicators was taken from the PHUF (1,2 and 3) reports which were present at facility level. Data produced by computerised HMIS system at district level. If data was not present at district level, information from national level was requested Internal verification reports which were present at the DHMT (district level) Cash books and ledgers present at facility Payment requests from MoHS to MoFED (national level) Excel sampling tool Tally sheets PHU verification sheet Data entered in EPI-Info Excel sampling tool PHU verification sheet Data entered in EPI-Info Excel sampling tool PHU verification sheet Data entered in EPI-Info PHU verification sheet Data entered in EPI-Info PHU verification sheet Data entered in EPI-Info

26 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE CROSS CUTTING ISSUES Apart from patient numbers (output indicators), payment to PHUs is also based on qualitative characteristics of the facility. These are measured through a checklist consisting of 9 crosscutting issues (see table 3). During the external verification, the coordinators and enumerators administered exactly the same checklist and their scoring was compared with the quarter 4, 2012 scoring from the internal verification. It has to be stressed though that some of the 9 cross cutting issues could not be verified objectively for the year Cleanliness of the facility may for instance have improved or deteriorated. Table 3 provides an overview of the cross cutting issues that were objectively verifiable and which were not. It also indicates how scoring was done if indicators were not objectively verifiable. Table 3 Methods used for assessment of cross cutting issues Cross Cutting Issue Facility attendance register is kept up-to-date and accurate All reports submitted to DHMT by 5th of the following month. Monthly minutes of facility health committee meetings are signed by chair and securely retained at facility. A wall chart is displayed, with upto-date information on each of the 6 PBF interventions and financial information. All paperwork kept in good order The PHU and surrounding area is clean and sanitary with no medical waste exposed, no tall grass, etc. Drugs records are accurate and upto-date. Appropriate waste management No stock-out of ACT, Amoxycillin or ORS Objectively verifiable Not objectively verifiable How measured? Could not be verified for Assessment looked at charts of the month before external verification. Actual situation during external verification was assessed. Assessment of actual situation. If burning pit is found it is assumed it was also present in Physical in store or medicines cupboard To ensure that the clear definitions were used during external verification, the definitions of the crosscutting issues were discussed with the DMOs during the inception workshop on the 19 th and 20 th of November. Definitions and additional instructions were included in the checklist for enumerators (see annex 5 in Volume II) QUALITATIVE DATA In order to review the effectiveness of the PBF program in Sierra Leone, qualitative information was gathered at different levels of the health system. Table 4 summarises the different sources for qualitative information and the tools used for data collection. It should be noted that most data at district, council, PHU and patient level were captured with the use of nominal scales. Patient satisfaction was measured through ordinal scales. This allowed for statistical analysis after data collection.

27 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 25 Table 4 Data sources and tools for qualitative information Level Sources Tools National Interviews with: Structured questionnaires - MOHS, MOFED - Development partners, (e.g. DFID, world bank, UNICEF) - NGO s (e.g. Save the Children and Action for Health) Districts/Councils Interviews with DMO and officials at Council level (often M&E or finance officer) Structured questionnaires Data entered in EPI-Info to enable statistical analysis PHU s Patients Interviews with in-charge and facility staff Interviews with Health Management Committee members Interviews with 8 patients randomly sampled from facility registers Structured questionnaires as part of PHU verification sheets Data entered in EPI-Info to enable statistical analysis Inventory sheet for investments and pictures taken Structured questionnaires for Health Management Committee Structured questionnaires for patient satisfaction Data entered in EPI-Info to enable statistical analysis 3.5 SAMPLING It is general practice that external counter verification of PBF programs is only performed in a sample of the total number of health facilities which are in the programme. Verifying all facilities is time consuming and costly, which might not be justified by the value of the extra information or payment corrections resulting from such an exercise. At the same time, the representativeness of the sample should be ensured. Conclusions from the sample should be valid for all facilities and therefore samples should be taken at random. Randomisation also serves another purpose. If each health facility believes it has an equal and relatively large chance of being selected in the sample, the temptation to intentionally over-report patient numbers will be reduced. In PBF programs a sample of 20-25% is generally accepted as sufficient to do justice to representativeness, whilst reducing intentional over-reporting. The TOR for the external verification in Sierra Leone originally mentioned a 25% sample. This however, was based on an assumption of 900 facilities in total. As the number of facilities participating in PBF was estimated at 1,200 in 2012, this percentage was reduced to 20%. For this assignment, a weighed stratified randomisation was used. This further enhances efficiency (grouping of facilities in geographical clusters) and representativeness (guaranteed geographical spread and inclusion of all levels of facilities). The facilities and Chiefdoms to be visited were sampled during the inception workshop on the 19 th and 20 th of November Sampling of months to be verified and patients to be traced took place during the actual verification in the districts. It should be emphasised that, while developing the sampling methodology, sustainability of the method was an important element. The method should be simple, robust and aligned with existing methods or administrative boundaries, to ensure that the sampling could be repeated easily in the future, with a minimum level of financial resources FACILITY SAMPLING Determining the number of facilities in the PBF program in 2012 Starting point for determining the sample of facilities was an overview, which was acquired from the national PBF team of the MOHS. This list contained all payments that were done to individual PHUs in the 13 districts in DMOs were asked to check for double entries, incorrect entries and missing

28 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 26 PHUs. Corrections were received during the inception workshop or closely thereafter and resulted in a total number of 1,163 PHUs that were in the program in Distribution of these facilities over the 13 districts can be found in table 5. Table 5 Number of PHUs in the PBF project in 2012 per district District # PHU Bo 110 Moyamba 99 Bombali 96 Port Loko 103 Bonthe 55 Pujehun 68 Kailahun 81 Tonkolili 96 Kambia 65 Western Area 106 Kenema 121 Koinadugu 72 Kono 91 Total 1163 Determining location of facilities and characteristics. In order to ensure geographical spread and an equal representation of different levels of PHUs in the sample, the list with PBF facilities was linked with a database with facility details from the 2011 WHO Service Availability and Readiness Assessment. This database contained information on the village and chiefdom in which the facility was located and the type of facility (CHC, CHP or MCHP). For approximately 90% of all PHUs the required details could be found. During the inception workshop, DMOs were requested to provide details regarding the remaining 10% of the PHUs. Sampling in urban areas PHUs in urban areas can be accessed easier; they are less geographically spread than rural areas. Therefore a different sampling approach was used. City councils and the entire district of Western Area were considered urban and for these areas a simple random sample of 20% of the total number of facilities was selected. The following additional criteria were applied: Minimal sample in a district is four (assuming that at least 1 CHC, 1 CHP and 2 MCHP s are covered). If there are less than 4 facilities in the council, then all facilities will be verified. The sample needs to contain at least one CHC, if present. If, after sampling, no CHC was selected, an additional CHC was randomly chosen and added to the selected facilities. Table 6 Number of PHUs selected per Council (urban areas) Urban - selected #PHU Bo city council 6 Makeni city council 4 Bonthe Municipal council 2 Kenema City Council 8 Koidu new Sembehun city council 4 Freetown an WA 21 Total 45 For each council, all facilities were listed and each PHU was given a unique identifier number. During the inception workshop a computerised random series of numbers was generated and linked to the list, thus resulting in a 20% sample in each city council. Table 6 shows the final results per council. A

29 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 27 detailed list with facility names can be found in Annex 5 of Volume II. All Council samples contained a CHC. Sampling in rural areas Contrary to the approach in urban areas, in rural areas stratified samples were selected. Each rural Council was divided into 10 geographical areas with a comparable number of facilities. These geographical areas were primarily based on the existing Chiefdoms. However, if the number of PHUs per Chiefdom was significantly below average, contiguous Chiefdoms with low PHU numbers were clustered. If the number of PHUs per Chiefdom was significantly above average, the Chiefdom was split in two parts. Annex 5 in Volume II provides insight in the number of facilities per geographical area before and after the re-grouping mentioned above. For each rural Council all 10 geographical areas were listed and each area was given a unique number. During the inception workshop two numbers between 1 and 10 were randomly generated in an Excel sheet for each Council and these numbers were linked to the geographical area. This resulted in the selection of the following Chiefdoms per council (table 7). Table 7 Geographical areas selected in each rural Council Council Tonkolili Pujehun Porto Loko Moyamba Kono Koinadugu Kenema Kailahun Kambia Bonthe Bombali Bo Chiefdoms Yoni A, Kunike Barina Soro Gbema, Yakemu Kpukumu Krim (YKK) Kaffu Bullom, Koya (Porto Loko) A Bagruwa, Timdale, Kaiyamba Gbane, Gorama Kono, Gbane Kandor, Mafindor Dembelia Sinkunia, Wara Wara Yagala, Wara Wara Bofadia Small Bo, Kandu Leppiama, Dodo, Simbaru, Malegohun Malema, Penguia, Yawei Magbema B, Masungbala Bendu Cha, Dema, Jong A Libeisaygahun, Sanda Tendaran, Biriwa Komboya, Niawa Lenga, Badjia, Lugbu The Chiefdoms of Koya, Magbema and Jong were split and the letter A or B refer to the subgroup. All facilities in the mentioned Chiefdoms were verified. A full list of facilities verified can be found in Annex 5, Volume II of this report. Additional Weighing As mentioned above, the sample was weighed for PHU level. During the inception workshops, the DMOs and other stakeholders were asked if additional weighing needed to be taken into account, for instance because of ethnicity, economic status, political orientation of different areas or populations within the councils. None of the workshop participants deemed this necessary and therefore no additional weighing was done SAMPLING FOR PATIENT TRACING AND PATIENT SATISFACTION SURVEY Similar to the sampling of facilities, it is not necessary to trace all the patients that visited a health facility to draw conclusions about the validity of the reported patient numbers and the extent to which patients are satisfied with the services rendered. In the independent verification of the Sierra Leone PBF program, 8 patients were randomly chosen in each facility.

30 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 28 Figure 5 Selection process for patient sampling Select Month Based on Simple Random Sampling (SRS) a number between 1 and 12 was selected, representing the month in which the patient is selected. Numbers were computer generated (Excel) and printed and handed over to the enumerator before his/her departure to the PHU. Select patient Using (SRS) a number from 1 to 25 was generated (Excel) and the patient appearing on the corresponding entry in the register was selected. Numbers were computer generated (Excel) and printed and handed over to the enumerator before his/her departure to the PHU. If entry is inadequate If name and address details of the patient were inadequate to trace the patient, the next patient in the register was taken, until a valid entry was found. If the patient was not from within the catchment area (e.g. from a neighbouring country), the next patient was selected. For reasons of privacy, it was decided not to include family planning services in the patient tracing. Due to expected difficulties in finding patients (see below) and challenges in relation to the reliability of patient-based feedback about OPD visits that took place nearly two years before the interview, OPD visits for under-fives were also not included. Therefore, patient tracing was done for: Antenatal care 4 th visits Deliveries Postnatal care 3 visits Children fully immunised For each of these four indicators 2 patients were randomly selected. Working procedure to trace selected patients As described earlier, the enumerators sought collaboration with opinion leaders in the Chiefdoms to get permission and collaboration in tracing before actually visiting the patients. The community leaders were asked to assist in the identification of the selected patient. The health committee also assisted in the identification of the patients and, if applicable, referred the enumerator to a community health worker in the residential area of the selected patients. 3.6 QUALITY ASSURANCE PRIOR TO DATA COLLECTION Before qualifying for the external verification, Cordaid selected local partner organisations, with proven competence in field research, interviewing and quality assessment. Each of these organisations provided members for the verification teams. Members were chosen after an elaborate selection procedure, which looked at background, general understanding of the health sector, minimal education levels, communication skills, commitment to fieldwork and willingness to travel long distances. To reduce the likelihood of errors and omissions during counting of patient numbers, assessment of cross cutting issues and interviews, much attention was given to the development of standardised tally sheets, reporting sheets, semi-structured interview sheets, electronic data entry forms, etc. Reporting sheet and interview sheet also contained additional instructions to the enumerators. All developed tools are available for use in subsequent external verifications.

31 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 29 Another important part of the preparation for data collection consisted of an 6-day training for coordinators and enumerators. This training took place from 6 to 11 January 2014 in Bo and focused on general information with regard to the health system and data collection, data entry in EPI info, recounting patients from the different registers, interpretation of PHUF forms, interpretation of internal verification forms and interview techniques. At the end of the training was a simulation exercise was done. This allowed for field-testing of all instruments and procedures by the enumerators and coordinators DURING THE VERIFICATION The actual external verification exercise start immediately after finalising the training. During the first two weeks, enumerators and coordinators operated in teams of two. This enabled them to correct each other and ask for a second opinion in case of doubt. After two weeks, enumerators and coordinators visited PHU and patients individually. However, several procedures were developed to allow the enumerators to consult their colleagues or supervisors. First of all, the verification team coordinators could be contacted by phone for consultation. In addition, it was agreed that the enumerators took pictures of registers and forms if they doubted the correctness of entries. These pictures were then discussed with the coordinators, and, if necessary, forwarded for a second opinion to the supervisors. During the fieldwork the coordinators crosschecked enumerators, when receiving their reports. This included an assessment of the completeness of forms and validation when entering data in computer. In case of large deviations the coordinators contacted the enumerator and discussed whether reverification was necessary. The national supervisors joined each verification team during a period of approximately one week. This allowed them to apply corrective measures and provide additional instructions to the coordinators and enumerators. In addition they validated data entry by analysing large deviations and discussing inconsistencies with the team coordinators AFTER DATA COLLECTION The final data that were entered in EPI-info were exported to Excel and STATA for further analysis. However, before starting the analysis, the database was validated in two ways: 1. Outliers and irrational entries, e.g. blank entries for external verification or zero-scores for all indicators in HMIS, were identified. For these cases, the original verification sheets were examined and, if necessary, corrections were made. Eight cases were found and corrected. 2. A sample of five PHUs per district was taken (representing 25% of all PHUs) and all entries on the original verification sheet of those PHUs were crosschecked with the data appearing in the database. Less than 0.2% errors were found. If applicable, data were corrected. 3.7 RELIABILITY AND SIGNIFICANCE RELIABILITY OF DATA As described in the previous paragraph, all data were validated before analysis took place. During validation, errors were found in approximately 0.2% of all data entries. In view of this low percentage and the fact that found errors were corrected, the internal reliability of the database for the external verification can be considered high EQUAL DISTRIBUTION OF FACILITIES Figure 6 shows the final number of PHU s selected for verification.

32 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 30 Figure 6 Number of PHU selected per district The percentage of the total number of PHU s in each district varies from 17% to 23%, with a national average of 20,2%. The selected facilities may thus be considered equally distributed over the districts. The final number of facilities per district that was externally verified deviated from the number that was initially selected. This was the case in the following districts: In Western Area one facility (Aberdeen Women) was part of the sample, but did not exist in 2012 and was therefore not included in the external verification; In Bombali, both Fulla Town MCHP and Kayongoro MCHP had closed and could not be verified; In Kono three facilities, which were initially not part of the sample, were found to be in the selected Chiefdoms and part of the PBF programme in 2012 (Boroma MCHP, Kayongoro MCHP and Kania MCHP). These were included. Because three facilities were left out of the external verification, and three new ones were added, the total number of facilities verified remained 235. Table 8 shows that in each district, all types of facilities were included in the external verification. For 6 out of the 235 facilities no formal PHU-classification could be provided by the DHMT. Table 8 Distribution of PHU type per district Row Labels CHC CHP MCHP Bo Bombali Bonthe Kailahun Kambia Kenema Koinadugu Kono Moyamba Port Loko Pujehun Tonkolili Western Rural Western Urban Grand Total

33 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE SIGNIFICANCE OF PHU DATA The sampling exercise that took place during the inception workshop resulted in 235 PHUs selected out of the total of that received a payment as part of the PBF programme in The question is how significant conclusions are, that can be drawn on the basis of the findings in the sample. For this purpose, a sample size calculation can be used (figure 7). Such a calculation is generally applied in statistical research 16. Figure 7 Formula for calculating sample size ( ) ( ) The formula indicates that, with a confidence level of 95% and a sample of 235 PHUs, the outcomes for the entire population (1,163 facilities) can be predicted with a 6% precision. In other words: If 50% of the facilities in the sample reported stock-outs of registers, there is a 95% chance that the score will be between 44% and 56% for all PHUs in the PBF programme. Therefore, conclusions from the selected sample can be considered powerful. The conclusions of the external verification may be considered valid for all PHUs in the PBF programme SIGNIFICANCE OF PATIENT SATISFACTION DATA Similar to the calculation for statistical significance that was made for the facility sample (see figure 7), a calculation can be made for the sample of patients. In total, 1,234 patients were interviewed. With this number and a confidence level of 95%, the outcomes for the entire population (estimated at 6 million for 2011 according to the United Nations) can be predicted with a 2.5% precision. In other words, it can be assumed that if the survey shows that 30% of the patients had to pay for services, there is a 95% chance that between 27.5% and 32.5% of the entire population of Sierra Leone had to pay for services. Therefore, conclusions from patient tracing and the satisfaction survey can be considered powerful. The conclusions of the external verification may be considered valid for the population of Sierra Leone COMPLETENESS OF DATA In order to analyse the consistency between external verification, internal verification, HMIS and the PHU F-reports, all information from mentioned sources has to be available and accessible. However, the external verification encountered problems with completeness of information. Figure 8 provides an example on the availability of data for family planning, one of the indicators in the PBF programme. 16 Naing L, Winn T and Rusli BN. Sample size calculator for prevalence studies, Version Available at: Daniel, WW (1999). Biostatistics: A Foundation for Analysis in the Health Sciences. Wiley & Sons, New York

34 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 32 Figure 8 Percentage completeness of family planning records per districts HMIS IV EV PHUF As figure 8 shows, unavailability of records was an issue for all of the four data sources. However, the availability of HMIS data (from automated systems at DHMT level) was found to be the most challenging. In Freetown and Western Area only 39% and 32% of all HMIS data on Family Planning was available. In Port Loko no HMIS records over 2012 were found. Similar patterns were found for all other indicators. Table 9 summarises the reasons why records were not found. Figure 9 Reasons for unavailability of records per information source Source HMIS Internal Verification External Verification PHU F-report Reasons for unavailability No data could be retrieved from servers at DHMT level and no back-up data could be found at national level No data (soft or hard copy) could be retrieved at DHMT level and no backup data could be found at national level No registers or incomplete registers could be found at the PHU A copy of the PHU F-forms could neither be found at facility level, nor at the district level If no records were found for an indicator in a specific source of information, these cases were not included in further analysis. For that purpose a data cleaning exercise was carried out. In Chapter 4 the impact of missing data is calculated. 3.8 DATA ANALYSIS SYSTEMS USED All data in gathered during the external verification was entered into a relational EPI Info Database 17, which was specifically developed for the purpose of this assignment. Apart from data entry, EPI Info 17 Epi Info is used in the MOPH, and various officers within the DPPI are conversant with this software programme. They will be able to continue to apply the software developed for this external verification in the future.

35 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 33 was also used for spatial analysis with GIS Maps. For more sophisticated analysis, a CSV file was created, that was exported to Microsoft Excel and STATA PRIMARY FACILITY OUTPUT DATA Comparison of patient numbers for the six output indicators in the programme is the most crucial element in this assignment. The computation of differences or deviations across different data sources was an initial step in the analysis of reporting accuracies. These statistics were stratified according to key study factors such as District, Council and Facility Type. A second step was the calculation of deviations beyond 25% of reported values at different levels DATA FROM PATIENT AND KEY INFORMANT INTERVIEWS Simple descriptive statistics were obtained from the analysis of the patient survey data and structured interview data from key informants. In particular: Percentage / number of patients positively identified and confirmed using the facility Percentage / number of patients who paid for services at the facility during the visit Average satisfaction scores Percentage of patients satisfied with the services received during the visit Number of joint internal verification sessions performed Trainings received PBF payments received These descriptive statistics were stratified according to study factors such as facility type, service received, region, district, age and sex of respondents. Furthermore, regression analysis was used to check for the association between the above study factors (including whether patients paid for services) and patient satisfaction scores. This was also done across different data sources, eg across patient satisfaction data, cross cutting issues and key informant interviews. 3.9 EXTERNAL VERIFICATION IN THE HOSPITALS Apart from the external verification at PHU-level, the external verification team also performed an external verification in the two tertiary hospitals in the PBF programme. In order to assess the impact of the PBF programme, the external verification was also done in two non-pbf hospitals. Three international consultants performed the external verification in hospitals. The hospitals that were visited include: Macauley Hospital (non-pbf) Rupoka Hospital (non-pbf) Princess Christian Maternal Hospital (PBF) Ola During Children s Hospital (PBF) Information for the external verification in hospitals was obtained in two ways. First, using exactly the same instrument that was used during internal verification, a re- assessment was done. Secondly, key staff, members of the management team and the in charge of the hospitals were interviewed. For the interviews a structured questionnaire was created. Some of the elements of the hospital checklist for PBF, like cleanliness, could not be re-assessed retrospectively. Instead, the consultants looked at the present situation. For objectively verifiable indicators, like the availability of patient records, the last quarter of 2013 was assessed. Data were compared with previous assessments. Because no thorough baseline was done, it is difficult to draw conclusions about the actual impact of the PBF program at facility level. However, the in-depth interviews with the management teams in the four hospitals provide some qualitative information regarding the impact.

36 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 34

37 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 35 4 EXTERNAL VERIFICATION FINDINGS 4.1 INTRODUCTION The External Verification (EV) consisted of two components of checking the Internal Verification (IV): Recalculating the output indicators based in recordings in the registers in health facilities; Repeating the verification of crosscutting (quality) indicators in PHUs and repeating the verification of hospital (quality) indicators In addition the EV performed: Patient tracing and satisfaction survey Hospital verification of two non-pbf hospitals The methodology is described in Chapter 3 of this report. This Chapter describes the findings, while in chapter 7 these findings are put in a bigger context INDICATORS Output indicators The PBF project in Sierra Leone focuses on improving a number of indicators from the Basic Package of Essential Health Services, which are part of the Free Healthcare Initiative. It pays a fee-for-service for: 1. Women of reproductive age using modern family planning (BPEHS 7.2) 2. Pregnant women receiving four antenatal consultations (ANC-IV) (BPEHS 7.1.1) 3. Deliveries conducted under safe conditions (BPEHS 7.1.2) 4. Women receiving three postnatal consultations (PNC-III) (BPEHS 7.1.4) 5. Children under one year of age receiving full and timely course of immunizations (BPEHS 7.6) 6. Outpatient visits with curative services for children under five years old according to Integrated Management of New-born and Childhood Illness (IMNCI) protocol (BPEHS 7.7) Crosscutting Issues In order to stimulate overall performance of the health facilities the PBF project provides additional incentives for so-called cross cutting issues, which address quality. The following indicators provide a multiplier for the total payment based on fee-for-service: 1. Recording of staff attendance. 2. Timely submission of DHIS, attendance and PBF reports. 3. A functioning Health Management Committee. 4. Display of up-to-date performance information at the facility. 5. All paperwork kept in good order at the facility. 6. Maintenance of appropriate standards of cleanliness. 7. Appropriate procedures for medical waste management in place and being observed. 8. Maintenance of up-to-date and accurate drugs records. 9. No stock-out of essential drugs for the three childhood diseases with highest mortality HOSPITAL PBF In 2012 a pilot was started with hospital PBF in two hospitals in Freetown, closely involved in maternal and child health, i.e. the Ola During Children s Hospital and the Princess Christian Maternity Hospital. The primary focus of hospital PBF is on performance indicators with regard to quality of services, measured by means of a composite performance score in eight domains as shown in the table below. Each domain has 3 7 indicators, which are either objectively verifiable or more qualitative in nature. For the performance assessment an extensive checklist is used.

38 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 36 Table 9 Domains for assessment of hospital performance Financial management Patient care Human resources management General organization Pharmacy management and prevention of drugs stock out Hygiene and sanitation Health care services Laboratory 4.2 OUTPUT INDICATORS IN PHUS VARIATION BETWEEN SOURCES OF INFORMATION In principle the four sources of information traced in this external verification (F-report, HMIS, IV and EV) should show no differences, as they all go back to the same basic source, i.e. the registers in the health facilities. F-reports, IV and EV directly used the registers as source; HMIS is based on the F- reports. The EV found the following figures for the six output indicators in the PHUs (see table 10 below) in the F-reports, the HMIS, the IV and the EV. The totals are for four sampled months in 2012 and are not representative for the annual figures, as different months were sampled in the various districts. Table 10 Totals per output indicator sampling 4 months 2012 Indicator F-reports HMIS IV EV Family planning 10,817 11,328 17,497 10,105 Antenatal Care IV 12,689 10,477 12,672 11,361 Deliveries in PHUs 9,554 8,342 9,521 10,060 Post Natal Care III 10,613 8,282 9,644 7,063 Fully vaccinated before 1 year 10,434 10,829 13,801 9,511 Outpatient cases children under five 115, , , ,893 Figure 10 below shows the comparison between sources of information for indicators. The EV figures for all indicators are put at 100%. It transpires that (with exception of deliveries) IV figures are 12% - 73% higher than the EV, while the F-reports show only one indicator (PNC) as outlier and others closer to the EV. Also HMIS is closer with a variation between 83% and 124% the EV figures.

39 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 37 Figure 10 Comparison Output indicators per source of information Figure 11 below shows the same information, but now per indicator. This figure shows that the Family Planning attendance recorded by IV and the Fully Vaccinated recorded by IV are (statistically significant) outliers. ANC, deliveries and OPD indicators show less prominent differences among sources of information. PNC-III shows differences of 17% - 50% in sources of information compared to EV. Figure 11 Comparison sources of information per indicator It is relevant to disaggregate the differences between sources of indicators per district, to identify whether some districts have consistently deviating figures, which could influence the national totals. For example, in Bonthe, Kambia, Moyamba and Western Area the reported numbers for family planning in the IV were considerably higher than the EV, but not in other Councils. In Freetown, Bo Tonkalili and Koinadugu the reported numbers for PNC-III in the IV were much higher than the EV, but less in other districts. The analysis does not prove that certain districts always score much higher for all indicators in the IV compared with the EV. Or in other words: differences are not systematic and can be found anywhere. The Council reports in the annex 4, Volume II of this report, provide detailed information on deviations per indicator.

40 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE INFLUENCE OF MISSING DATA ON CALCULATIONS As has been discussed in the chapter on methodology, missing data were a serious problem in some areas. The verification team tried to assess whether missing data could be a cause for differences between sources of information. Based on the missing data analysis (see Chapter 3) service utilisation figures were extrapolated. Figure 12 below shows that correction for missing data increases the differences in FP, while reducing the differences in ANC (figure 13). Table 11 below shows that for ANC, deliveries and PNC difference get smaller after correction for missing data, but bigger for FP, OPD and EPI. The verification team therefore concluded that missing data could be ruled out as general cause for differences between sources of information. Figure 12 Extrapolation of service utilisation FP based on missing data Figure 13 Extrapolation of ANC service utilisation figures based on missing data

41 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 39 Table 11 Extrapolation of all service attendance based on missing data % missing F -report HMIS IV EV Total plus missing % missing Total plus missing % missing Total plus missing % missing Total plus missing Family 4% 11,211 15% 13,394 9% 19,259 4% 10,507 Planning ANC IV 3% 13,062 15% 12,310 9% 13,948 8% 12,303 Delivery 5% 10,079 17% 10,002 10% 10,585 6% 10,655 PNC III 5% 11,122 19% 10,244 11% 10,889 24% 9,273 Fully 4% 10,862 15% 12,756 9% 15,226 8% 10,288 Vaccinated OPD under 5 5% 121,716 17% 171,856 8% 149,134 4% 121, DIFFERENCES IN DATA IN FACILITIES In the previous section aggregated numbers have been discussed. Such figures are influenced by the statistical phenomenon of regression towards the mean. Differences between one source of information and another can be positive or negative, i.e. showing more or less client contacts. Adding all numbers blurs the view on differences at the source. The verification team looked at the frequency and size of deviations of records to assess accuracy of recording at the source. Figure 14 below shows the distribution of facilities with deviating figures between sources of information for family planning (14a) and OPD under 5 (14b). The green area shows differences of 10% and less; the yellow area differences between 10% and 25% and the red area shows differences of over 25%. In principle, all bars in the figure should be green; the redder the bar, the more serious disagreement between data sources at facility level. Figure 14 Distribution of facilities by differences Figure 14a Family Planning differences in data 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Distribution of Facilities By Differences - Family Planning HMIS and IV HMIS and EV IV and EV HMIS and F- Report 33 EV and F- Report Above 25% 10%-25% Below 10%

42 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 40 Figure 14b OPD under 5 differences in data 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Distribution of Facilities By Differences - OPD U HMIS and IV HMIS and EV IV and EV HMIS and F- Report EV and F- Report Above 25% 10%-25% Below 10% Table 12 below shows the differences over 25% (the red segments in the graphs above) for all indicators. Table 12 Differences between data sources above 25% in PHUs HMIS and IV HMIS and EV IV and EV HMIS and F rep F-Report and EV Family Planning 68% 53% 71% 46 % 43% ANC IV 60% 57% 61% 28% 51% Delivery 42% 36% 33% 14% 27% PNC III 65% 69% 67% 40% 61% Fully Vaccinat 51% 62% 63% 71% 63% ed OPD under 5 60% 69% 49% 46% 59% Average 58% 58% 57% 33% 51% Internationally in external verifications plus or minus 10% difference between sources of information is considered as big, given the fact that they all use the very same root source, i.e. the patient registers in the facility. Differences of more than 25% are hardly seen. In Sierra Leone there are serious issues of data consistency, as seen in the comparison between all sources of data: HMIS, F-rep, IV and EV. On average over half of entries differ more than 25% between sources of information, with exception of HMIS and F-reports (which should be simply retyping data from a paper sheet into a computerised data system) DIFFERENCES PER LEVEL OF FACILITY In table 13 below the differences between data sources are shown for family planning, split out per facility type. The association between facility type and the differences between HMIS and Internal Verification is significant. CHCs more often have differences below 10% and MCHPs more often have differences above 25%. The higher the level of the health facility the more concordance between figures from the four data sources.

43 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 41 This can to some extent be attributed to smaller numbers of clients in peripheral health facilities. For example, if in one month there are 2 deliveries recorded on the F-form of an MCHP and 3 are reported in the HMIS, the deviation is 50%, while the absolute number difference is only 1. However, in general the conclusion has to be that the quality of data aggregation and reporting is not adequate in MCHPs and only slightly better in higher-level facilities. Table 13 Differences between data sources FP per level PHU CHC CHP MCHP Total HMI S and External Verification HMI S and I nternal Verification Below 10% 10% to 25% Above 25% Total Below 10% 10% to 25% Above 25% Total DIFFERENCES PER GEOGRAPHICAL AREA Data analysis shows no major variation in geographical as regards facilities with over 25% deviation between data sources. Figure 15 below shows differences between HMIS and external verification, with no significant differences between areas. Data quality is an issue all over the country. Figure 15 Geographical spread of facilities with differences data sources

44 Internal External Internal External Internal External Internal External Internal External Internal External Family Planning ANC OPD Deliveries PNC EPI EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE SATISFACTORY AND UNSATISFACTORY ENTRIES General In the internal and external verification a distinction is made between satisfactory and unsatisfactory records in the PHU. 18 In the PBF Operational Manual no exact criteria were given of these qualifications, but during the inception workshop of the EV they were formulated with inputs from the DMOs. For example, a delivery record is unsatisfactory when no partograph can be shown, or a fully vaccinated record is unsatisfactory when vaccinations are marked with tick, instead of dates. The external verifiers used these standard criteria, while internal verifiers used local criteria. Satisfactory records On average between 83% and 93% of the records in the IV are satisfactory, and between 92% and 96% of the EV are meeting the defined criteria for proper recording. Disaggregating scores per council, as reflected in table 14, shows that incidentally there are variations. For example, Bo DHMT in the internal verification consistently marked all entries as satisfactory, while Bonthe DHMT was very critical. The EV showed a consistent picture for all districts. DHMTs often apply local criteria to determine satisfactory or unsatisfactory entries. Table 14 Percentage satisfactory entries per Council per indicator 19 Council Bo City Council Bo District Coun Bombali District Bonthe District Bonthe municipal Freetown City Co Kailahun Distric Kambia District Kenema City Coun Kenema District Koidu new Sembeh Koinadugu Distri Kono District Co Makeni City Coun Moyamba District Port Loko Distri Pujehun District Satisfactory refers to completeness and correctness of data (e.g. adding address, age, weight, BP). Eligibility refers to the case definition, e.g. vaccination before reaching age of one year. 19 Some fields are left blank, where data were missing

45 Internal External Internal External Internal External Internal External Internal External Internal External Family Planning ANC OPD Deliveries PNC EPI EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 43 Council Tonkolili Distri Western Area Rur Overall Figure 16 below shows that overall there is no major difference between percentages of satisfactory records in IV and EV. The main differences range between 2 and 3.5. Figure 16 Distribution of Absolute Differences between IV and EV The average differences between the internal verification and the external verification for the total group of indicators ranged from -6% to 3.3% for the indicators with a negative difference indicating the percentage in internal verification was lower than that in external verification. (See table 15 below.) However, Bonthe District and Municipal Councils, Koidu New Sembehun and Western Area Rural districts show consistently high differences above 10% across the six indicators. Other councils such as Freetown City Council and Pujehun City Council fluctuate with some indicators having very minimal differences and others having high values. The indicators that had relatively higher observed differences were Deliveries and OPD with averages of 3.3% and 6.0% respectively.

46 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 44 Table 15 Differences between average satisfactory scores (IV and EV) per district District Family Planning ANC OPD Deliveries PNC EPI Bo Kambia Bonthe Kailahun Kenema Kono Moyamba Port Loko Pujehun Tonkolili Western Rural Western Urban Overall Overall, the differences are distributed around the averages between -6 and 3, although a number of outlier facilities have significantly higher differences as shown by the box plot below, in figure 17. Figure 17 Plot box differences satisfactory entries Differences satisfactory entries per level of facility Table 16 below shows the percentages of satisfactory scores by level of health facility. From this analysis no conclusion can be drawn that recording at higher level is consistently better than at lower level. Nearly all levels are around 90% satisfactory entries.

47 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 45 Table 16 Percentage satisfactory records per level health facility CHC CHP MCHP IV EV IV EV IV EV FP 94% 95% 93% 94% 93% 97% ANC 93% 90% 90% 92% 93% 94% OPD 95% 88% 91% 90% 94% 91% PNC 92% 90% 88% 90% 92% 90% Deliveries 89% 93% 79% 86% 83% 90% EPI 96% 91% 92% 95% 92% 90% THE EXTERNAL VERIFICATION OF OUTPUT INDICATORS IN THE PBF SYSTEM SHOWED: CONSIDERABLE, SOMETIME SIGNIFICANT DIFFERENCES EXIST BETWEEN AGGREGATED NUMBERS IN INTERNAL AND IN EXTERNAL VERIFICATION. WITH EXCEPTION OF DELIVERIES, THE AGGREGATED IV FIGURES ARE 12% - 73% HIGHER THAN THE EV. RECORDED ATTENDANCE IN THE IV IS IN THE MAJORITY OF INDICTORS ALSO HIGHER THAN OTHER SOURCES OF INFORMATION (HMIS, F-FORM). IN GENERAL, THE AGGREGATED FIGURES FROM VARIOUS SOURCES OF INFORMATION DIFFER, WHEREBY THE EV SHOWED MOST CONCORDANCE WITH F- REPORTS. THE DIFFERENCES CANNOT BE ATTRIBUTED TO MISSING DATA. AT FACILITY LEVEL FOR ALL INDICATORS THE DIFFERENCES BETWEEN SOURCES OF INFORMATION ARE LARGE, OFTEN MORE THAN 25% HIGHER OR LOWER. THE DIFFERENCES IN RECORDING ARE SPREAD OVER THE COUNTRY AND NOT RELATED TO SPECIFIC DISTRICTS. LOWER-LEVEL HEALTH FACILITIES SHOW LARGER ERROR MARGINS THAN HIGHER- LEVEL FACILITIES. THERE IS NO STATISTICALLY SIGNIFICANT DIFFERENCE BETWEEN IV AND EV AS REGARDS PERCENTAGE OF SATISFACTORY OR UNSATISFACTORY RECORDS. THERE IS NO SIGNIFICANT DIFFERENCE IN SATISFACTORY ENTRIES PER LEVEL OF FACILITY, AND NOT PER DISTRICT. IT VARIES BETWEEN 90% AND 95% SATISFACTORY ENTRIES, WITH EXCEPTION OF DELIVERIES, WHICH SHOW LOWER PERCENTAGE OF SATISFIED RECORDS IN IV (83%). 4.3 CROSSCUTTING ISSUES GENERAL The external verification performed an assessment of crosscutting issues related to quality of care. There are nine indicators, as explained in paragraph 4.1, scoring each either 3 or +3 respectively -4 or +4. In the inception phase the EV team together with representatives of the DHMTs formulated standards for the indicators, to ensure that all enumerators would use similar assessment criteria. Such standard criteria were not applied during the internal verification. Part of the indicators was checked going back to 2012 data, e.g. attendance registers, report submission, minutes of health management committees and administration. Part of the indicators was checked during the EV, like cleanliness and waste management. (See explanation in Chapter 3.) The average scores on crosscutting indicators were significantly higher during the internal verification (last quarter of 2012) than during the external verification, as shown in figure 18 below.

48 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 46 Figure 18 Crosscutting issues IV and EV Figure 19 Distribution crosscutting issues scores in EV Statistical analysis shows a normal distribution in the external verification (in figure 19 above), which is missing in the internal verification (figure 20 below). In the internal verification in some districts many facilities received a maximum score, with some outliers. Figure 20 Distribution of crosscutting scores in IV

49 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 47 All facilities in Kono district received the maximum score for all crosscutting indicators in the internal verification of the last quarter of 2012, but not in the external verification, as shown in figure 21 below. Figure 21 Kono District Comparing Crosscutting Indicators IV and EV SPECIFIC INDICATORS Scores on indicators, which used similar reference documents in IV and EV (e.g. attendance registers, or timely submission HMIS reports), were mostly lower in EV. Especially the indicator on administration scored very low (with exception of Port Loko), as in many health facilities documentation and registers over 2012 went missing (figure 22). Figure 22 Comparing crosscutting indicator Administration IV and EV None of the other indicators assessed with present information in 2014 in the EV scored higher than the IV in 2012 (e.g. cleanliness, drug records or waste management). Maybe the standardised assessment in the EV applied stricter criteria than the less-structured assessment in the IV.

50 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 48 The scores for stock outs of medicines were considerably lower in EV compared with IV. This may reflect a worsening supply of essential medicines in 2014 in the country compared with Figure 23 Comparison crosscutting indicator Stock Outs IV and EV THE EXTERNAL VERIFICATION FOR CROSSCUTTING INDICATORS SHOWED: THE SCORES FOR THE CROSSCUTTING ISSUES IN THE EXTERNAL VERIFICATION WERE CONSISTENTLY LOWER THAN IN THE INTERNAL VERIFICATION IN NEARLY ALL DISTRICTS, FOR NEARLY ALL THE INDICATORS. IN THE EXTERNAL VERIFICATION STANDARDISED ASSESSMENT CRITERIA WERE APPLIED, REDUCING THE CHANCES OF PERSONAL BIAS. THOSE CRITERIA MIGHT HAVE BEEN STRICTER THAN APPLIED IN THE INTERNAL VERIFICATION. DUE TO THE TIME LAPSE BETWEEN 2012 AND 2014 DIFFERENCES MAY HAVE BEEN CREATED, E.G. MISSING REGISTERS, LEADING TO LOWER SCORES. WORSENING SUPPLIES OF MEDICINES MAY HAVE CAUSED LOWER SCORES IN AVAILABILITY OF ESSENTIAL MEDICINES IN 2014 COMPARED WITH IN THE INTERNAL VERIFICATION SOME DHMTS GAVE PHUS MAXIMUM SCORES FOR ALL INDICATORS. THIS NEVER OCCURRED IN THE EXTERNAL VERIFICATION. 4.4 HOSPITAL EXTERNAL VERIFICATION The external verification team performed an external verification of PBF in the two hospitals in Freetown, which are included in the pilot on hospital-based PBF and two hospitals not in the PBF. In paragraph 4.1 the domains of inspection are explained. Per domain a series of 8 15 questions is be answered, which each give a minus or plus score. The total maximum score is 1,000 points OLA DURING CHILDREN HOSPITAL Figure 24 below shows the scores for Ola During Children Hospital (ODCH) comparing the latest IV in 2013 and the EV. The EV scored slightly higher in most domains and much higher in the general domain. During the latest IV the administration was found not to be in order and scored very low.

51 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 49 Figure 24 Ola During Children Hospital IV and EV Figure 25 Trend in IV scores in Ola During Children Hospital Figure 25 shows the trend of scores. Quarter 2 and quarter 3 in 2013 were done in one assessment. The ODCH hospital always scored 85% - 95% of the maximum, except in latest IV. In the EV ODCH scored 79% PRINCESS CHRISTIAN MATERNITY HOSPITAL Figure 26 below shows the comparison between the latest IV in 2013 and the EV for Princess Christian Maternity Hospital (PCMH). Here scores in most domains were slightly lower in the EV compared to the IV.

52 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 50 Figure 26 Princess Christian Maternity Hospital IV and EV The figure 27 below shows the trend analysis for PCMH. The hospital scored in the past between 82% and 90% of the maximum scores. In the EV the score was 84%. Figure 27 Trend analysis Princess Cristian Maternity Hospital NON-PBF HOSPITAL The MOHS selected two secondary level hospitals in Freetown not in the PBF programme, to perform a similar external verification. These hospitals provide general services at the primary referral level, unlike the two PBF hospitals, which are specialised tertiary hospitals. Figure 28 below shows the comparison of scores.

53 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 51 Figure 28 Comparison PBF and non-pbf hospitals in EV One non-pbf hospital (Macauley) scored only 4% of the maximum scores and the other (Rokupa) 59%. In some areas Rokupa hospital scored even higher than one of the PBF hospitals. According to information Rokupa has more income from patients, more support from the Council and other support to make ends meet. THE EXTERNAL VERIFICATION OF THE HOSPITAL PBF FOUND THAT: THE EV TEAM GAVE SLIGHTLY HIGHER SCORES TO ODCH COMPARED TO THE LATEST IV (79% VS. 61%), BUT LOWER THAN IN OTHER IVS (85%-95%). THIS WAS DUE TO THE INDICATOR ADMINISTRATION THE EV TEAM GAVE SLIGHTLY LOWER SCORES TO PCMH COMPARED TO THE LATEST IV (84% VS. 89%), BUT WITHIN THE RANGE OF OTHER IVS (82%-89%) NON-PBF HOSPITALS SCORED LOWER THAN PBF HOSPITALS, BUT ONE OF THOSE SCORED ONLY SLIGHTLY LOWER, WHILE THE SCORE OF THE OTHER HOSPITAL WAS WIDE OFF RANGE.

54 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 52

55 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 53 5 PATIENT TRACING AND SATISFACTION SURVEY 5.1 PATIENT TRACING The patient tracing and satisfaction survey served two purposes: 1. Confirming existence of patient or clients, to check on reliability of recording at facility level; 2. Getting feedback from patients with regard to their experiences in the health facilities. The methodology for tracing patients is described in Chapter 3 of this report. Per PHU eight patients/clients were randomly selected from registers to be traced and interviewed. In total in all health facilities 1,680 persons were identified for tracing. Out of this total, community leaders, neighbours or family members could not identify 124 persons as existing persons, which is 7.4% of the total persons identified for tracing. Table 17 below shows no clear pattern of nonidentifiable persons. Koinadugu District Council and Bo City Councils are outliers in this statistic. Table 17 Percentage of persons who could not be traced in EV Persons identified for tracing Number not identified Percentage not identified Bo City Council % Bo District Council % Bombali District Council % Bonthe District Council % Bonthe municipal Council % Freetown City Council % Kailahun District Council % Kambia District Council % Kenema City Council % Kenema District Council % Koidu new Sembehun City % Council Koinadugu District Council % Kono District Council % Makeni City Council % Moyamba District Council % Port Loko District Council % Pujehun District Council % Tonkolili District Council % Western Area Rural District % Council Total 1, % Of the 455 persons who were not interviewed in the EV patient tracing survey, 4% had died, 64% had moved, and 32% were unknown in the community, as shown in table 18 below. The long time between the service delivery (13 months to two years) may have contributed to a larger number of patients who could not be interviewed. The EV team concludes that it highly unlikely that PHUs recorded ghost patients to inflate the numbers of attendance and therefore the payments through PBF.

56 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 54 Table 18 Reasons for not interviewing persons in EV Districts Died Exists But Relocated Untraceable, No Information Bo 8.33% 60.42% 31.25% Bombali 0.00% % 0.00% Bonthe 11.11% 66.67% 22.22% Kailahun 0.00% 61.11% 38.89% Kambia 0.00% 83.33% 16.67% Kenema 3.33% 90.00% 6.67% Koinadugu 0.00% 12.00% 88.00% Kono 11.11% 77.78% 11.11% Moyamba 0.00% 73.85% 26.15% Port Loko 0.00% 69.70% 30.30% Pujehun 6.25% 75.00% 18.75% Tonkolili 5.88% 70.59% 23.53% Western Rural 4.00% 40.00% 56.00% Western Urban 0.00% 36.67% 63.33% Grand Total 4.18% 63.57% 32.25% 5.2 PATIENT SATISFACTION SATISFACTION SCORES The enumerators of the EV interviewed in total 1,233 persons after explaining the purpose of the survey and confidentiality of obtained information. These persons were women who attended services, or parents (generally mothers) of children who were vaccinated. The average score of satisfaction was 7.3 (out of 10) with the lowest average score of 4.1 in Koidu New Sembehun City Council and the highest average score of 9.9 in Port Loko. Table 19 Patient satisfaction scores and contributing factors Average satisfaction score Percentage which received friendly treatment Percentage with reasonable waiting time Percentage which received prescribed medicines Bo City Council Bo District Council Bombali District Council Bonthe District Council Bonthe municipal Council Freetown City Council Kailahun District Council Kambia District Council Kenema City Council Kenema District Council Koidu New Sembehun City Council Koinadugu District Council Kono District Council

57 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 55 Average satisfaction score Percentage which received friendly treatment Percentage with reasonable waiting time Percentage which received prescribed medicines Makeni City Council Moyamba District Council Port Loko District Council Pujehun District Council Tonkolili District Council Western Area Rural District Council Grand Total Higher satisfaction was correlated with short waiting times, kind attitudes of staff and availability of medicines. There is no baseline survey available to compare the potential increase in patient satisfaction. However, compared to international standards patient satisfaction is good SATISFACTION SCORES PER LEVEL OF FACILITY Table 20 below shows the patient satisfaction per level of facility in the Councils. Contrary to the expectation, patients are in general more satisfied with the services provided by lower level facilities than provided by higher-level facilities. CHC score lower in staff attitude, waiting time and medicines available. The average satisfaction scores were much higher for the MCHPs. The proportion of facilities with average client satisfaction scores above 7.5 is 67% for MCHPs and 54% for CHPs. Nearly two thirds of facilities with average scores above 7.5 are MCHPs. Table 20 Patient satisfaction per level of facility District CHC CHP MCHP Total Bo City Council Bo District Council Bombali District Council Bonthe District Council Bonthe municipal Council Freetown City Council Kailahun District Council Kambia District Council Kenema City Council Kenema District Council Koidu New Sembehun City Council Koinadugu District Council Kono District Council Makeni City Council Moyamba District Council Port Loko District Council Pujehun District Council Tonkolili District Council Western Area Rural District Council Total

58 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE PAYMENT FOR SERVICES As free health care for reproductive and child health is important for Sierra Leone, the EV also asked for details of payments made for services to pregnant mothers and children. Table 21 below shows that 146 out of the 1,233 persons interviewed (12%), were asked to pay for services, with a variation between 0% and 52%. The amounts paid are reflected in table 22. The average was 7,881 LE, with a variation between 200 LE and 50,000 LE. Table 21 Persons interviewed who were asked to pay for services Number Percentage Bo City Council 4 17% Bo District Council 2 3% Bombali District Council 1 1% Bonthe District Council 6 8% Bonthe municipal Council 0 0% Freetown City Council 2 5% Kailahun District Council 38 52% Kambia District Council 1 1% Kenema City Council 1 2% Kenema District Council 29 21% Koidu new Sembehun City Council 6 35% Koinadugu District Council 2 3% Kono District Council 19 22% Makeni City Council 0 0% Moyamba District Council 15 19% Port Loko District Council 0 0% Pujehun District Council 6 13% Tonkolili District Council 1 1% Western Area Rural District Council 3 5% Total % Table 22 Average, minimum and maximum amounts paid District Average Amount Le Min Amount Le Max Amount Le Bo 13,600 3,000 25,000 Bombali 15,000 15,000 15,000 Bonthe 10,833 5,000 30,000 Kailahun 6, ,000 Kambia 5,000 5,000 5,000 Kenema 5, ,000 Koinadugu 17,500 15,000 20,000 Kono 8, ,000 Moyamba 8,958 5,000 20,000 Pujehun 18, ,000 Tonkolili 5,875 1,000 10,000 Western Rural 1,000 1,000 1,000 Western Urban 10,000 10,000 10,000 Grand Total 7, ,000

59 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 57 Of the patients who paid, 37 paid for general consultation, 82 for MNCH services and 44 for medicines. Payments were sometimes related to purchase of exercise books as patient records, gloves, soap, etc. 5.3 PATIENT TRACING BY COUNCILS AND DHMTS In the EV interviews 44% of the Councils and 46% of the DHMTs mentioned they did some type of patient tracing, although not formalised. Some use HMCs to assist in this effort. Patient tracing is part of the PBF Operational Manual, but no tools have been developed. None of the DHMTs could show results of the tracing and therefore the EV team could not make a comparison with findings of previous patient tracing activities. THE EXTERNAL VERIFICATION OF THE PATIENT TRACING AND SATISFACTION FOUND THAT: 92.6% OF THE PATIENT/CLIENTS EARMARKED FOR TRACING COULD INDEED BE IDENTIFIED EITHER BY MEETING THE CLIENTS IN PERSON, OR BY IDENTIFICATION BY A MEMBER OF THE COMMUNITY. THERE IS NO REASON TO BELIEVE THAT PHUS RECORDED GHOST PATIENTS TO INFLATE THE NUMBERS OF ATTENDANCE. THE AVERAGE SATISFACTION SCORE OF CLIENTS WAS 7.3 (OUT OF 10), WITH A VARIATION BETWEEN 4.1 AND 9.8. CLIENT SATISFACTION WAS STRONGLY RELATED TO SHORT WAITING TIMES, FRIENDLY TREATMENT, AVAILABILITY OF MEDICINES AND NON-PAYMENT FOR SERVICES (FREE HEALTH CARE. 12% OF PATIENTS INTERVIEWED HAD TO PAY FOR SERVICES, ALTHOUGH THEY WERE SUPPOSED TO BENEFIT FROM FREE HEALTH CARE.

60 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 58

61 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 59 6 SYSTEMS ASSESSMENT 6.1 INTRODUCTION The External Verification team performed a system analysis as per Terms of Reference. The EV enumerators collected information from PHUs and interviewed in-charges and from health management committees. The EV coordinators collected information from DHMTs and Councils and interviewed relevant staff. The international consultants interviewed several parties at national level in MOHS, MOFED, NGOs, civil society, etc. Most topics were already discussed in the Inception Report of the EV and in the Issues Paper for the Validation Workshop in March The inception report discussed issues of system design 20. Those will not be repeated here. This final report analyses the system as it is implemented in practice. PBF programmes in general consist of six building blocks, which together strengthen the health system and produce better healthcare and better services utilisation. Figure 29 below shows the building blocks. In this chapter topics are discussed per building block. Figure 29: Building blocks of RBF Accessibility and Equity Autonomy and Accountability of Health Institutions Community Involvement Contracting Indicators and Monitoring Separation of functions 6.2 ACCESSIBILITY AND EQUITY Free Healthcare was introduced in 2010, and was supported by several donors an agencies, e.g. through human resources management, provision of medicines, etc. Free health care has resulted in considerable increase in service delivery in reproductive and child health services, although recently there has been a levelling off of service utilisation at a higher level than before the introduction of Free Healthcare. The PBF programme works complementary to Free Healthcare, and offers to health facilities a compensation for the loss of income through patient fees. The programme has been successful in this, as according to interviewed respondents the income from PBF is much higher than from patient fees. However, late payment affects this element, as will be discussed below in paragraph Bonuses may contribute to improved staff attitudes for service delivery although there is no baseline study to compare present patient satisfaction figures. Free Healthcare is provided in the whole country and does not target specific vulnerable groups or specific geographic areas. The PBF programme provides an equity bonus to health facilities and personnel in remote districts. The equity bonus ranges from 0% (e.g. Freetown an Bo), 20% (e.g. Port Loko and Pujehun), 30% (e.g. Kailahun), 40% (e.g. Kenema and Koinadugu) to maximum 50% (Bonthe). This equity bonus could reduce staff turnover and motivate people to continue working in 20 Cordaid (2012), External verification Performance Based Financing in the health sector in Sierra Leone, Inception Report

62 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 60 their duty station. However, no baseline exists to compare the present staffing levels. The EV could not measure an impact of the remoteness bonus. In interviews the retention effect of extra bonuses for health workers in remote areas was not clear, especially because of payment problems. 6.3 AUTONOMY AND ACCOUNTABILITY PHUS CAPACITIES In the PBF approach, decentralisation to the facility level of planning, budgeting implementation and accounting for small projects, is at the heart of the system. This is new for the health system in Sierra Leone, which is still building up after a long period of decline. Not only the numbers of staff are insufficient (often only one qualified staff in lower level PHUs), but also the level of training is low (e.g. MCH-aid). DHMTs rank human resources problems highest in the list of challenges in the health sector. This creates challenges for the tasks decentralised to the PHUs. High turnover was noticed both in health facilities and in DHMTs. When the PBF programme started, training was provided to DHMTs and health staff through cascade training. Most new staffs are dependent on on-the-job orientation in PBF, provided by the DHMT. In 77% of PHUs visited during the EV, one or more of staff members had been oriented in PBF in the past. Of the PHUs visited 48% of present in-charges understood PBF system completely and 62% of health workers understood bonus calculations applied in PBF. 82% of the staff understood the quality aspects assessed in the crosscutting indicators. Capacity building therefore was mentioned as key issue in the validation workshop in March PLANNING AND MANAGEMENT OF SMALL PROJECTS The PBF programme stimulates entrepreneurial capacities of health workers, who can actively improve their working environment. Of the PHUs in-charges 87% indicated having sufficient autonomy for small project management, but only 57% felt the staff had enough capacities to make a plan. Of all PHUs 62% made action plans, at least a plan how to utilise the money paid from the PBF programme. Sometimes PHUs just produced procurement lists. Often those plans were made shortly before money was withdrawn from the bank accounts. The action planning should be a collaborative effort between in-charge, staff and Health Management Committee according to the operational manual. However, this was not always the case, as shown in table 23 below. Community participation is discussed in the next paragraph. Table 23 Involvement in action panning of PHUs Involved in the action planning according to PHU No health facilities Percentage DHMT 35 15% Health Management Committee % Facility staff % Council 11 4% In-charges manage the small projects which are paid from the PBF programme sometimes with assistance from the DHMT or from the HMC. As per instructions in the PBF operational manual PHUs has to spend 60% on incentives for all staff members (formula based on positions) and 40% on investments. Health facilities invested in: Making the work environment more conducive for services (furniture, painting and repairs of building, solar lights, repair of motorcycles, etc.) Equipment and supplies (BP machines, weighing scales, registers, patient cards, stationery, paper, kerosene for stoves, etc.) Sanitation and hygiene in the PHU or hospital (improvement of water supply, cleaning materials, utensils, waste management, etc.) Medical supplies (small quantities of medicines)

63 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 61 Figure 30 Word cloud investments in PHUs Most investments were small, not exceeding amounts of LE 500,000 (US$ 1,000). Figure 30 above is a word cloud of recorded investments made with PBF funds. The Council reports in annex 4 volume II provide details on the top 10 of investments per districts. Health facilities considered the PBF funding a welcome addition to government funding. However, they still struggle with problems of shortage of equipment and supplies, which was listed as the number 1 constraint in interviews FINANCIAL MANAGEMENT IN PRACTICE The PBF operational manual does not provide instructions on financial management. Most PHUs keep very simple records of income and expenditure, often not even meeting the minimum standards of a cashbook. This is not surprising as the in-charges were never instructed how to perform these duties. During the EV only 62% of PHUs could show records in a cashbook or ledger book of any PBF amount received. They could not produce payment slips from the bank, or other explanation of transfers made to their bank accounts. One reason for missing records given was that the former incharge had taken the cashbook on transfer. In general there was no handover of finances on replacement of the in-charge. The tables 24 and 25 below show details of amounts, which should have been paid and amounts identified by the enumerators. Kono, Moyamba and Tonkolili show very low percentages of amounts identified. Table 24 PBF payments to PHUs requested by MOHS to MOFED District Payment request Q Payment request Q Payment request Q Bo Bombali Bonthe Kailahun Kambia Kenema Koinadugu

64 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 62 District Payment request Q Payment request Q Payment request Q Kono Moyamba Port Loko Pujehun Tonkolili Western Area Total Table 25 Percentage of quarterly payments, traced in PHUs during the EV 2 nd quarter rd quarter th quarter 2012 Bo 54% 73% 30% Bombali 79% 54% 53% Bonthe 45% 117% 14% Kailahun 63% 59% 38% Kambia 51% 70% 79% Kenema 72% 74% 56% Koinadugu 79% 43% 6% Kono 10% 0% 2% Moyamba 2% 4% 59% Port Loko 73% 24% 1% Pujehun 36% 35% 25% Tonkolili 2% 1% 67% Western Area 71% 43% 5% Total 41% 35% 27% Investments of LE 621 million were traced in PHUs during External Verification, which amounts to roughly 75% of the estimated LE 830 million, the available amount for investments in Most health facilities had not yet received full payment for 2012 when the EV took place. Therefore, most PHUs could not have reached 100%. It may be concluded that in PHUs recording of expenditure was more precise than recording of income. Most in-charges kept receipts for expenses and all DHMTs checked expenditure during the IV (and sometime took the receipts to their office). However only four DHMTs collected financial reports from the PHUs and only two forwarded financial reports from the PHUs to the Councils DELAYS IN PAYMENT The timelines as described in the PBF operational manual were not kept, due to delays in all stages of the process: delayed internal verification reports by DHMTs, delayed processing of payment requests by MOHS and delayed payments by MOFED. Payments for the fourth quarter of 2012 were processed by the MOFED-LGFD in the first quarter of MOHS and MOFED now process all requests for payments as they come in without first accumulating all claims. This will considerably shorten waiting times for most facilities. However, the operational manual was too optimistic in its planning. Financial procedures take their time and a six-moths procedure is more realistic than a 3-months procedure. In PHUs there seems to be no insight in the relation between performance and payment. Some incharges interviewed claimed that they did receive payments for 2013, but not for They could not see a relation between outputs, quality scores and amounts paid. In the third quarter 2012 corrections were made over payments in the previous quarters: some facilities received no or even negative payments in the third quarter of In-charges were not 21 MOHS could not provide exact amounts of disbursements for first quarter The EV team assumed it was more or less equal to other quarters.

65 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 63 informed, and some claimed they were denied payment over the third quarter. Due to insufficient feedback from the IV, PHUs do not know the amount of PBF payment, which they could expect over quarters under review. The payment into the bank accounts came with little information and therefore PHUs often did not understand for which quarter the payment was. The complete lack of explanation and transparency with regard to payments to PHUs was experienced as very frustrating by PHU incharges interviewed. Late payment affects continuity of the PBF programme and had high opportunity costs: PHUs were eager to implement health services improvements, but had to wait for over one year to get their due payments. Health workers expressed fear that they would not receive their bonuses after such a long period of delay, e.g. after transfer. Not receiving a performance bonus created frustration, rather than motivation for better performance. 6.4 COMMUNITY INVOLVEMENT The PBF programme aims at increased community participation and co-ownership. According to the PBF manual Health Management Committees 22 (HMCs) should contribute to planning, management and monitoring of health services. The HMC chair was supposed to be the co-signatory of the accounts. The Health Sector Strategic Plan explicitly mentioned community involvement as priority in the health sector. The programme succeeded in some areas in enhancing this community participation. In 58% of the PHUs the HMC was involved in planning and in 35% of the PHUs the HMC chair was co-signatory of the PHU account. However, in general HMCs do not see co-management of the health facility as their priority. Often HMCs have a rather traditional perspective regarding their roles. Participating in planning and priority setting is mentioned by only 49 PHUs (20%), as shown in table 26 below. Table 26 Roles of HMC according to chairs HMC Roles of the HMC No of HMCs Percentage Sensitisation of community and health education % Mobilisation of funds or voluntary labour % Provide feedback to the community % Prioritisation of activities in the health facility 49 20% Other* 27 11% *(e.g. monitoring of arrival of drugs and settling disputes) In general, the element of strengthening community involvement did not get much priority in the implementation of the PBF programme so far. According to the programme design in line with the PBF theory the Councils represent the interests of the population. They aim for value for money on behalf of the citizens. Therefore one of the roles of the Council was to strengthen community participation and to stimulate involvement of the Health Management Committees in the management of PHUs. 82% of the Councils indicated that the HMCs were not sufficiently involved in the PBF programme. 6.5 SEPARATION OF FUNCTIONS IN THE PBF PROGRAMME The PBF principle of separation is built on the theory of split of functions between provider, purchaser and regulator, with good reasons. The regulator formulates national policies and sets the standards for quality of care and defined the rules of the game. The purchaser procures at the best value for money, taking into account the policies as defined by the regulator. The provider sells its products, which meet standards as defined by the regulator, for a price negotiated with the purchaser. In the Sierra Leone PBF this approach is followed as well, as shown in figure A variety of titles are used for this committee, e.g. village health committee, health facility monitoring committee.

66 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 64 Figure 31 Roles in PBF in Sierra Leone MOH-Regulator focuses on quality National policies for healthcare Technical guidance and supervision DHMT Council-Purchaser focuses on costs PHU-Provider focuses on accessibility Value for money As described in Chapter 2 Sierra Leone has embarked on a decentralisation-by-devolution, whereby the MOHS is the steward in the health sector, having only direct control over tertiary hospital. The Councils are the owners of the health facilities and responsible for the health service provision. Administratively the DMOs and the DHMTs are under the Council. Funds for managing the health services are channelled via the Councils. In Sierra Leone a light PBF approach is applied, which means that not all theoretical concepts of PBF are fully implemented. The Council is officially responsible for the health services, but is at the same time the contracting agency. The DHMT is the technical supervisor and at the same time the internal verifier COUNCILS In the present situation most Councils do not play their role of purchaser as envisaged in the programme design. From the interviews it transpired that half of the Councils did not understand the contents of the contract and 44% did not understand the criteria applied to PHUs for eligibility of contracting. Only 28% of the Councils kept the contracts with the PHUs in their files. Of all Councils 30% indicated not to have capacities for the tasks attached to the PBF programme. One third of all Councils was never involved in the quarterly internal verification, one third was rarely involved, and only one third was regularly involved in IV. If the Council was involved it was mostly the M&E officer (66%) and occasionally elected Council members (16%) DHMTS DHMTs have a triple role in the health sector and in the PBF programme: Supervision and technical guidance of PHUs on behalf of the MOHS, which is the regulator in PBF; Management of the health services on behalf of the owner of the health services, which is the Council according to the devolution policy; In the PBF programme internal verification on behalf of the purchaser, which again is the Council. In figure 31 above the DHMT is positioned in the middle, and sometimes it is caught in the middle, serving two masters. Both the regulator and purchaser have to provide clear instructions to the DHMT

67 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 65 ensuring that all interests are covered adequately. The EV came across several challenges concerning the DHMT s role. Internal verification is taking place systematically. 94% of the PHUs indicated that DHMTs performed regular supervision and this was confirmed by the DHMTs. The IV puts a huge strain on the DHMTs, as it is time consuming. With 50 to 100 PHUs in one district the process may take weeks every quarter, claiming time from DHMT members and claiming means of transport from the office. More than half of the DHMTs perform the IV within 4 hours; one-fourth spends between 4 and 6 hours and few more than 6 hours. Based on the experience in the EV it is impossible to do a thorough verification including review of registers and assessment of the crosscutting issues within 4 hours, certainly if it is combined with supervision and if it includes travel time. The results of the internal verification have been discussed extensively in Chapter 4. In general, in the districts the results of internal verification of output indicators and crosscutting issues were higher (in some cases significantly higher) that the external verification. Financial management. DHMTs perform some supervision regarding financial management in PHUs; One-third of the DHMTs receive a financial report from PHUs, but more often copies of receipts of purchases. Only two of the 13 DHMTs forwarded them to the Council. DMOs in two districts are cosignatory of the PHU accounts, and eight DMOs have to check the investment plan before the PHU can withdraw money from the bank. None of the DHMTs introduced cashbooks in the PHUs. Capacity building. Officially, the purchaser has the task to ensure that provider understands the rules for payment for service delivery, i.e. how to register, how to report, how to manage funds, how to plan for activities, etc. The Councils could delegate the task of capacity building to the DHMT. DHMTs have indicated to perform on-the-job training, but concentrate more on technical issues than administrative issues (like financial management). During the EV only half of the in-charges of PHUs indicated to understand the PBF programme fully, and DHMTs indicated a huge need for capacity building MOHS The MOHS has more functions than strictly according to the PBF theory. Besides producing payment requests to MOFED, the PBF technical team is responsible for overseeing the verification process by signing contracts with the internal verification teams, for the M&E systems (e.g. through HMIS) and quarterly and annual reporting. The PBF programme is managed by a technical team consisting of representatives from different departments in MOHS (Directorate of Policy, Planning and Information, Directorate of Primary Health Care, Directorate of Reproductive and Child Health, Directorate of Financial Management, Human Resources Division). The technical team provides training and capacity building. The PBF technical team interacts with the Development Partners in the Steering Committee. Last year the PBF technical team was reshuffled. The new PBF technical team has not yet performed a national verification; the previous team did perform verification, but did not produce a report. The EV team could not check which issues they came across. The new PBF team did not perform data triangulation of HMIS, IV and F-reports, which did show major data differences in the EV. It processed payment requests to the MOFED based on IV reports only MOFED The MOFED - Local Government Finance Department is responsible for disbursement of the PBF fund after request by the MOHS and supervision of financial management by the Local Councils. The MOFED completed financial reports on PBF for the World Bank, but in those reports only accounted for money transferred to the PHUs, not how the money was spent in practice.

68 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE DEFINITION OF INDICATORS The domains and indicators and described in Chapter 4.1. Indicators are the backbone for payment to the PHUs and hospitals. Several indicators did not have SMART definitions in the operational manual or other documentation. In practice internal verification teams applied no uniform definitions. Therefore the consultants invited the stakeholders during the inception workshop to work jointly on the formulation of definitions for indicators and certain terminology. These agreed definitions have been applied in the external verification. Not having very clear indicators may have affected the internal verification, leading to variation in scores between DHMTs. 6.7 CONTRACTS PHUs were contracted for service delivery in tripartite contracts with DHMTs and Councils. As has been explained above those contracts did not play any further role in the PBF implementation. The procedures described, e.g. with regard to financial reporting were not followed. Never were any sanctions applied in case when parties were not adhering to their duties. Non-governmental healthcare providers would be eligible for contracting as well, according to the PBF manual. In practice, none was contracted. 6.8 HOSPITAL PBF CONTRACTS Princess Christian Maternity Hospital (PCMH) and Ola During Children s Hospital (ODCH) are the only hospitals benefiting from the PBF programme. In the hospital PBF system there is strong emphasis on learning, exchange of best practices, recommendations for improvement and assessment of the follow-up of these recommendations during the next verification. Following the PBF operational manual, in 2012 tripartite agreements were made between the Freetown Council, the MOHS and the hospitals, in which roles and responsibilities were described. The Council acted as the purchaser. However, the Council did not play a role in internal verification. Per January 2014 tertiary hospitals were transferred back to the MOHS, but contracts have not been adjusted IMPLEMENTATION ODCH has installed a Quality Assurance (QA) Committee; in PCMH the management team acts as QA committee. These committees perform self-assessment on a monthly basis, using the PBF checklist, as described in Chapter 4.1. The self-assessment is the starting point for setting priorities for quality improvement. The national PBF technical team, together with invited experts from other ministries, performed the internal verification. It was noted that every time a different team did the internal verification, which according to the hospitals resulted in inconsistent recommendations for improvement. The second and third quarter 2012 internal verification was combined INDICATORS The indicators in eight domains all contribute to a score for that domain. Not all indicators are SMART (e.g. are there functional toilets in the hospital ), and not all indicators are applicable (e.g. clean labour ward in ODCH). The scoring may therefore be subjective. Nevertheless, scoring was often high in the IV, as discussed in chapter 4. The IV forms often only have scores, with the remarks column left blank. It is therefore not clear where the scoring is based on, making it difficult for a next team to follow up. The indicators only concentrate on quality and do not take output into account, with as argument that tertiary hospitals are dependent on referrals and should not increase their output, taking patients away from lower level facilities. However, the introduction of free health care has put a strain on quality of

69 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 67 care in hospitals. According to respondents, patient numbers increased 4 to 10-fold as a result of abolishment of user fees, while financial compensation and inputs did not increase proportionally EXPENDITURE Similar to the PBF project in PHUs, hospitals spend a maximum of 60% on staff incentives and 40% on investments. The incentives are for all staffs and tied to their salary. For most staffs it means an extra month s salary every quarter. All workers are therefore very much engaged in PBF. Both hospitals have introduced a performance element in bonus payment, first only presence and now also commitment, to vary the height of performance bonus. The 40% investment money is a lifeline for the hospitals, constituting 40% - 60% of their funds for running costs. As Government funds are tied to strict procedures through the Freetown City Council, and often delayed due to cumbersome bureaucracy, this free money offers an opportunity to solve problems quickly, e.g. shortage of medicines, laboratory supplies, gloves, maintaining water supply, etc. The funds are also used to procure equipment, stationery and bed linen. Hospitals make a quarterly investment plan together with the heads of department. Like PHUs, hospitals are affected by delayed verification and slow procedures of disbursement, which affects continuity of the work. Recently, Government has informed that their regular budget would be lowered, because the hospitals get PBF funding NON-PBF HOSPITALS The two non-pbf hospitals verified, operate at very different levels. Rokupa Hospital is a very busy well-equipped district hospital with many workers and patients. It has a busy theatre and labour ward. Macaulay Medical Centre is a dilapidated health centre, with a labour ward and theatre under construction. Their scores in the external verification cannot be compared. The lesson learned form this comparison is that introduction of PBF requires a good programme of training and introduction of systems and procedures. Verification forms should be more custom-made as every level of hospital has other circumstances to be assessed. SUMMARY OF SYSTEMS ANALYSIS THE PBF PROGRAMME WORKS COMPLEMENTARY TO FREE HEALTHCARE, AND OFFERS TO HEALTH FACILITIES A COMPENSATION FOR THE LOSS OF INCOME THROUGH PATIENT FEES. THE PROGRAMME HAS BEEN SUCCESSFUL IN THIS. THE PROGRAMME HAS SUCCEEDED IN PROVIDING MORE AUTONOMY TO HEALTH FACILITIES TO MANAGE THEIR OWN SMALL PROJECTS, WHICH CONTRIBUTE TO BETTER WORK ENVIRONMENT: MORE HYGIENE, BETTER EQUIPPED BUILDINGS AND BETTER SUPPLIES HAVE BEEN ACHIEVED. FINANCIAL MANAGEMENT IS A WEAK AREA, WITH VIRTUALLY NO SYSTEMS IN PLACE AT GRASS ROOT LEVEL. LATE PAYMENT DURING THE PERIOD OF REVIEW AFFECTED CONTINUITY OF THE PBF PROGRAMME AND HAD HIGH OPPORTUNITY COSTS: PHUS WERE EAGER TO IMPLEMENT HEALTH SERVICES IMPROVEMENTS, BUT HAD TO WAIT FOR OVER ONE YEAR TO GET THEIR DUE PAYMENTS. HEALTH WORKERS EXPRESSED FEAR THAT THEY WOULD NOT RECEIVE THEIR BONUSES AFTER SUCH A LONG PERIOD OF DELAY, E.G. AFTER TRANSFER. NOT RECEIVING A PERFORMANCE BONUS CREATED FRUSTRATION, RATHER THAN MOTIVATION FOR BETTER PERFORMANCE. THE PROGRAMME HAS SUCCEEDED TO SOME EXTENT IN IMPROVING COMMUNITY CONTRIBUTION TO MANAGEMENT OF HEALTH FACILITIES, ALTHOUGH THE CAPACITIES ARE STILL LIMITED. IN SIERRA LEONE A LIGHT PBF APPROACH IS APPLIED, WHICH MEANS THAT

70 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 68 NOT ALL THEORETICAL CONCEPTS OF PBF WITH REGARD TO SEPARATION OF RESPONSIBILITIES (E.G. HEALTH RESULTS INNOVATION TRUST FUND 23 ) ARE FULLY IMPLEMENTED. THE LOCAL COUNCIL IS OFFICIALLY RESPONSIBLE FOR THE HEALTH SERVICES, BUT IS AT THE SAME TIME THE CONTRACTING AGENCY. THE DHMT IS THE TECHNICAL SUPERVISOR AND AT THE SAME TIME THE INTERNAL VERIFIER. IN PRACTICE THE COLLABORATION BETWEEN COUNCILS AND DHMTS OFTEN IS NOT AS ENVISAGED IN SIERRA LEONE S PBF PLAN. THE DHMTS OFTEN OPERATE INDEPENDENTLY, AND COUNCILS DO NOT FEEL ENGAGED IN THE PROGRAMME. THE HOSPITAL PBF HAS STIMULATED THE TWO INVOLVED HOSPITALS TO IMPROVE PERFORMANCE IN MANY AREAS. HOSPITALS ARE BECOMING DEPENDENT ON THESE FUNDS AS PART OF THEIR CORE FINANCING. 23

71 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 69 7 DISCUSSION 7.1 QUALITY The general objective of the PBF system is to change the behaviour of health providers at facility level for them to deliver more quality services under the free health care policy. The External Verification assessed elements of improvement of quality of care. The patient satisfaction survey was an instrument to assess the perceived quality of care and the crosscutting issues quality assessment was an instrument to objectively score quality. The EV found a statistically significant correlation between high total scores for crosscutting quality issues in EV (not in IV) and high scores for patient satisfaction in facilities. Within the set of crosscutting topics cleanliness and availability of medicines were most prominent in the correlation with patient satisfaction. Good supervision by the DHMT and good feedback from internal verification were also in a statistically significant way linked to higher scores on crosscutting issues. The regression analysis showed that the involvement of the health management team in the action plans was significantly associated with client satisfaction. The positive coefficient indicates that higher satisfaction scores are observed for facilities where this is practiced. The EV also found a positive correlation between investments, patient satisfaction and higher scores on crosscutting issues. A coherent system of enabling factors for quality improvement is in place (figure 32). Figure 32 Relations in the quality system The quality system is dependent on more factors, like human resources, drug supply, etc. Quality in the health system in Sierra Leone is a collaborative effort, and not exclusively linked to PBF. However, the PBF programme was able to create leverage by well-directed triggers for quality improvement. This can be enhanced when the link between performance and payment is strengthened, i.e. when the system becomes more transparent, better understandable, and when payments of bonuses follows shortly after provision of quality services. 7.2 DATA QUALITY Data quality issues dominated this EV. Here four topics are discussed: missing data, data consistency, case definitions and triangulation.

72 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE MISSING DATA Of the F-forms 3%-5% were missing, of HMIS data 15%-17%, of IV reports 9% - 11%. The EV encountered 4%-24% missing registers, as explained in chapter 4. Missing hard copies of reports and soft copies of computer files indicates poor storage practices at all levels. In 2012 the HMIS system crashed and no sufficient back-ups were available to restore data. Missing IV reports at national level were caused by reshuffling of the PBF team and lost data on computers, but at district level by poor practices of storing and backing-up computerised information. Registers are frequently out of stock in PHUs (table 27). Especially the registers for maternal health often are improvised registers in ledger books, notebooks, or loose papers with many shortcomings, e.g. missing columns or data sets. In the EV many of these improved registers from 2012 were lost. Table 27 Registers reported out of stock by PHUs Register Books out of stock No of PHUs OPD under 5 47 Immunization 33 Maternal Health Registers 118 Family Planning 21 None 79 The external verifiers occasionally came across situations where new maternal health registers were present but not used, as the staff could not understand them. In few cases, even under-five cards were missing. The erratic supply of stationery definitely affects the quality of the PBF programme DATA CONSISTENCY F-forms. As explained in Chapter 4, between 90% and 95% of the registers are filled in satisfactory, which means that information in the register is in general reliable, if the proper stationery is used. Quite a number of staff do not understand the F-report forms fully and fill in wrong information, or copy from wrong registers. At times, health workers have problems with mathematic skills. One example of errors in reporting is the box for filling in daily attendance on form PHU-F1. See figure 33 below. It is not clear what is meant total head count (all services) and total OPD cases. Does head count include all OPD services or all preventive and curative services in the facility? Does total OPD cases mean number of patients or number of diagnoses? In the EV it was found that PHUs apply different definitions. Clarity and consistency in use of report forms is needed. Figure 33 PHU-F1 form box totals HMIS. The deviations between HMIS, entered in the automated DHIS-2 system, and the F-forms may be considerable. The PHUs with more than 25% data variation ranged from 14%-71% for output indicators. Some districts mentioned in interviews making corrections in HMIS when figures in F- reports were wrong, without correcting the F- forms. There may also be copy errors, when data entry clerks exchange forms from facilities. Between 36%-69% of PHUs had a 25% data deviation between HMIS and EV, which is very high. In the validation workshop it was concluded that capacities of M&E units in the DHMTs are too low, and that quality control by DMOs was minimal. Poor quality of HMIS not only affects the PBF system, but also the health sector as a whole.

73 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 71 Internal verification. The data in the internal verification deviated much from the EV. Between 33%- 100% of PHU had more than 25% data deviation. Discussions during the validation workshop pointed at time constraints to complete the job in time or lack of capacities in the DHMT. The high scores in internal verification cannot be contributed to deliberate over-scoring but rather to interpretation errors of case definitions CASE DEFINITIONS In previous chapters issues of clear case definitions were mentioned. The EV team noticed that different interpretations of case definitions might have affected the IV, even for assessing the output indicators: Family planning: condom distribution was to be excluded from the count (more for prevention of STD than for FP), but some DHMTs included it; ANC: the fourth attendance was to be counted, but some included the fifth and later visit as well. PNC: the third attendance was to be counted, but some included all post-natal controls after 6 weeks, even if it was the first after delivery. Fully vaccinated: only children below the age of one were to be counted, but some included all, even above the age of one. Apparently, there was not sufficient guidance from the MOHS to come to uniform approach in IV TRIANGULATION F-forms, IV and EV all base their information on the registers; HMIS is an electronic copy of the F- forms. The deviations between F-forms, IV and HMIS could have been noted in 2012 if triangulation had been done. Before sending the IV reports, DHMTs could have compared data with F-forms and HMIS, and correct or explain deviations. Similarly, at national level comparison could have been done between HMIS and IV district data. Improving data quality should be one of the top priorities in advancing the PBF programme. This requires agreement on definitions and procedures as well as capacity building at different levels to ensure that at the grass root level in PHUs and DHMTs the right approach is followed. Incentives for quality of information could be considered. Quality control, e.g. through triangulation, should be considered. At the same time the burden of work created by internal verification should be reduced. 7.3 PBF LIGHT Sierra Leone has opted for a PBF light approach, which means that not all theoretical concepts of separation of functions and systematic changes were taken on board. In the PBF operational manual systems and procedures have been described. However, the practice shows that PBF has become more a programme of MOHS, DHMTs and health facilities, than an intersectoral programme, with involvement of the Councils, Ministry of Local Government and Rural Development, and MOFED. Councils were in two third of the cases not or not sufficiently involved and did not carry out (or were not enabled to carry out) their part of the PBF programme. From the PBF perspective the value-for-money or purchasing perspective was not given sufficient attention. The relation between payment and performance has not been sufficiently clear, especially not at the PHU level. One of the reasons given during the validation workshop was that the Councils did not have an official role in accounting for the money, as funding was bypassing Government structures at Council level. There was no real incentive or obligation to take responsibilities, while so many other programmes cried for more attention in an overburdened administration. The design of the PBF provides a bigger role for communities than actually implemented in many PHUs. Capacities of HMCs to co-manage may be limited, or willingness on the side of the health workers to share responsibilities may be limited. But for sure, it is an area, which requires further attention. Patient tracing and satisfaction surveys are mentioned in the design, but not elaborated. It can be a powerful instrument, if linked to incentives for health facilities.

74 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 72 The PHU design as described in the operational manual is sufficiently robust to guarantee a performance based financing approach, and reconfirmation of that commitment and restart of that system would bring more balance in the PBF programme. It requires new engagement of stakeholders, maybe making adjustments in the procedures where needed. 7.4 FINANCIAL MANAGEMENT Financial management is discussed in paragraph The omission in the PBF manual to formulate requirements and to provide recording and reporting instructions, did never get a follow up. It more or less fell through the cracks in the system. Given the very limited financial management capacities of grass root health workers, it is surprising that most expenditure could be explained and even justified with documents and receipts in this EV (although income could not be explained, partly because banks do not provide payment slips). The fact that Councils did not feel obliged to control financial management, because money was bypassing their system, is remarkable. Apparently in the contracting process no clear agreement was reached on roles and responsibilities of partners. The most important observation during this External Validation was that due to long delays in payments, the lack of transparency in how amounts were reached, health workers at grass root level completely missed the emotion of payment for performance. Money was not seen anymore as a reward for an effort during a given quarter, but as a (long-overdue) welcome addition to other sources of income or supplies. MOHS and MOFED have started to catch up with clearing the backlog of payments over 2013, and have changed to processing of payment requests. Hopefully it will go hand in hand with an effort to reinstate the payment for performance feeling in the system. The issue of payment for travel allowances and payment for incentives for DHMTs and Councils is another issue to be clarified for involved stakeholders. Present confusion does not contribute to motivation of DHMT members or Council officials to perform internal verification.

75 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 73 8 CONCLUSIONS AND RECOMMENDATIONS 8.1 CONCLUSIONS THE GENERAL AND SPECIFIC OBJECTIVES OF THE PBF PROGRAMME The general objective of the PBF system is: to change the behaviour of health providers at facility level for them to deliver more quality services under the free health care policy. The EV team concludes that systems have been put in place and are operational to a reasonable extent in a number of health facilities (see Chapter 7.1). Further strengthening of the system is possible within the present design of PBF in Sierra Leone, when a number of implementation issues can be solved. The specific objectives of the system are: 1. Provide cash at facility level to cover the local costs of delivering services and removing the need for 'informal' fees. The EV team concludes that this has been largely achieved, with only 12% of the patients paying for those services, which supposedly are free. (See Chapter 5.) Late transfers of PBF funds may have forced PHUs to ask for contributions for patient records, etc. when funds dried up. Payments by patients may reduce further if PBF payment timeliness and accuracy improves. Incidental misbehaviour by health workers cannot be ruled out. 2. Provide financial incentives to facilities in order to increase productivity and quality of care, especially for the identified key indicators. The EV team concludes that this has been partially achieved. There is an increase in service utilisation, although that increase is levelling off. There are signs of improved attention for quality. However, the relation between performance and payments is too weak for health workers. The incentive system is not transparent enough and payments so delayed, that they are no longer seen as reward for good performance. 3. Increase the equity of distribution of resources with funds from PBF allowing facilities to hire contractual workers and finance outreach activities. Equity of distribution of funds may have taken place using district-based payment formula, but was not visible for grass root workers. The flow of funds in general was not regular enough to hire contract workers (with exception of the two PBF hospitals). Outreach may have benefited from PBF funds, e.g. by repair of motorcycles and purchase of fuel. In general, funds were used for repairs of the building, furniture, equipment and supplies, water and sanitation, etc. These investments have contributed to patient satisfaction and higher scores for crosscutting quality indicators THE TERMS OF REFERENCE In response to the TOR for the external verification the EV team presents the following conclusions: 1. Review the accuracy of the facility data from the registers and other records The registers at facility level were fairly accurate, with on average 90%-95% satisfactory entries. Unfortunately, too many registers went missing over the period between 2012 and However, the accuracy of F-form reports, HMIS reports and Internal Verification in too many cases is below standard. Not only is the variation between the sources of information and the External Verification too high, also between the sources of information amongst themselves there are too many inconsistencies. This could have been avoided by triangulation of information from F-forms, HMIS and Internal Verification. 2. To analyse the data of the first full year of PBF implementation (2012) The methodology of analysis can be found in Chapter 3. Extensive analysis is reported in Chapter 4 of this document. Data quality and analysis of reasons for differences in numbers required much

76 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 74 attention. Data quality issues were caused by missing data, inaccurate data collection and transfer of data, as well as variation in interpretation of case definitions. 3. To review the roles and responsibilities of the different PBF stakeholders and advise on the areas of improvement if necessary Chapter 6 analyses the PBF system and chapter 7 discusses some system successes and failures. In general roles and responsibilities are sufficiently clear in the design of the PBF programme (operational manual), but require new commitments and reconfirmation, especially to increase the role of the Councils and communities in the PBF system. Councils should be incentivised to actually pick up their purchasing role. 4. To evaluate the benefits of the performance based financing in term of services delivery, strengthening the health system information (verification of data and timely reporting), the governance of health facilities (management of human resources, environmental health, financing, etc.). Benefits of PBF are clearly visible, but more at micro-level than at meso- or macro-level. In specific health facilities (and especially the two PBF hospitals) the impact can be seen and effects of system strengthening can be proven: increased staff motivation, increased hygiene, better supplies, etc. However, this is not the case across the board. The link between payment and performance is not yet strong enough to motivate all health workers to go the extra mile to achieve better. However, the EV has documented good practices and the potential effect has been shown in the external validation. There is no proof of improved performance of DHMTs and information systems. However, at the district level the burden of work caused by the PBF programme might have been too much to cope with. There is not enough clarity on payment for performance for IV and payment of allowances to DHMT members and Council officials. At the national level, discontinuity in the PBF team has caused major disruptions in further progress of the programme, through loss of information, interruption of supervisory visits, loss project implementation memory 8.2 RECOMMENDATIONS VALIDATION WORKSHOP The validation workshop on 20 March 2014 offered an opportunity for the EV team to discuss preliminary findings with stakeholders from MOHS, MOFED, DHMTs, Councils, Civil Society, Development Partners, etc. In the plenary and group discussions recommendations were formulated, which found their way into this report. One proposal by the Director of Reproductive Health Services in the MOHS should be mentioned here, i.e. visiting all districts and discuss on the spot the way forward with stakeholders there, i.e. DMOs, DHMTs, Councils and PHUs. The EV team supports this initiative. The specific district reports annexed to this main report may guide this discussion, as well as the urgent recommendations SHORT-TERM RECOMMENDATIONS Reconfirm roles and responsibilities The MOHS district visits will offer an opportunity to confirm with the Councils the roles and the responsibilities as laid-down in the PBF operational manual. The roles of the Councils in contracting, in internal verification, and in financial management and reporting may have to be renegotiated per Council, as circumstances and conditions may vary. The roles of HMCs in the management of health facilities have to be clarified and their roles as described in the operational manual have to be confirmed. New Memoranda of Understanding can be signed to confirm commitments.

77 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 75 Clarify internal verification During the district visits the MOHS could provide an orientation workshop on quality of internal verification. The quality of internal verification has to improve: uniform case definitions have to be applied, and DHMT members, Council officials and elected Councillors involved should understand their tasks of verification of output indicators and crosscutting issues well. On-the-job training in verification of cross-cutting issues may improve the consistency and quality of this important part of the verification process. On the spot double check of IV report, F-forms and HMIS form (brought from the DHMT s M&E office) should be introduced to identify data inconsistencies and resolve them, or explain them in case mistakes are corrected. The by-product of the PBF programme should be more reliable HMIS. Simplify internal verification The validation workshop called for simplification of the Internal Verification, while improving the quality. The idea was to introduce sampling, not only months (one month per quarter), but also PHUs (e.g. 25% of PHUs). HMIS data would be guiding in payment for performance, rather than the data from IV. This is possible, but only if certain criteria are met. (See figure 34 below.) The first step in this process is to guarantee data quality of registers, F-forms and HMIS. Facilities should have the required registers and forms. HMIS and F-forms should be filled completely and should match. Districts, which cannot meet minimum criteria of HMIS quality, should first bring their house in order. The second step is to select PHUs, which meet criteria of data quality, with matching IV and HMIS. Those with reasonable data quality are admitted to the pool. But they can be removed from the pool if in a control they are found to be missing the quality standards. From there, step-by-step, more facilities are added to the pool introducing gradually a system of sampling months and facilities. Throughout the time, random sampling should be used, which even makes control in consecutive quarters possible. NB: quarterly supervision and assessment of crosscutting issues should continue in all health facilities! This has been shown to be a crucial element of quality improvement and cannot be done through sampling.

78 EXTERNAL VERIFICATION PERFORMANCE BASED FINANCING SIERRA LEONE 76 Figure 34 Step by step introduction of sampling IV Step 1 Select districts which Have all data cells filled in HMIS since January 2014 (no missing data) Have HMIS PHU report which are similar to F-forms District does not meet criteria Continue full Internal Verification Capacity building and control HMIS until meeting criteria District meets criteria Step 2 Select PHUs which In Internal Verification have for all six output indicators less than 10% deviation from HMIS (CHC and CHP) In Internal Verification have for all six output indicators less than 25% deviation from HMIS (MCHP) PHU does not meet criteria Continue full Internal Verification Capacity building and control HMIS When criteria are met during next Internal Verification, move to sample pool Capacity building and control HMIS PHU meets criteria Step 3 Perform random sampling at MOHS of PHUs in sample pool Implement Internal Verification of sampled PHUs PHU has more than allowed deviation for one or more indicators in Internal Verification Remove from sampling pool Perform Internal Verification in next quarter; build capacity Step 4 Continue process until all PHUs meet criteria When criteria are met during next Internal Verification, move to sample pool Capacity building and control HMIS