September 2014 DISCLAIMER:

Transcription

1 i

2 September 2014 Recommended Citation: Government of Kenya Data Quality Audit Report, August 2014, Nairobi, Kenya: Division of Health Informatics Monitoring and Evaluation, Ministry of Health, AfyaInfo Project. This Data Quality Audit Report was derived from the findings of a country-wide data quality audit exercise conducted for the health sector in March The report and the audit were done with assistance from the AfyaInfo project. AfyaInfo is a technical assistance program to support the Government of Kenya to strengthen their health information systems. The program is implemented by Abt Associates, Inc. in partnership with Training Resources Group, ICF International, the University of Oslo, Knowing Inc., the Kenya Medical Training College, and the University of Nairobi. It is funded by the United States Agency for International Development (USAID), under the AIDS Support and Technical Assistance Resources (AIDSTAR) Sector II IQC, contract number GHH-I AID-623-TO , Kenya Health Information System. DISCLAIMER: The author s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government ii

3 Table of Contents Foreword...vii Acknowledgements...viii List of Tables...v List of Figures...v List of Acronyms... vi Executive Summary...vii 1. INTRODUCTION Background Health Information System Data Quality Audit Data Quality Assurance Protocol Purpose of the Data Quality Audit Objectives METHODOLOGY Assessment Design Sampling and Site Selection Indicator Selection Assessment Tools Data Collection Process Data Analysis Ethical Considerations FINDINGs Background Availability of Services corresponding to Assessment Indicators DHIS Reporting Rates for Indicators Assessed Data Verification Availability and Completeness of Source Documents Availability and Completeness of Summary Tools...24 iii

4 3.2.3 Missing Data/Value for Indicators under assessment for both Source and Summary Tools Accuracy of Summary Sheet against the Source Documents Accuracy of DHIS Data against the Source Documents Accuracy of DHIS Data against the Summary Sheets Verification of KePMS data against Source Documents Comparison of DHIS data To KePMS data Sub County Level Findings Sub County Reports Availability, Completeness and Timeliness Systems Assessment- Sub County Level Systems Assessment- Facility Level Overall Scores Monitoring and Evaluation Structure, Functions and Capabilities Indicators Definitions and Reporting Guidelines Data Collection and Reporting Forms and Tools Data Management Processes Computerized Information Systems / Software applications Link with National Reporting Systems Challenges Experienced in the DQA CONCLUSIONS AND RECOMMENDATIONS Conclusions Recommendations REFERENCES: ANNEXES...53 ANNEX A: RDQA TOOL ANNEX B: SUMMARY OF FINDINGS BY INDICATORS iv

5 List of Tables TABLE 1: DISTRIBUTION OF DQA FACILITIES BY OWNERSHIP/LEVEL TABLE 2: LIST OF INDICATORS, REGISTERS AND SUMMARY TOOLS TABLE 3: PROPORTION OF FACILITIES IN THE SAMPLE WHERE THE SERVICES CORRESPONDING TO INDICATORS WERE NOT AVAILABLE/DELIVERED TABLE 4: DHIS2 REPORTING RATES FOR SUMMARY TOOLS INCLUDED IN AUDIT TABLE 5: AVAILABILITY AND COMPLETENESS OF SOURCE DOCUMENTS TABLE 6: PROPORTION OF MISSING DATA/VALUE FOR INDICATOR (SOURCE DOCUMENTS AND/OR SUMMARY SHEET) IN FACILITIES WHERE SERVICE IS PROVIDED TABLE 7: PROPORTION OF FACILITIES WITH SOURCE DOCUMENTS DATA MATCHING SUMMARY SHEET DATA TABLE 8: PROPORTION OF FACILITIES WITH SOURCE DOCUMENTS DATA MATCHING DHIS2 DATA TABLE 9: PROPORTION OF FACILITIES FOUND SUMMARY SHEETS DATA MATCHING DHIS DATA TABLE 10: VERIFICATION OF KEPMS DATA AGAINST SOURCE DOCUMENTS TABLE 11: COMPARISON OF DHIS DATA AGAINST KEPMS TABLE 12: FACILITIES SUMMARIES OF SCORES ON DATA MANAGEMENT AND REPORTING SYSTEMS ASSESSMENT List of Figures FIGURE 1: DATA VERIFICATION (REVIEW AND COUNTING) IN PROGRESS FIGURE 2: NUMBER OF SUB-COUNTIES BY PROPORTION OF REPORTS AVAILABLE, COMPLETENESS AND TIMELINESS FIGURE 3: SUB-COUNTY SUMMARY STATISTICS ON SYSTEMS ASSESSMENT AVERAGE SCORES BY FUNCTIONAL AREAS FIGURE 4: SAMPLE CHECKLIST TOOL USED TO TRACK REPORTING PERFORMANCE PER HEALTH FACILITY FOR EACH DATASET EXPECTED PER MONTH (T=TIMELY; L=LATE) FIGURE 5: STRUCTURE, FUNCTIONS AND CAPABILITIES OF THE SYSTEM TO MANAGE DATA FIGURE 6: SNAPSHOT OF ACTUAL RECORDS SHOWING ALL CHILDREN WEIGHED RECORDED AS 'Y' IN THE UNDERWEIGHT COLUMN (CONFIRMED AS ERRONEOUS) FIGURE 7: SNAPSHOT OF REGISTER RECORDS ILLUSTRATING POOR HANDWRITING, LIMITED SPACE TO RECORD IMPORTANT DATA SUCH AS DIAGNOSIS OR TREATMENT, AND POOR RECORDING EVEN WHERE SPACE IS ADEQUATE FIGURE 8: ERRONEOUS ADDITION ON THE ART ACTIVITY TALLY SHEET FIGURE 9: EPI REGISTER ILLUSTRATING COLUMN FOR FIC LEFT BLANK EVEN WHERE RECORDS INDICATE CHILDREN WHO HAVE COMPLETED THEIR SCHEDULE AS REQUIRED FIGURE 10: EPI REGISTER WITH SEVERAL PAGES LOST TO POOR STORAGE OR HANDLING FIGURE 11: INDICATORS DEFINITIONS AND REPORTING GUIDELINES FIGURE 12: SURPLUS REGISTERS IN STOCK AT COUNTY HRIO OFFICE FIGURE 13: DATA COLLECTION AND REPORTING FORMS/TOOLS FIGURE 14: MATERNITY RECORDS SHOWING 'D' UNDER 'CONDITION AFTER DELIVERY (A/D)' WHICH WAS CONFIRMED AS REFERRING TO THE BABY (COLUMN RECORDS SHOULD REFER TO THE MOTHER) FIGURE 15: DATA MANAGEMENT PROCESS FIGURE 16: PERCENT OF FACILITIES WITH PARALLEL REPORTING CHANNELS v

6 List of Acronyms ANC Ante Natal Clinic ARVs Anti-Retrovirals CBO Community Based Organizations CWC Child Welfare Clinic DHIS District Health Information Software Div-HIM/E Health Informatics Monitoring and Evaluation Division Div-HIS Division of Health Information Systems DQA Data Quality Audit DQI Data Quality Improvement EMR Electronic Medical Records FBO Faith Based Organizations GoK Government of Kenya HIS HIS Health Information System HIV Human Immunodeficiency Virus HIV/AIDS Human Immunodeficiency Virus/ Acquired Immunodeficiency Syndrome HRIO Health Records and Information Officer KePMS Kenya HIV/AIDS Program Monitoring System KHSSP Kenya Health Sector Strategic Plan MCH Maternal and child health MFL Master Facility List MoH Ministry of Health NASCOP National AIDS/STD control programme OPD Outpatient Department Pre-ART Pre- Anti-Retroviral Therapy RDQA Routine Data Quality Audit RH/FP Reproductive Health/Family Planning TB Tuberculosis vi

7 Foreword The management of health services is an important function that has been devolved to the established County Governments to ensure the delivery is closer to the people, and is of continually improving quality. Consequently, National level and County Governments are obligated by law to give periodic reports and evaluate performance of the health outcomes and recommend appropriate actions. This has seen an increased demand for accountability and the need to demonstrate results at Country and County Levels. However, performance can only be adequately monitored if the data used is above reproach i.e. it is reliable and of high quality because this impacts its utility and value to users. Data which are incomplete, inaccurate, out of date, etc. has a negative impact on the quality and utility of the information used to disseminate results or to measure performance over time. A countrywide Data Quality Audit (DQA) exercise was undertaken in 2010, the results of which went on to inform the implementation of a national web-based routine data collection system, the DHIS2 platform for reporting of routine health facility service delivery and community health services. Since its installation, there has been no other national data quality assessment or audit carried out in Kenya. The DQA carried out in 2014 is the first nation-wide exercise since the rolling out of the DHIS reporting platform in 2011, whose main purpose was to provide baseline information on the quality of routine data on service utilization being captured in the current system. Additionally, the findings were to provide insights into the development of the Data Quality Improvement strategies and methods for Ministries of Health at County Level as well as National Level. This report outlines the findings as well as recommendations for improving data quality. I am confident the content will provide appropriate guidance on data quality improvement strategies that can ensure data collected is acceptable for use in measuring performance, in evidenced based decision making and ultimately in improving the quality of health services at the grassroots levels. Dr. Nicholas Muraguri DIRECTOR OF MEDICAL SERVICES vii

8 Acknowledgements The Ministry of Health wishes to acknowledge the contribution of all the core team members who participated in the planning, preparation and implementation of the Nationwide Data Quality Audit (DQA) 2014 as a whole. We wish to extend our gratitude to the field technical teams that comprised of Health Information officers from Div-HIME as well as several programs & departments within MOH. Special thanks and appreciation goes to the Director of Medical Services Dr. Nicholas Muraguri as well as the Head of the Division of Health Informatics, Monitoring and Evaluation (Div-HIME) Dr. David Soti who both gave full support and guidance to the process. We especially appreciate the Head of the HIS Unit within Div-HIME, Dr. Martha Muthami who committed time as well as technical direction for the exercise which ensure successful implementation in the field. We also appreciate the technical and financial support provided by USAID-AfyaInfo in conceptualization of the exercise, field activities, data consolidation and analysis and report writing. We also would like to thank MEASURE-Evaluation for their technical input into the report viii

9 Executive Summary The country s health sector performance can only be adequately monitored if the data used is reliable and of high quality. With devolution, national level and counties are obligated by law to give periodic reports and evaluate performance of National and County government with regard to health outcomes and recommend appropriate actions as stipulated in the various governing Acts. The Data Quality Audit 2014 was the first nation-wide DQA since the rolling out of the DHIS reporting platform in 2011 for routine health facility service delivery and community health services. The last DQA was conducted in 2010 whose results informed the development of the DQA protocol. Since then, only vertical and sporadic DQAs have been conducted by programmes to meet localized needs. A descriptive cross sectional design was utilized to collect data from 178 facilities of which 33.7 % were GoK level 2 and 3, a further 30.3% were GoK level 4 & 5, and 6 while 15.7% were Faith Based Organisations and 20.2% were privately owned. The assessment utilised both qualitative and quantitative methods to verify the data from source documents for selected indicators against summary data, DHIS data, and Kenya HIV/AIDS Program Monitoring System (KePMS) data collected during the months of July- September Nine indicators were selected for this assessment taking into account the key programmatic areas in the health sector. From the findings, it was noted that the reporting rates for the summary sheets/reporting forms for the assessed indicators was fairly high with MOH 711 (Integrated RH, HIV/AIDS, Malaria, TB and Nutrition) and MoH 705 A & B (Outpatient Morbidity) having a reporting rate of about 90%, MOH 515 (Community Health Extension Worker Summary) and MOH 710 (Immunization summary) had the lowest reporting rates ranging from 34.6% to 64.8%. The availability of audit documents ranged from 91.1% for number of women of reproductive age receiving family planning to number of pregnant women referred for ANC at 39.1%. However, the calibre of available documents ranged from the standard registers to improvised counter books to older versions of the registers. The number of fully immunized children had the least complete audit documents at 64%. Notably the private facilities had the highest rate of missing audit documents with availability of documents being as low as 29.4% for some indicators. ix

10 The accuracy of summary sheet data against source documents was calculated as the proportion of facilities assessed found to have matching records; findings were number of HIV+ pregnant mothers receiving preventive ARVs (36.6%), total number of patients currently on prophylaxis Cotrimoxazole (32.6%), number of facility based maternal deaths (72.6%), number of women of reproductive age receiving family planning (6.2%,), number of pregnant women referred for ANC (32.1%), number of fully immunized children (22.9%), number of children under 5 treated for malaria (31.9%), number of new outpatient cases with high blood pressure (29.5%) and number of children under 5 who are under weight(31.4%) with an average accuracy of 30.8%. The MoH Level 4, 5 & 6 had least accurate data at 20.9%, followed by MoH level 2 & 3, at 26.8%, FBO at 31.8% and private facilities at 36%. The accuracy of DHIS data against source documents was; number of HIV+ pregnant mothers receiving preventive ARVs (38.8%), total number of patients currently on prophylaxis Cotrimoxazole (26.5%), number of facility based maternal deaths (75%), number of women of reproductive age receiving family planning (7.2%), number of pregnant women referred for ANC (24.2%), number of fully immunized children(16.1%), number of children under 5 treated for malaria (12.8%), number of new outpatient cases with high blood pressure (21.3 %) and number of children under 5 who are under weight (27.4%). In addition, the accuracy of DHIS data against the Summary Sheets was established for the period. The number of HIV+ pregnant mothers receiving preventive ARVs had an accuracy of 58%, total number of patients currently on prophylaxis Cotrimoxazole (32.9%), number of facility based maternal deaths (74.4%), number of women of reproductive age receiving family planning (13.2%), number of pregnant women referred for ANC (25.2%), number of fully immunized children (21.9%), number of children under 5 treated for malaria (12.1%), number of new outpatient cases with high blood pressure (60.7%) and number of children under 5 who are under weight (35.6%). The average accuracy was 36.9%. Verification of KePMS data against source documents established that only 19.6% of facilities had KePMS matching data with the source documents for total number of patients currently on prophylaxis Cotrimoxazole. A comparison of DHIS data to KePMS established that only 28.9% had matching data for total Number of HIV+ pregnant mothers receiving preventive ARVs and 39.1% of the facilities assessed had matching data for total number of patients currently on prophylaxis Cotrimoxazole. x

11 Performance at Sub-County level in terms of reporting rate, completeness and timeliness was assessed and found to range from 88% for reports availability, timeliness at 82% and completeness at 77%. Noteworthy is that only the habitually reporting facilities are tracked by the sub counties. On the qualitative aspect of the DQA, the systems assessment findings on factors affecting data quality were lack of training and support supervision for staff handling data, non-medical staff handling data (casuals), lack of data review measures, complex aggregation procedures, unclear indicator definitions especially for Immunization and HIV/AIDS, Family Planning, Malaria, and underweight with staff not sure what to count, chronic lack of tools resulting to improvising, lack of instructions especially on summary tools; some facilities not utilizing the standard tools and using those of partners, no written guideline available on data collection, aggregation, and manipulation procedures. In addition among the 44 health facilities assessed with Electronic Medical Record Systems (EMRs), the majority of them were installed for managing outpatient service utilization and for financial purposes rather than data generation and were either non-functional in data aggregation or malfunctioned when efforts were made to retrieve data. The conclusion drawn from this DQA were that the accuracy of summary data and DHIS data against the source documents was generally low and was aggravated by several systemic issues including lack of standardized tools, governance, standard operating procedures, indicator definitions, and unclear roles and responsibilities. There was only a slight improvement of accuracy of DHIS data against summary sheets despite having qualified HRIOs keying in this data due to lack of aggregation instructions and multiple service delivery sites generating data. The low availability of audit documents in private health facilities highlights the minimal inclusion of these facilities in the national HIS. Among the emerging recommendations were; sensitization and collaborative efforts by all stakeholders in investing in good data quality, development of data quality improvement plans to strategize on addressing the myriad of systemic issues affecting data quality, dissemination of the data quality assurance protocol, investment in technology to ease work load with regards to data and targeted efforts towards data use including data reviews and performance review forums, as well as regular data products generation and dissemination. xi

12 1. INTRODUCTION 1.1 Background Establishing a robust health information system able to support performance monitoring of health programs and track progressive improvement of health of Kenya citizenry is one of the flagship projects of Kenya Vision The Constitution of Kenya 2010 states that every person has a right to the highest attainable standard of health which includes the right to health care services including reproductive health and hence the need for transparency and accountability and public participation in monitoring health sector performance. The investments and inputs in health, accompanied with effective and efficient management of resources should translate to better and demonstrable health outcomes. In this regard, the quality of all health data and health related data needs to be beyond reproach in order to cast an authentic picture of progress and inform sound evidence based decision making process. This is of interest to County and National government who are obligated by law to give periodic reports and evaluate performance of National and County government with regard to health outcomes and recommend appropriate actions as stipulated in the County Government Act (2012) and Intergovernmental Relations Act (2012). 1.2 Health Information System Health Information is one of the key orientations of the attainment of the Health Policy objectives in order to reach the overarching goal of Better Health, in a responsive manner. The (draft) KHSSP III has outlined its mission of deliberately building progressive, responsive and sustainable technologically-driven, evidence-based and client-centred health system for accelerated attainment of highest standard of health to all Kenyans. It has further outlined the impact targets which include reduction in neonatal and maternal deaths. A well-functioning Health Information System (HIS) is critical for evidence-based decision making and monitoring of the interventions geared towards the attainment of these targets. Quality data is needed to inform the design of interventions and to monitor and evaluate plans and quantify progress towards predetermined treatment, prevention, and care targets. Attention to data quality ensures that target-setting and results reporting are informed by valid and sensitive information, and that reporting service providers are collecting and organizing this information in a consistent manner. 12

13 Quality data is data that is reliable, accurate, precise, and complete, provided in a timely manner, is truthful and maintains client confidentiality. 1.3 Data Quality Audit 2010 The last country-wide Data Quality Audit (DQA) was conducted in 2010 and among the key findings were; Timeliness of reporting and completeness of data was lower than expected. A comparison of the regions assessed in the three parameters revealed that all were above 60% for completeness, timelines and availability. Lower Eastern had all the three parameters being over 80% while Nyanza and Western had availability and completeness of over 90% though the timeliness was 80% for Western and 78% for Nyanza. The lowest performing region was North Eastern with availability of 79%, Completeness of 65% and timeliness of 69%. Data verification was either over-reported or under-reported for most of the indicators assessed. There was over-reporting for Women of Reproductive Age accessing FP commodities (105%), Pregnant Women receiving IPT2 (103%), New-borns with Low Birth Weight (115%), and PMTCT/ART at 122%. The verification revealed under-reporting of data for four ANC visits and Delivery by skilled health attendants in health facilities (99%) and gross under-reporting for maternal deaths in health facilities (90%). Data verification documents were not always available pointing to issues with storage of records Use of multiple tools to aggregate the data and the lack of data collection tools contributed to discrepancies observed in reported and recounted data Failure to use registers as per instructions was also noted while some indicators were not well understood. Since then, only vertical DQAs spearheaded by programmes have been carried out albeit sporadically and the findings have been at best shared within the specific programmes. There had been no country-wide data quality audit carried out in Kenya since installation and use of the DHIS platform in 2011 for the reporting of routine health facility service delivery and community health services. 1.4 Data Quality Assurance Protocol From the findings of the DQA report 2010, the Division of Health Information Systems (now the Division of Health Informatics and Monitoring and Evaluation DivHIME) developed a Data Quality Assurance Protocol whose purpose is to provide a uniform 13

14 approach and framework in which all stakeholders could participate in ensuring data quality. The DQA protocol offers a guideline for the implementation of DQA by all departments in the Ministry of Health, development partners, Non-governmental Organizations (NGOs), private sector, Faith Based Organizations (FBOs) and Community Based Organizations (CBOs) (HIS Policy, ). Data quality assurance strategies are envisioned to be implemented at all levels of the health system and to have a comprehensive approach to reflect national, regional, county and community coverage. The DQA 2014 will contribute to enriching the protocol and further elucidate on the roles of the different stakeholders in data quality assurance and improvement while highlighting the investments required. 1.5 Purpose of the Data Quality Audit The delivery and management of Health Service has been devolved to the counties. Each County is responsible for the health outcomes of their inhabitants and therefore need to implement a sound Monitoring and Evaluation framework to keep track of its targets. The quality of data collected to track progress will need to be high to ensure that counties are making well informed decisions on where to deploy resources. This DQA serves to provide baseline information on the quality of routine data on service utilization being captured in the current system and provide insights into the development and/or improvement of the DQI strategies and methods for the national and county governments. 1.6 Objectives The objectives of conducting the data quality audit were to: Verify the quality for key indicators at selected sites/levels Assess the ability of data management systems to collect, manage and report quality service utilization data. Identify corrective measures and develop action plans for strengthening the data management and reporting system and improving data quality 14

15 2. METHODOLOGY 2.1 Assessment Design A descriptive cross sectional design was adopted for this assessment targeting the service utilisation data collected between the months of July- September The survey utilised both qualitative and quantitative methods to verify the data from source documents (registers) for select indicators against summary data (reporting forms), DHIS data (software), and KePMS data (software). The assessment also collected qualitative data on the data management systems to determine their ability to collect, manage and report quality data. 2.2 Sampling and Site Selection The DQA was country-wide exercise conducted in 181 facilities selected across the 44 out of the 47 counties in the country (data from the counties of Garissa, Mandera, and Wajir was not collected due to (in) security circumstances surrounding the period of data collection. In each county, four (4) facilities were selected, which comprised at least one hospital and 2-3 primary health-care facilities -Health Centres, Nursing homes, Medical Clinics, and Dispensaries. There was purposive inclusion of the county hospitals where necessary. For the remaining three facilities, one sub-county (district) was selected from each county and 3 health facilities were randomly selected based on purposive sampling approach with an effort to make the sample as representative as possible in terms of facility ownership (Government, Private, and FBOs) and type/level of facilities (Hospital, Health Centre, Nursing Home and Dispensary). The sample also included three national referral hospitals (Spinal Injury Hospital, Mathari Referral Hospital, and Moi Teaching and Referral Hospital). Sixty-two percent (62%) of the sampled facilities are government owned while the remaining 38% belonged to private or faith-based organizations. Thirty-eight (38%) of the facilities are hospitals while the remaining 62% of are primary health-care facilities. However, at the time of data collection, some facilities were found non-operational and this altered the proportions slightly resulting in the proportions illustrated in Table 1 below. 15

16 Table 1: Distribution of DQA Facilities by Ownership/Level Ownership % of Number of % of Number of Facilities in Facilities in Facilities in Facilities in MFL MFL DQA DQA FBO % % MOH (levels 2,3) % % MOH (levels 4,5,6) % % Private % % Total % % 2.3 Indicator Selection In consultation with different programmatic service delivery areas of the MOH, nine (9) indicators were selected taking into account representation of programmatic areas and types of registers and summary tools used collect data to feed into the indicators. The following programmatic areas and health service delivery units were included as part of the assessment: National AIDS/STD control programme (NASCOP)-two indicators, Maternal and child health (MCH) programme-three indicators, Outpatient diagnosis (OPD)-two indicators, and RH/FP service and community health service unit-one indicator each. The nine indicators and their corresponding registers and summary tools are listed in Table 2 below: Table 2: List of Indicators, Registers and Summary tools Programmatic Area/ Health Indicators Service Unit Number of HIV+ pregnant mothers receiving preventive ARVs (ANC only) NASCOP Total No. of patients currently on prophylaxis - Cotrimoxazole RH/MCH OPD Community Health Services Summary Tool Register MOH 711 ANC MOH 405 MOH 711 Number of facility based maternal deaths MOH 717 Number of women of reproductive age receiving family planning MOH 711 Number of fully immunized children MOH 710 Number of children under 5 who are under weight Number of children under 5 treated for malaria Number of new outpatient cases with high blood pressure Number of pregnant women referred for ANC Daily Activity Sheet Maternity MOH 333 Family Planning MOH 512 Immunization MOH 510 MOH 711 CWC MOH 511 MOH 705 A MOH 705 B Under 5 OPD MOH 204A Over 5 OPD MOH 204B MOH 515 CHIS MOH

17 2.4 Assessment Tools The assessment tool was adapted from the global DQA tool developed by Global Fund. The tool comprises two main components: data verification and data management and systems assessment sections. The data verification section used to assess the availability, completeness, and accuracy of data for each of the audited indicators. The data management and systems assessment section include interview questions to assess the strength of the underlying factors that may affect data quality. Generally, the quality of reported data is dependent on the underlying data management and reporting systems; stronger systems should produce better quality data. 2.5 Data Collection Process A total of 22 teams each comprising of two officers were trained on data collection techniques using the RDQA tool (Error! Reference source not found.) during a three day workshop held at Lukenya Getaway. The teams had a one-day pre-test/practice on the tools during the training period. The pre-test/practice took place in Machakos Level 5 hospital, Bishop Kioko Mission Hospital and Muumandu Health Centre in Machakos County. An earlier pre-test of the tools by the DQA preparatory committee had taken place at Mbagathi Hospital in Nairobi County. The data collection teams were deployed to the field to collect data in 44 counties. Each team was assigned to eight (8) facilities selected across two counties and also to collect sub county level data in each of the assigned counties. A total of eight (8) supervisors supported the teams with six (6) supervising the teams in the field and two supervisors assisting with the technical issues arising from the tools. The period selected for review was July 1 st, 2013 to September 30th, 2013, which corresponds to the 1 st quarter of the 2013/14 fiscal year. All sampled facilities and subcounty health recording offices were visited between March 17 and March 28, 2014 excluding Lodwar County which was visited from 31 st March to 4 th April 2014 due to travel logistics. The data collection was done through document review for data verifications, and key informants interviews for systems assessment. Ideally, routine health data is collected in standardised registers at each facility where health-care services are delivered. Every month the staffs at the different primary health-care facilities collate the data and send monthly summaries on paper to sub-county Health Records and Information Officer (HRIO). The monthly summaries are then entered into web-based District Health Information Software (DHIS2) system by HRIO based within the sub-county health 17

18 office. Larger facilities such as referral hospital and sub-county hospitals have dedicated facility recording officers who themselves enter the facility data into the DHIS2 system. Sub-county HRIOs oversee the input of all data into the DHIS2. With this background in mind, the data verification was done through recounting data from the source documents for each indicator. In addition, the teams copied the figure in the summary tool for the corresponding month. The recounted and reported values were entered into corresponding cells of the excel RDQA questionnaire. As per the values entered, recounted figure were compared to the reported values. Summary statistics of all indicators are also calculated and presented graphically in the dashboard of the tool for each site and aggregation level scores. Figure 1: Data Verification (review and counting) in progress In addition the data reported in the DHIS for the period was obtained and compared with the recounted and summary data. The same case applied for data reported through the KePMS. Systems assessment was conducted through qualitative questions administered to health workers within the facilities to evaluate data management capacity. Information was collected on five areas of data management and reporting systems: Monitoring and evaluation capabilities, roles and responsibilities/ training Indicator definitions and data reporting requirements Data collection tools and reporting forms Data management processes and data quality controls Links with national reporting system 18

19 2.6 Data Analysis The data collection Excel sheets from the different health facilities were aggregated to summarize the data. Due to technical issues with the tools and the magnitude of the data collected, the automatic summation could not take place and compiling the data was done manually. A single sheet showing the summary of all the recounted data, summary data from the summary tools, DHIS data and KePMS data for the relevant indicators was prepared. A comparison across the different data sources was done and the proportion for accuracies calculated. The data was disaggregated by ownership and level GoK, level 4, 5, 6; GoK level 2, 3, Faith based facilities and Private. The proportions were weighted in relation to the contribution of level/ownership to the national data. The qualitative analysis/ system assessment utilized the summary scores obtained from scoring the different aspects of the five key areas under assessment. This was disaggregated into same level and ownership categories with a national outlook presented in a spider diagram. The comments and notes taken during the data collection exercise were analysed thematically and used to beef up the findings of the systems assessment and other qualitative findings. The Sub County analyses were also done to shed light on completeness, availability and timeliness of received reports. 2.7 Ethical Considerations Relevant authorization from the Cabinet Secretary of Health was sought before the commencement of the exercise. Authorization from the relevant county organs was also sought and from facilities in charges. The data collection teams were briefed on confidentiality and care was taken to ensure that the facility records were treated with utmost care and no records were carried away from the facilities. Where photographs were taken to illustrate the qualitative findings, care was taken not to expose the patients details. 19

20 3. FINDINGs 3.1 Background A total of 181 facilities were visited for the DQA exercise; of these 178 had data that could be used for the DQA. Sixteen percent (16%) were FBO, 34% MoH Levels 2 & 3, 30% MoH Levels 4, 5, & 6 and 20% were privately owned health facilities Availability of Services corresponding to Assessment Indicators Among the facilities assessed, some did not offer some of the services that corresponded to the indicators under assessment and therefore did not contribute data to these indicators. The facilities not providing a given service were excluded from the denominators in the calculation of proportions in the different DQA findings as detailed in Table 3. Table 3: Proportion of Facilities in the Sample Where the Services Corresponding to Indicators were not Available/Delivered Facility Ownership Indicator Number of HIV+ pregnant mothers receiving preventive ARVs Total No. of patients currently on prophylaxis - Cotrimoxazole Number of facility based maternal deaths Number of women of reproductive age receiving family planning No. of pregnant Women referred for ANC Number of fully immunized children Number of children under 5 treated for malaria Number of new outpatient cases with high blood pressure Number of children under 5 who are under weight Service associated with indicator PMTCT in ANC Care and Treatment Maternity / Deliveries Family Planning Community Health Unit Immunization Outpatient for Under 5 years Outpatient for over 5 years Growth Monitoring for Children FBO (n=28) 10.1% (3) 17.8% (5) 7.1% (2) 32.1% (9) 25% (7) 0% (0) 0% (0) 0% (0) 0% (0) MOH (levels 2,3) (n=60) 26.7% (16) 36.7% (22) 18.3% (11) 1.7% (1) 28.3% (17) 5% (3) 0% (0) 1.7% (1) 5% (3) MOH (levels 4,5,6) (n=54) 0% (0) 1.8% (1) 1.8% (1) 0% (0) 113% (7) 0% (0) 1.8% (1) 1.8% (1) 0% (0) Private (n=36) 36/1% (13) 50% (18) 22.2% (8) 11.1% (4) 44.4% 1 (16) 16.7% (6) 2.8% (1) 2.8% (1) 11.1% (4) 1 Ordinarily, Community units which report this indicator are attached to MoH and in some cases FBOs facilities. The finding could be as a result of data collection error 20

21 3.1.2 DHIS Reporting Rates for Indicators Assessed The reporting rates of data contributing to the nine indicators assessed in the DQA were determined from the DHIS. While the reporting rates for the forms for the facilities in the sample were not calculated, the reporting rates in the DHIS were used as an indicator of the level of reporting for each of the indicator. These are illustrated in Table 4. Table 4: DHIS2 Reporting Rates for Summary Tools Included in Audit Indicator Number of HIV+ pregnant mothers receiving preventive ARVs Total No. of patients currently on prophylaxis - Cotrimoxazole Number of women of reproductive age receiving family planning Number of children under 5 who are under weight Number of facility based maternal deaths No. of pregnant Women referred for ANC Number of fully immunized children Number of children under 5 treated for malaria Number of new outpatient cases with high blood pressure Summary Tool (n=expected reports nationally) MOH 711 (n=6,805) Integrated RH,HIV/AIDS, Malaria, TB and Nutrition MOH 717 (n=6,755) Service Workload MOH 515 (n=3,289) Community Health Extension Worker Summary MOH 710 V1(n=5,613) Vaccines and Immunizations (version 1) MOH 705 A (n=6,815) Outpatient Summary < 5 years MOH 705 B (n=6,896) Outpatient Summary > 5 years MOH (2375) HIV/AIDS Care and Treatment Reporting Rate DHIS2 2 July 2013 August 2013 September % 91.4% 89.8% 89.7% 89.7% 88.8% 54.3% 52.7% 54.6% 87.6% 86.8% 84.4% 91.7% 90.8% 91.4% 90.1% 89.8% 90.3% 80% 81.7% 80.3% 2 Reporting Rates for July-September 2013 as found in DHIS2 on 20th April 2014 calculated as No. of expected reports / No. of received reports x 100; Note that not all facilities offer all services 21

22 3.2 Data Verification Availability and Completeness of Source Documents In some facilities assessed, source documents i.e. standard registers, for different indicators were not available or were found to be incomplete for the auditing period. Table 5 presents by indicator and facility levels/ownerships, the proportion of facilities that were found at the time of the assessment to have source documents that were available and complete. Overall Pregnant women referred for ANC (39%), Number of fully immunized children (64%), Total number of patients currently on prophylaxis Cotrimoxazole (73%) and Under 5 children underweight (73%) had the lowest availability rates of source documents. In general, availability of source documents was found to be lower for private facilities than for government and faith-based facilities. In most cases the community-based service register (MOH514) was not available at facilities record offices for the auditing purpose as they were kept by Community Health Extension Workers (CHEWs) or Community Health Volunteers, in their homes. In some facilities HIV/AIDS care and treatment register (PRE-ART and Activity Sheets) were not available in the records offices because partners had custody of the registers. Where available, the audit teams assessed completeness of a register based on the extent to which the required data elements were filled in. A completeness rate was therefore, defined as the percentage of audited registers that have complete records for the required data elements. The lowest register completeness (64%) was observed for child immunization CWC register (MOH510) while the highest (91%) was for high blood pressure OP register (MOH204B). Some pages of source documents (such as CWC and OP registers) were also found missing (e.g. torn out) due to mishandling of the tools. Reasons cited for this, was size of the registers and presence of too many handlers including students. The results show that incompleteness of source documents was more pronounced among private facilities compared to government and faith-based facilities. In several facilities, tally sheets were presented as source documents particularly for immunization, whereas their true purpose is for intermediary use to aggregate numbers for different services provided. 22

23 Table 5: Availability and Completeness of Source Documents Indicator/Data collection tool (%)* FBO (n=28) GoK Level 2-3 (n=60) GoK Level 4-6 (n=54) Private (n=36) Weighted/ National Number of HIV+ pregnant mothers receiving preventive ARVs (ANC MOH 405) Available (%) % Complete (%) % Total No. of patients currently on prophylaxis Cotrimoxazole (Activity Sheet) Available (%) % Complete (%) % Number of facility based maternal deaths (Maternity MOH 333) Available (%) % Complete (%) % Number of women of reproductive age receiving family planning (FP MOH 512) Available (%) % Complete (%) % Number of pregnant women referred for ANC (MOH 514) Number of fully immunized children (Immunization MOH 510) Available (%) % Complete (%) % Available (%) % Complete (%) % Number of children under 5 treated for malaria (OPD MOH 204A) Number of new outpatient cases with high blood pressure (OPD MOH 204B) Number of children under 5 who are under weight (CWC MOH 511) Available (%) % Complete (%) % Available (%) % Complete (%) % Available (%) % Complete (%) % *It should be noted that the n shown in the title of columns is the TOTAL facilities assessed in each category. However, the percentages were calculated ONLY for those providing the service as detailed in Table 3. 23

24 3.2.2 Availability and Completeness of Summary Tools In some facilities, summary reports were also not available or were found to be incomplete during the data verification exercise. It was not possible to access summary reports for HIV/AIDS, community health services, child immunization and underweight indictors in more than 10% of the facilities visited. Some summary reports were incomplete and boxes for recording summary figures found blank e.g. the maternal deaths data. Therefore it was not possible to tell whether there were no maternal deaths or if data was missing. Unavailability or incompleteness of summary reports was more common for private and faith-based facilities than government facilities. The audit teams observed that some facilities used photocopied forms citing a shortage of reporting tools. In some facilities where there were multiple service delivery points, including community outreach and satellite clinics, other summary forms were used as intermediaries and collectively used to aggregate the final summary tools Missing Data/Value for Indicators under assessment for both Source and Summary Tools The assessment found that not all facilities with audit documents had data available to be audited for the months under review. This is illustrated in Table 6. The missing data was especially high for Number of HIV+ pregnant mothers receiving preventive ARVs (private), Total number of patients currently on prophylaxis Cotrimoxazole (FBO, MOH levels 2 &3, and Private) and the Number of pregnant Women referred for ANC across all facilities. The recording of Number of fully immunized children in private facilities was generally poor. 24

25 Table 6: Proportion of Missing Data/Value for Indicator (source documents and/or summary sheet) in Facilities Where Service is provided Facility Ownership MOH MOH Indicator FBO (levels (levels Private (n=28) 2,3) 4,5,6) (n=36) (n=60) (n=54) Number of HIV+ pregnant mothers 24% 11.4% 5.5% 43.4% receiving preventive ARVs (6/25) (5/44) (3/54) (10/23) Total No. of patients currently on 30.4% 26.7% 11.3% 55.5% prophylaxis - Cotrimoxazole (7/23) (8/30) (6/53) (10/18) Number of facility based maternal 30.7% 14.3% 7.5% 35.7% deaths (8/26) (7/49) (4/53) (10/28) Number of women of reproductive age 15.7% 0% 3.7% 9.4% receiving family planning (3/19) (0/59) (2/54) (3/32) No. of pregnant Women referred for 61.9% 25.6% 46.8% 70% ANC (13/21) (11/43) (22/47) (14/20) Number of fully immunized children 14.3% (4/28) 7.0% (4/57) 9.2% (5/54) 63.3% (19/30) Number of children under 5 treated for 7.1% 5% 7.5% 14.3% malaria (2/28) (3/60) (4/53) (5/35) Number of new outpatient cases with 3.6% 3.4% 13.2% 14.3% high blood pressure (1/28) (2/59) (7/53) (5/35) Number of children under 5 who are 28.6% 8.8% 12.7% 40.1% under weight (8/28) (5/57) (7/54) (13/32) Accuracy of Summary Sheet against the Source Documents From the recounting done on the available source documents, the assessment determined the extent to which the data on the summary sheets reflected what was recounted in the source documents. The findings are illustrated in Table 7. The accuracy of data for majority of the indicators under assessment was very low especially for Number of women of reproductive age receiving family planning. The explanation for this was that it was not clear what constituted family planning. Some facilities recorded condoms while others did not and some facilities included men issued with condoms in the summation. It was also not clear on how to treat first time visits and revisits. Another example was the definition of a fully immunized child with 25

26 some facilities recording Measles vaccine at 1 year, 2 nd Measles vaccine at 18 months while others recorded Measles vaccine whether the child was up to date with other vaccines or not. Poor recording of diagnoses of malaria with multiple terms being used also contributed to the inaccuracies between the two records. It was also noted that amongst the four categorized health facilities, MOH hospitals at levels 4, 5 and 6 were found to have the lowest levels of accuracy for several indicators even where there were designated HRIOs on site. Most of the data reported in the summary sheet did not match what was recounted in the source documents. Some reasons cited included multiple service delivery points, high volume of clients/patients versus low staff numbers, unsupervised recording by students on site and handling of source documents by multiple staff. Table 7: Proportion of facilities with Source documents data matching Summary Sheet data Indicator Number of HIV+ pregnant mothers receiving preventive ARVs Total No. of patients currently on prophylaxis - Cotrimoxazole Number of facility based maternal deaths Number of women of reproductive age receiving family planning No. of pregnant Women referred for ANC Number of fully immunized children Number of children under 5 treated for malaria Number of new outpatient cases with high blood pressure Number of children under 5 who are under weight Summary Sheet MOH 711 MOH 711 MOH 717 MOH 711 MOH 515 MOH 710 MOH 705 A MOH 705 B MOH 711 FBO (n=28) 32% (8/25) 21.7% (5/23) 69.2% (18/26) 21% (4/19) 28.5% (6/21) 21.4% (6/28) 35.7% (10/28) 32.1% (9/28) 25% (7/28) MOH (levels 2,3) (n=60) 40.9% (18/44) 36.7% (11/30) 85.7% (42/49) 1.7% (1/59) 34.9% (15/43) 3.5% (2/57) 21.7% (13/60) 27.1% (16/59) 29.8% (17/57) % Accurate MOH (levels 4,5,6) (n=54) 20.3% (11/54) 15.1% (8/53) 49% (26/53) 3.7% (2/52) 34% (16/47) 3.7% (2/54) 24.5% (13/53) 28.3% (15/53) 9.2% (5/54) Private (n=36) 34.7% (8/23) 33.3% (6/18) 60.7% (17/28) 6.2% (2/32) 30% (6/20) 47,4% (9/19) 42.9% (15/35) 31.4% (11/35) 37.5% (12/32) Weighted National (n=8523) 36.6% 32.6% 72.6% 6.2% 32.1% 22.9% 31.9% 29.5% 31.4% Average 31.8% 26.8% 20.9% 36.0% 30.8% 26

27 3.2.5 Accuracy of DHIS Data against the Source Documents The accuracy of DHIS data against the source documents was assessed for the period under review to establish whether the entries matched what was in the source documents. Notably, the proportion of matching data between Source documents and DHIS was very low for all indicators under review. The Number of women of reproductive age receiving family planning had the lowest accuracy at 7.2%, while the Number of children under 5 treated for malaria had an accuracy of 12.8% while the Number of fully immunized children was at 16.1%. The overall average for matching source/dhis data was at 27.7% for the data of the indicators assessed. The results are illustrated in Table 8. Notable, was the poor performance of MoH levels 4, 5, & 6 in all indicators despite having trained HRIOs and accessibility to Electronic Medical Records. Table 8: Proportion of facilities with Source Documents data matching DHIS2 Data 3 Indicator Number of HIV+ pregnant mothers receiving preventive ARVs Total No. of patients currently on prophylaxis - Cotrimoxazole Number of facility based maternal deaths Number of women of reproductive age receiving family planning No. of pregnant Women referred for ANC Number of fully immunized children Number of children under 5 treated for malaria Number of new outpatient cases with high blood pressure Number of children under 5 who are under weight Summary Sheet MOH 711 MOH 711 MOH 717 MOH 711 MOH 515 MOH 710 MOH 705 A MOH 705 B MOH 711 FBO (n=28) 32% (8/25) 17.4% (4/23) 73.1% (19/26) 21.1% (4/19) 19% (4/21) 10.7% (3/28) 14.3% (4/28) 21.4% (6/28) 21.4% (6/28) MOH (levels 2,3) (n=60) 43.2% (19/44) 30% (9/30) 85.7% (42/49) 6.8% (4/59) 25.6% (11/43) 5.3% (3/57) 11.7% (7/60) 25.4% (15/59) 19.3% (11/57) % Accurate MOH (levels 4,5,6) (n=54) 5.5% (3/54) 1.9% (1/53) 62.3% (23/53) 0% (0/54) 19.1% (9/47) 3.7% (2/54) 3.8% (2/53) 15.1% (8/53) 9.3% (5/54) Private (n=35) 39.1% (9/23) 27.8% (5/18) 64.3% (18/28) 3.1% (1/32) 25% (5/20) 31.6% (6/19) 14.3% (5/35) 17.1% (6/35) 40.6% (13/32) Weighted National (n=8523) 38.8% 26.5% 75.0% 7.2% 24.2% 16.1% 12.8% 21.3% 27.4% Average 25.6% 28.1% 13.4% 29.2% 27.7% 3 The data used for comparison was extracted from DHIS2 as at May,