REPUBLIC OF RWANDA MINISTRY OF HEALTH. Data Validation and Verification. Procedure Manual

Transcription

1 REPUBLIC OF RWANDA MINISTRY OF HEALTH Data Validation and Verification Procedure Manual Version, 2011

2 2

3 ACKNOWLEDGEMENT The Ministry of Health is grateful to all the different stakeholders who contributed to the development of the Standard Operating Procedures for Management of Routine Health Information in Rwanda. These procedures would not have been finalized without the usual and dedicated participation of the Ministry of health staff who played an active role throughout the entire development process of this procedure manual; and these include but not limited to: The Ministry of Health Central level staff from HMIS, the Unit in charge of integration and decentralization, MCH; HIV/AIDS, Malaria, TB units of the Rwanda Biomedical Centre. The School of Public Health at the National University of Rwanda (SPH-NUR) for their tremendous support. To all the Directors of District Hospitals, District M&E Officers and Data Managers and other stakeholders who participated actively in writing the different chapters. Our appreciation also goes towards the USG (United States Government) whose, technical and financial support contributed to the realization of these procedures; please accept our heartfelt gratitude. 3

4 TABLE OF CONTENTS ACKNOWLEDGEMENT INTRODUCTION DATA VALIDATION AND VERIFICATION REQUIREMENT WHAT IS DATA VALIDATION AND VERIFICATION IMPORTANCE OF DATA VALIDATION AND VERIFICATION REQUIREMENT TO USE THIS MANUAL AND TOOLS OVERVIEW OF THE DATA VALIDATION AND VERIFICATION PROCESS GENERAL PRINCIPLES OF DATA QUALITY DATA VALIDATION AND VERIFICATION SCOPE BASIC CONCEPTS AND TERMS USED IN THIS MANUAL DATA VALIDATION AND VERIFICATION PROCEDURES Complete Data Validation and Verification Training Preparing for Data Validation Composition of the Review Team Roles of Each Team Member Decide on which institution and program/disease area to be assessed Decide on the indicators Select Period of Review Notification Perform the desk review of documentation Required Materials for Data validation Site Visit Data Validation Conduct entrance briefing Conduct interviews with staff Observe data collection and reporting process Source document review Review of transcription process Spot checking Completeness and Accuracy of Monthly Report Extract Report Conduct exit briefing PROCEDURES FOR FEEDBACK/DEBRIEF Feedback after the Data validation and verification Requirements for Regular Feedback LIST OF APPENDICES Appendix A: Final Report Template Appendix B: Example of Notification Letter Appendix C: Correction Form Appendix D: Examples of Quantitative Feedback

5 INTRODUCTION 1.1 DATA VALIDATION AND VERIFICATION REQUIREMENT The Ministry of Health and Development Partners require that Health care institutions and implementing partners delivering health care services be subject to routine review to validate and verify data reported to Government and Development Partners. The purpose of the routine data validation and verification is to ensure that health care institutions and implementing partners are reporting health and other health-related data that are reliable, valid, complete, comparable, and timely. The validated and verified data will improve reporting and will provide MOH and key stakeholders with assurance that data are credible and consistently collected and reported in accordance with standard procedures and guidelines. The primary purpose of this Procedure Manual (Manual) is to provide health care institutions and implementing partners with information regarding the conduct of data validation and verification in order to meet the reporting requirements. The Manual provides background information and an overview of the data validation and verification process, discusses the scope and timeframe required for the data validation and verification, and describes the tools and processes used for conducting the data validation and verification. 1.2 WHAT IS DATA VALIDATION AND VERIFICATION Once data has been collected, its quality must be verified before it is electronically or entered or captured in a computer or used for reporting or making decisions. Data validation and verifications forms part of the criteria of information quality. Data validation and verification is the confirmation of the accuracy, completeness, consistency and timeliness of data. 1.3 IMPORTANCE OF DATA VALIDATION AND VERIFICATION Reduce errors and inconsistencies within the data collected, reported and used. Improve the quality of data to meet reporting requirements. Provide accurate, consistent, complete and timely data to all stakeholders Decisions made based on validated data are more defensible Ability to compare data across institutions and organizations As the data becomes more accurate, less time and effort is spent on the validation and verification process, ultimately saving time and money. 5

6 1.4 REQUIREMENT TO USE THIS MANUAL AND TOOLS The Ministry of Health requires Health Care Institutions and Implementing Partners (IPs) to use the processes and tools contained in this Procedure Manual and its appendices to conduct the routine data validation and verification. This includes each of the following documents: 1. Data Validation and Verification Standards 2. Routine Data Quality Assessment Checklist and Tool 3. DQA Report Template and Action Plan Form 4. Error Correction Form 5. Standard Operating Procedures for Management of Routine Health Information 2 OVERVIEW OF THE DATA VALIDATION AND VERIFICATION PROCESS 2.1 GENERAL PRINCIPLES OF DATA QUALITY This manual recognizes that they are two keys to the improvement of data quality they are prevention and correction. The following general principles are recommended for any institution or individual dealing with data quality: 1. Set a data vision or develop and implement a data policy and strategy not by carrying out unplanned, uncoordinated and non-systematic data cleaning activities 2. Assign responsibility for the quality of data to those who create them. If this is not possible, assign responsibility as close to data creation as possible 3. Most data comes into an organization from suppliers, and it is much easier to develop good data collection practices than to correct errors downstream 4. Data ownership and custodianship not only confers rights to manage and control access to data, it confers responsibilities for its management, quality control and maintenance. Custodians also have a moral responsibility to superintend the data for use by future generations 5. Users and collectors have important roles to play in assisting custodians in maintaining the quality of the data in the collections, and both have a vested interest in the data being of the highest possible quality. 6. Your or your organization is not the only one that is dealing with data quality. To make the data of highest value to the greatest number of users in the shortest possible time, it may be necessary to prioritize the capture and/or validation of the data 7. Not all data are created equal, so focus on the most important, and if data cleaning is required, make sure it never has to be repeated. 8. Before measuring data quality levels, first consider how users of the results might use them and then structure the results so that they can be used most effectively. 6

7 9. Don t be seduced by the apparent simplicity of data cleaning tools. They are valuable and help in the short-term but, over the longer-term, there is no substitute for error prevention. 10. Outlier detection can be a valuable validation method, but not all outliers are errors. 11. Performance targets are a good way for an organization to maintain a consistent level of quality checking and validation for example 95% of all records are documented and validated within 6 months of receipt. 12. Effective feedback channels with users and suppliers is an easy and productive mechanism of improving data quality 13. Poor training lies at the root of many data quality problems 14. The data must be documented with sufficient detailed description to enable its use by third parties without reference to the producer of the data 15. Data which are no longer required (for legal or other reasons) should not be destroyed, or put at risk without exploiting all other possibilities including archiving 16. Data integrity is preserved through good data management, storage, backup and archiving 17. Data integration produces higher quality results when contributing data custodians have followed and used consistent data storage standards 18. Effective data quality programs help prevent embarrassment to the organization and individuals both internally and publicly 19. Most agencies and groups involved in producing data will be judged on the ease at which the data and information is made available, and the quality of the information. Those that are able to publish, share, access, integrate and use information are those that will benefit most 2.2 DATA VALIDATION AND VERIFICATION SCOPE The Ministry of Health requires that data validation and verification be conducted routinely every after a report has been submitted. The routine data validation and verification must retrospectively review all data submitted to MOH during the previous reporting period and any other reporting periods deemed necessary by the review team. The findings from the data validation and verification review must be submitted to MOH within five working days after each review and discussed by the institution reviewed within two working days after the review. 7

8 Data Measures Requiring Data Validation and Verification Measure Health Center and Post Monthly Report District Hospital Monthly Report Reporting Period Data Submission due date(s) to MOH Monthly 2 nd day of the following month Monthly 5 th day of the following month District Health Report Monthly 10 th day of the following month MOH Quarterly Quarterly 15th April report, Global Fund Report and other Key stakeholder reports 15 th July 15 th October 15 th January PEPFAR Report Semi-Annual 20th April 20 th October Data Validation and Verification findings due to MOH 15th April 15 th July 15 th October 15 th January 15th April 15 th July 15 th October 15 th January 15th April 15 th July 15 th October 15 th January 15th May 15 th August 15 th November 15 th February 15 th May 15 th November Responsible District Hospital Central level DQA team(s) coordinated by MOH/HMIS& M&E Central level DQA team(s) coordinated by MOH/HMIS& M&E Central level DQA team(s) Central level DQA team(s) 2.3 BASIC CONCEPTS AND TERMS USED IN THIS MANUAL Data element and Data: Data Element is a record of a health event or health-related event. Data is an aggregation of data elements - in the form of numbers, characters, images -that gives information after being analyzed Example 1: it is raining Example 2: Number of deliveries assisted by a skilled health worker Information: is data organized with reference to a context.- which gives data a meaning Example 1: the temperature dropped 15 degrees and then it started raining Example 2: Percent of deliveries assisted by a skilled health worker and percent of deliveries with no skilled health worker assistance Knowledge: when information is analyzed, communicated and acted upon, it becomes knowledge 8

9 Example 1: If the humidity is very high and the temperature drops substantially the atmosphere is often unlikely to be able to hold the moisture so it rains Example 2: why do some pregnant women received assistance from skilled health workers during labor? Why some pregnant women were not attended to by a skilled health worker during labor? Who are those women? What are the issues related to access to service? An indicator: is a data element placed in a given context so that it becomes information that can be used for program monitoring, management, and action Example: total number of children aged between 12 to 23 months who have been given measles vaccine Data quality refers to the extent to which data measures what they intend to measure Dimensions of data quality: Accuracy/Validity: Accuracy/validity of data is the degree to which data correctly reflects the true value or how close the data is to the true measurement Examples of Data Accuracy The age of the client in the database is the true age of the person. The reported number of VCT clients who got tested and received their results in the database/register is the actual number of clients tested. The address of the client in the register is the true address Completeness Completeness of data is the extent to which the expected data entries are provided i.e. data represents the complete list (and just a fraction) and no data fields left empty. For example, a client data is considered as complete if: All the data entries and other information supposed to be filled in the register have been actually entered. Data Completeness definition is the 'expected completeness'. It is possible that data is not available, but it is still considered completed, as it meets the expectations of the user. Data can be complete, but inaccurate: All the information about the VCT client is filled in the VCT register, but many of them are not correct. The health records of all patients have 'last visit' date, but some of it contains the future dates. Data Consistency/Reliability: The data generated by the health facility are based on protocols and procedures (e.g standardized tools) that do not change according to who is using them and when or how often they are used. The data are reliable because they are measured and collected consistently. Examples of data in-consistency are: 9

10 A nurse measuring the patient s heart beat by hand using a carotid pulse (neck) or radial pulse (wrist) in one facility and the other using a heart rate monitor or stethoscope in another facility. Both may record the actual counts but the measurement procedures or methods are not uniform (i.e. they are inconsistent),an ART client is dead, but he still has his drugs supply active. Data can be accurate (i.e., it will represent the true value), but still inconsistent because the instruments are different with different measurements. An VCT promotion campaign closure date is Jan 31, and there is a VCT client reached through the same campaign on Feb. 2. Data Timeliness: As someone once said 'Data delayed' is 'Data Denied' data are timely when they are upto-date (current-belong to the reporting period), and when the information is available on time. Timeliness is affected by: (1) the rate at which the program s information system is updated; (2) the rate of change of actual program activities; and (3) when the information is actually used or required. The timeliness of data is extremely important. This is reflected in: Health Facilities are required to publish their quarterly results with in a given frame of time. Doctors need up-to date information on the ART Clients. Precision: This means that the data have sufficient detail. For example, an indicatorrequires the number of individuals who received HIV counseling & testingand received their test results, by sex of the individual. An information system lacks precision if it is not designed to record the sex of the individual who received counseling & testing. Integrity Integrity is when data generated by a program s information system are protected from deliberate bias or manipulation for political or personal reasons. An example of an integrity issue is when the facility changes the figures to please the donor. Data Cleaning: A process used to determine inaccurate, incomplete, or unreasonable data and then improving the quality through correction of detected errors and omissions. 3 ATA VALIDATION AND VERIFICATION PROCEDURES This section outlines procedures and processes that the reviewer or review team will use to determine whether the institution s data is accurate, valid and reliable. 10

11 3.1 Complete Data Validation and Verification Training The Ministry of Health with support from USAID/MEASURE Evaluation Project has developed a training course that provides an opportunity for all institutions providing health care services to learn more about data validation and verification. USAID/MEASURE Evaluation is in the process of also developing a web-based Data Validation and Verification Training for those that cannot attend the 5-day trainings organized at different sites and for those that require refreshment on data validation and verification processes and procedures. A link to the web-based training site will be provided by MOH once it is finalized in During the data validation and verification preparation phase, all staff involved in the data validation and verification should complete the Data Validation and Verification training individually to familiarize themselves with the data validation and verification process and requirements. 3.2 Preparing for Data Validation In preparation for a data validation exercise, it is recommended that you develop a written scope or statement of work. A scope or statement of work (SOW) is a formal document that captures and defines the work activities, deliverables and timeline an individual or team will execute against in performance of specified work. The scope of work must outline the following: Composition of the Review Team During the development of the SOW, the team to conduct the data validation and verification should be identified and communication made in time to all team members. The followings are suggested positions/members to be part of the review team: District M&E Officer Data Manager Health Facility Supervisor with thorough knowledge of the disease program area OPTIONAL: A member from the partners that support the health facility The Health Facility In-charge or Unit In-Charge- whose role is to observe The leader of the team should be the District M&E Officer. In his/her absence, the Data Manager can be the Team Leader Roles of Each Team Member District M&E Officer shall be the Team Leader and is responsible for: 1. Arranging for any courtesy call with the relevant Health Facility Management Staff 2. Planning all the field visits with health facilities 3. Overseeing team fieldwork and allocating tasks as necessary 4. Facilitating introduction of the team and explanation of the RDQA at all visits 11

12 5. Delegate tasks and take part in data verification, crosschecks and spot-check activities 6. Appointing someone to list all tools used to collect, aggregate and report data at the health facility. 7. Responsible for developing an action plan specific to the Health Facility 8. Conducting all Health Facility debriefing 9. Responsible for the final RDQA Report 10. Coordinate and compile the final DQA Report 11. Keep files of all original documents collected during fieldwork (electronic and paper-based) 12. In the field, communicate any urgent matters to other members. Data Manager 1. Liaise with Health Facility Staff to avail all the source documents like registers and monthly reports 2. Performs a cross-checks between source documents and data from the database 3. Coordinate logistics and meeting times for all health facilities 4. Take part in data verification, crosscheck and spot-checks activities 5. Write sections of the final DQA Report as directed by the Team Leader. 6. Fill in and maintain electronic copies of the RDQA tool for all visits 7. Performs any other duty assigned by the Team Leader Health Facility Staff 1. Responsible for making the team access the facility and source documents and other information 2. Provides explanations to questions in the RDQA tool 3. Invite other health facility staff to the debrief meetings 4. Participate in the cross-checks, spot checks, data verification and discussion of the action plan 5. Performs any other duty assigned by the Team Leader Representative from the Implementing Partner, 1. Act as independent observer 2. Participate in data validation and verification 3. Participate in the development of action plan Decide on which institution and program/disease area to be assessed. The institution(s) where data validation is intended to be conducted should be specified in the SOW including the program or disease areas to be assessed. It is recommended to review a maximum of three program/disease areas per each visit in order to comprehensively cover all the items and issues and then cover the rest during subsequent follow-up visits. 12

13 3.2.4 Decide on the indicators. The criteria for selecting indicator to be reported include the followings: The indicator is one of the key indicators in the disease area The indicator is being monitored or supposed to be monitored by the institution Since the data validation activity is designed to improve on the system s performance, in certain situations, it is advisable to combine indicators that have had problems during their collection and reporting with at least one indicator that has not been problematic Select Period of Review The Team has to decide which Month, Quarter, or Semi-Annual Report to assess. In most cases, the latest quarter comprising quarterly and Semi-Annual reports is preferred in order to validate whether what was reported is accurate, reliable and valid Notification Notify the staff of the institution to be visited. If the institution is supported by a particular partner, the partner should also be notified. The notification letter should go two weeks before the visit. See appendix for sample of notification letter Perform the desk review of documentation Before field work, every team member should review: The previous monthly, quarterly or annual reports Data collection and reporting forms for the indicator(s) being assessed. Instructions for completing the data collection and reporting forms How the data is recorded on a source document, to other documents, such as clinic registers and into Facility Monthly Report How the data moves from Health Facility to District Hospital (e.g. district offices, project offices, etc.) and eventually, to the central level database Required Materials for Data validation 1. Hard Copy of data validation tool 2. Electronic version of data validation tool 3. Monthly Report(s) to be assessed 4. The Standard Operation Manual (this manual) 5. Plain Sheets of paper to be used as tally sheet 6. Note book 7. Calculator 8. Pens 13

14 3.3 Site Visit Data Validation The Review Team should use the Excel template of the data validation and verification tool to record notes and enter findings for the different dimensions of data quality measured. For each step while conducting a site data validation, a corresponding section of the data validation and verification tool is attached. When using the tool, the review team should only complete areas displayed in white for review results and findings, and those displayed in yellow for comments. Areas displayed in gray are not applicable and should not be completed Conduct entrance briefing The review team should pay a visit first to the institution in-charge or director. The review team should request to have an introduction with the specific site staff of the program/disease areas to be reviewed to discuss expectations for the site visit. During the entrance briefing, the review team should discuss the objectives for the review and discuss any administrative needs of the review team. They should review the day s agenda with the institution s staff and get updates. Name of Site Reviewed Program/Disease area reviewed Reporting period reviewed Date of review RWANDA DATA VALIDATION AND VERIFICATION TOOL Names of Review Team Conduct interviews with staff The review team is advised to conduct interviews with site staff involved in the collection, compilation and reporting of health related data for each program/disease area being assessed. These interviews provide a first-hand opportunity for the review team to gain a thorough understanding of each institution s data collection and reporting processes. Any outstanding questions and follow-up issues identified during the previous site visits should be addressed during the interviews Observe data collection and reporting process During the site, request to observe the data collection and reporting processes. The review team can request for visual demonstrations of the data systems and reporting process including data transcription from source documents to registers, data compilation using tally sheets and into monthly report, and any electronic data entry if it takes place at the site. Visual demonstrations provide a clear illustration of the data collection and reporting processes, provide the review team with insight into the institution s ability to ensure accurate, valid and timely data, and allow an opportunity to get immediate responses to any questions or concerns about the reported data. 14

15 1. DESCRIPTION OF THE RECORDING PRACTICES IN RELATION TO SERVICE DELIVERY - Describe the connection between the provision of Services and the completion of the source document Specific notes for Review Team It is recommended that the Review Team ask staff to describe the process through which the source documents are filled in relation to the provision of services. Determine the source document used for recording of services provided to clients. Source documents are the first place services provided are recorded. It may be patient dossier/patient card or clinical review forms or register. If access to patient dossier/patient card or clinical review forms is not possible, an alternative source document may be the register Describe the source document for recording the provision of services (is it a standardized form following Ministry of Health guidelines or a tailored form? If tailored, specify the source of the form, e.g. a project). Obtain a blank copy, if possible. Does the site have sufficient supplies of blank source documents (prompt for experience of stock-outs of source documents)? Describe when recording of the provision of services takes place, on what form(s) and by which staff member(s). Are there indications that there are delays between delivery of the provision of services and recording of the provision of services on the source document? If the provision of services and recording of the provision of services are not done at the same time, please describe how the disconnect might affect data quality. 1.6 Does the site have Standard Operating Procedures (SOPs) for health information management? 1.7 Have site staff been trained in the use of Standard Operating Procedures (SOPs) for health information management? Additional Comments (if any) Source document review A review of source documents should be done to ensure that all source documents accurately capture the required data fields and are properly completed. The following criteria should be followed by the review team while reviewing source documents: Criteria for validating source documents: Storage and record keeping check a. Ensure that all source documents are properly secured b. Ensure that locations of source documents (filing system) is properly referenced so that source documents can be retrieved at any time to validate the information c. Ensure that there s enough copies of source documents available d. Ensure that the records are legible Completeness check e. Mandatory data fields have been filled with data. This means that data or information must be present for a mandatory field e.g. name, age, sex, location, provenance etc 15

16 f. Expected data fields have been filled with data. This means that data or information for the service provided has been filled in the appropriate data field. Accuracy and consistency check: g. Data type if valid. Ensure that the data type for the field is valid, for example, checking that an integer field does not contain any alphabetic characters or that a date field contains a valid date h. Range checking to ensure that the values contained in the field are within acceptable limits. A minimum and a maximum limit sometimes is defined for certain fields, for example, for complete immunization, usually the age range is from 0 to 23 months (depending), for maternity mothers or reproductive health, it is usually from 15 years to 49 years though there can be some unusual cases i. Coding checking ensures that values are checked against a set of codes usually defined in the data definition or instructions, for example, in the new HMIS registers have ICD codes. The in-patient register under the column reason for exit has code 1 authorized discharge, 2 for unauthorized discharge etc. j. Check if terms used, for example medical abbreviations or ICD codes, are used properly as per defined regulations and guidelines k. Check if all data elements are accurately recorded l. Check if the right forms or registers are used for recording of data for the program/disease area being reviewed 16

17 2. DOCUMENTATION REVIEW - Review availability and completeness of all source documents for the selected reporting period Specific notes for Review Team Due to confidentiality regulations, it may be important that prior permission to review the source documents be obtained from the site's senior manager. Furthermore, the assessment team should ask the site manager if he/she would prefer that another staff member be present while the source documents are being reviewed. A) Check Availability and Completeness of Documentation Review available source documents for the reporting period. Is there any indication that source documents are missing? If yes, determine how this might have affected reported numbers. Are all available source documents complete, notably the mandatory data fields have been filled with data (e.g. date; serial number, name; address; sex; age, catchment area and important information regarding services provided/received by the patient) Review the dates of provision of services on the source documents. Do all dates fall within the reporting period? C) Check Accuracy and Validity of data Specific notes for the Review Team: If feasible, select 5% of patient records (or at least 20 patient records) of patients who were provided/received services during the review period 2.4 How many patient records were selected? Check each record to verify whether the data elements are: accurately record, valid, within acceptable range, proper terms 2.5 or abbreviations or codes used, no missing data. How many patients records have accurate or valid data elements? Calculate % difference for accuracy If difference is below 90%, select an additional 5% of patient records (or at least an extra 10 records) and redo the calculation (ADD the numbers to the existing numbers in the above cells); repeat up to three times. Additional Comments (if any) Review of transcription process A review of the transcription process should be done to ensure that data items transcribed are the same as the data items in the source documents. The following criteria should be followed by the review team while reviewing source documents: Criteria for validating transcribed documents: Storage and record keeping check a. Ensure that all transcribed documents are properly secured b. Ensure that locations of transcribed documents (filing system) is properly referenced so that they can be retrieved at any time to validate the information c. Ensure that there s enough copies of transcription documents available d. Ensure that the transcribed records are legible Completeness check e. Mandatory data fields have been updated with data. This means that data or information must be present for a mandatory field e.g. name, age, sex, location, provenance etc 17

18 f. Expected data fields have been updated with data. This means that data or information for the service provided has been updated in the appropriate data field. For example, a patient visit for HIV care and treatment should ensure that the source document (patient dossier) as well as the pre-art or ART register has been updated with the same information for the data fields. Accuracy and consistency check: g. Data type if valid. Ensure that the data type for the field is valid, for example, checking that an integer field does not contain any alphabetic characters or that a date field contains a valid date h. Range checking to ensure that the values contained in the field are within acceptable limits. A minimum and a maximum limit sometimes is defined for certain fields, for example, for complete immunization, usually the age range is from 0 to 23 months (depending), for maternity mothers or reproductive health, it is usually from 15 years to 49 years though there can be some unusual cases i. Coding checking ensures that values are checked against a set of codes usually defined in the data definition or instructions, for example, in the new HMIS registers have ICD codes. The in-patient register under the column reason for exit has code 1 authorized discharge, 2 for unauthorized discharge etc. j. Check if terms used, for example medical abbreviations or ICD codes, are used properly as per defined regulations and guidelines k. Check if all data elements in the source document where properly transcribed to the right data fields in the transcribed document and are accurately recorded l. Check if the right forms or registers are used for recording of data for the program/disease area being reviewed Spot checking A spot check is an inspection or investigation carried out at random or limited to a few cases. A spot check is designed to ensure whether the availability of source documents and other documents used for data collection, and if data collected, transcribed and reported is valid, accurate and complete. Performing a spot check on transcribed documents a. Randomly select a manageable number of records from the transcribed documents e.g. registers b. Identify the records and request to review the source documents of those records in the transcribed document c. Review the completeness, accuracy and validity of the data items in the source document and transcribed document Performing a spot check on source documents a. Randomly select a manageable number of source documents b. Identify the summarized records in the transcription documents e.g. registers c. Review the completeness, accuracy and validity of the data items transcribed and the source document data items 18

19 3. CROSS CHECKS - Perform cross-checks to ascertain the accuracy of the source document. Specific notes for Review Team: The review team can add other relevant cross-checks as appropriate. To the extent relevant, the cross-checks should be performed in both directions (for example, from patient dossiers to the Register and from Register to patient dossiers ). CROSS-CHECK 1.1 : From patient dossiers to the Register. Was this cross check performed? If feasible, select 5% of patient dossiers (or at least 20 patient dossiers) of patients who were provided/received services during the period of review. How many patient dossiers were selected? How many of the patients selected were recorded in the Register with all of the necessary/required information matching? Calculate % difference for cross check 1.1 If difference is below 90%, select an additional 5% of patient dossiers (or at least an extra 10 patient dossiers) and redo the calculation (ADD the numbers to the existing numbers in the above cells); repeat up to three times. CROSS-CHECK EXAMPLE 1.2 : From the Register to patient dossiers Was this cross check performed? If feasible, select 5% of patients listed in the Register (or at least 20 patients) who are listed as having provided/received services during the review period. How many patients were selected? How many of the patients selected had patient dossiers with matching information as in register? Calculate % difference for cross check 1.2. If difference is below 90%, select an additional 5% of patient records (or at least an extra 10 records) from the register and redo the calculation (ADD the numbers to the existing numbers in the above cells); repeat up to three times. - - Additional Comments (if any) Completeness and Accuracy of Monthly Report Using tally sheets designed for the register, recount all data in the register for the monthly report under review. Ensure that disaggregations as required by the monthly report are included in your tally sheet. Compare the recounted data in your tally sheet (including the disaggregations and totals) with the reported data in the monthly report (including the disaggregations and totals). These are two independent tables prepared by the staff at the health Facility and the review team during the time of assessment. The green part is from the Monthly Report while the yellow part is prepared during the assessment. From Monthly Report From your Tally Sheet Zone None-Zone None-District Total Zone None-Zone None-District Total New pregnant women registered ANC visit 1st trimester ANC visit 2nd trimester ANC visit 7th or 8th month ANC visit during 9th month Number of women with 4 standard ANC visits Number of women who made non-standard ANC visits Number of high risks pregnancy detected Number of high risk pregnancy referred

20 Compare the recounted data in your tally sheet (including the disaggregations and totals) with the reported data in the monthly report (including the disaggregations and totals). 4. TRACE AND VERIFICATION - Recount results from source documents, compare the verified numbers to the site summary reports and explain discrepancies. A) Recount results from source documents and compare the verified numbers to the site reported numbers Recount the number of people/cases/events recorded that were provided with services during the reporting period by reviewing the source document (e.g. patient dossiers or register) Copy the number of people/cases/events provided with services by the site during the reporting period (from the site monthly or quarterly or annual report). Calculate Service Point Indicator Result Verification (i.e., ratio of recounted to reported results) B) Identify possible reasons for any differences between the verified and reported results Specific notes for Review Team Record any reasons for the discrepancy (if any) observed by the Review Team. What are the reasons for the discrepancy (if any) observed by the Review Team (i.e., any data entry errors, arithmetic errors, missing source documents, other reason). Timeliness: Check the dates on the report. Were the summary reports (monthly or quarterly or annual reports) prepared and submitted on time? Additional Comments (if any) 3.4 Extract Report The next step for the review team is to extract a report from the data available at the institution using the recommended summary forms (e.g. monthly, quarterly or annual report form and tally sheets). It is recommended that the review team extracts the entire report for the period of review for the program/disease area. The institution being reviewed is expected to provide the review team with all documents and tools used for extracting reports and the institution s report for the review period. The review team is expected to extract the report and compare with the institution report. The review team should critically look at all data fields and items for accuracy, completeness, validity, consistency and timeliness. 3.5 Conduct exit briefing This manual recommends that the entire review team meet briefly with the institution s staff and management at the end of the site visit to go over any action items or outstanding documentation needs. The site visit should conclude with an exit briefing, where the review team should provide the institution with a summary of next steps and not any follow-up that may need to occur. 20

21 4 PROCEDURES FOR FEEDBACK/DEBRIEF 4.1 Feedback after the Data validation and verification The health facility must be debriefed before the team leaves the site. Discuss the key action points that the team feels should be addressed during debrief. The Team should nominate someone to take notes. The debrief should cover the following topics: Thank the Health Facility Staff for their cooperation and hard work they are doing at the health facility Review of the DQA process of what was done; Discussion of action points Inform them how they will receive the full final DQA report Make sure to highlight and praise all areas that show system strength. i.e. don t just focus on the weaknesses At the end of presentation, discuss each specific weakness that was identified. Ask the health facility staff to comment on the findings. Discuss with the staff what they think would be good ways to address the weaknesses Revise the action plan where each specific weakness is addressed in one action point At the end of debrief, provide electronic copies of debrief to the Health Facility staff as required You must write on the supervision book the strength at the health facilities and everybody in the DQA team has to sign in their names. End debrief on a positive note. Please recognize the value of the work that health facility staff are already doing every day by saving Rwandese lives, stress the fact that RDQA is just for capacity building and not fault finding mission. Finally, re-emphasize the important role that each staff member plays in generating quality data for Rwanda Government, providing quality service provision to Rwandese, and shaping effective implementation of the Health Sector Strategic Plan 4.2 Requirements for Regular Feedback Regular feedback is very important for the following reasons: Establishes strong relationship between data collectors and users at all levels Important element of management and supervision Leads to greater appreciation of data: o Improved data quality o Influencing collection of appropriate data o Expanded information use Benefits community programs and service delivery Benefits program reporting by understanding reasons behind numbers and trends 21

22 Incentive/motivation Requirements for Regular Feedback It has to be constructive and include positive and negative comments Be Selective: Should be a one page focusing on a particular desease area (indicators) Timely: 2 weeks after receiving data or monthly report Descriptive qualitative and quantitative (see examples in Appendix H). Make sure the quantitative part is appropriate to the skill level of the recipient Helpful 22

23 List of Appendices Appendix A: Final Report Template Full DQA Report Outline 1. Cover Page 2. Table of Contents 3. List of Acronyms 4. Executive Summary (one page) 5. Introduction a. Brief Outline of DQA Activities (RDQA objectives, when it took place, period reviewed, disease area assessed, team members, brief description of the tool used, documents reviewed etc) b. Assessment Limitations 6. Assessment Findings 7. Conclusions and Recommendations to Improve the data collection and reporting system 8. Annexes a. Annex 1: Action Plans for Sites b. Annex 2: List of Staff Who Participated in Assessment c. Annex 3: Schedule of visits 23

24 Appendix B: Example of Notification Letter Template for Notification Letter Date Address Dear : [Your Hospital/Health Center) will undergo a routine data quality assessment on the following dates: The purpose of this assessment is to (1) assess the ability of the data management systems of the Program/project(s) you are managing to report quality data; (2) check the quality of the results being reported; and (3) contribute to improvements data collection and reporting systems strengthening and capacity building. District M&E Team will be conducting the assessment and will contact you soon regarding the assessment. This data quality assessment relates to the [disease], [program area] and [indicator name(s)]. The assessment will: 1. Verify past reported numbers for a limited number of indicators; and 2. Hold a debriefing with your management staff on assessment findings and suggested improvements in Action Plan. To help the District M&E Team perform the initial phase of the review of your overall data management system and to limit the team s on-site presence to the extent possible, we request that you send us the documentation: Blank copies of your data collection tools (source documents) Previous monthly reports of the most recent quarter To facilitate site visits, we request that a staff member responsible for M&E or who receives, reviews and or/compiles monthly reports be present during the assessment period. Generally, the assessment will not take more than 3 hours. Thank you for submitting the requested documentation to at by. If any of the documentation is available in electronic form it can be e- mailed to Again, we emphasize that we will make every effort to limit the impact our assessment will have on your staff and ongoing activities. In that regard, it would be very helpful if you would provide us the with key contact early on in this process (your chief data management official, if possible) so we can limit our communications to the appropriate person. If you have any questions please contact at. Sincerely, Cc: 24

25 Appendix C: Correction Form MINISTRY OF HEALTH LOG BOOK FOR ERROR CORRECTION Health Center Name: Date of Correction Problem (please also state which document has the problem) Type of Correction Name of Person who has made the correction 25

26 Appendix D: Examples of Quantitative Feedback APPENDIX H: EXAMPLES OF QUANTITATIVE FEEDBACK Exhibit 1: Feedback to Community Health Workers This simple pencil-and-paper form could be applied by community-based programs or primary health clinics during regular supervision visits. It does not require significant levels of literacy or numeracy, nor does it require an understanding of how to interpret graphs. The form contained a bar representing the average monthly performance of the middle 80% of community distributors during the previous year. The supervisor marked the distributor s current performance with an X, thereby enabling them to see how it compared to other distributors in the program and stimulating discussion of how to improve performance. The form was positively received by distributors and supervisors alike and significantly increased the amount of time spent on discussing program issues[1]. [1] Foreit, J.R. & Foreit, K.G. Quarterly versus monthly supervision of CBD family planning programs: an experimental study in northeast Brazil. Studies in Family Planning, 1984, 15, Community Distribution Program Monthly Performance Review Distributor: Region: Supervisor: Month: Year: Number of NEW clients: MARK WITH AN X ON THE BAR BELOW Number of RETURNING clients: MARK WITH AN X ON THE BAR BELOW

27 Exhibit 2: Feedback to Health Facility Staff (The numbers achieved for the month can also be calculated against the targets. So the percentage column can be those against the targets) Quarterly Performance Indicators # Indicator Numerator Denominator Percentage 1 % of eligible clients placed on ART # of new clients on ART 39 ART Sum of # of new clients on ART and clients on ART waiting list % % of current ART clients % of ART clients in 6 month cohort undergoing repeat CD4 testing % of children current on ART % of non active ART patients who have stopped ART. % of non active ART patients who transferred out. % of non active ART patients who died. % of non active ART patients who have been lost to follow-up. # of active clients on ART # of cumulative clients on ART 92% # of clients for whom repeat CD4 Total # of active ART clients in 6 testing was done at 6 months month cohort Pediatric ART # of active children on ART # of cumulative children on ART 78% ART Care Follow-up # of patients who stopped ART. # of non active ART patients % # of patients who transferred out. # of non active ART patients % # of patients who died. # of non active ART patients % # of patients who have been lost # of non active ART patients. to follow-up % 16% 27

28 Exhibit 3: Feedback to District M&E Officers/Data Managers 28

29 Name of Site Reviewed Program/Disease area reviewed Reporting period reviewed Date of review RWANDA DATA VALIDATION AND VERIFICATION TOOL Names of Review Team 1. DESCRIPTION OF THE RECORDING PRACTICES IN RELATION TO SERVICE DELIVERY - Describe the connection between the provision of Services and the completion of the source document Specific notes for Review Team It is recommended that the Review Team ask staff to describe the process through which the source documents are filled in relation to the provision of services. Determine the source document used for recording of services provided to clients. Source documents are the first place services provided are recorded. It may be patient dossier/patient card or clinical review forms or register. If access to patient dossier/patient card or clinical review forms is not possible, an alternative source document may be the register Describe the source document for recording the provision of services (is it a standardized form following Ministry of Health guidelines or a tailored form? If tailored, specify the source of the form, e.g. a project). Obtain a blank copy, if possible. Does the site have sufficient supplies of blank source documents (prompt for experience of stock-outs of source documents)? Describe when recording of the provision of services takes place, on what form(s) and by which staff member(s). Are there indications that there are delays between delivery of the provision of services and recording of the provision of services on the source document? If the provision of services and recording of the provision of services are not done at the same time, please describe how the disconnect might affect data quality. 1.6 Does the site have Standard Operating Procedures (SOPs) for health information management? 1.7 Have site staff been trained in the use of Standard Operating Procedures (SOPs) for health information management? Additional Comments (if any) 2. DOCUMENTATION REVIEW - Review availability and completeness of all source documents for the selected reporting period Specific notes for Review Team Due to confidentiality regulations, it may be important that prior permission to review the source documents be obtained from the site's senior manager. Furthermore, the assessment team should ask the site manager if he/she would prefer that another staff member be present while the source documents are being reviewed. A) Check Availability and Completeness of Documentation Review available source documents for the reporting period. Is there any indication that source documents are missing? If yes, determine how this might have affected reported numbers. Are all available source documents complete, notably the mandatory data fields have been filled with data (e.g. date; serial number, name; address; sex; age, catchment area and important information regarding services provided/received by the patient) Review the dates of provision of services on the source documents. Do all dates fall within the reporting period? B) Check Accuracy and Validity of data Specific notes for the Review Team: If feasible, select 5% of patient records (or at least 20 patient records) of patients who were provided/received services during the review period 2.4 How many patient records were selected? 2.5 Check each record to verify whether the data elements are: accurately record, valid, within acceptable range, proper terms or abbreviations or codes used, no missing data. How many patients records have accurate or valid data elements? Calculate % difference for accuracy If difference is below 90%, select an additional 5% of patient records (or at least an extra 10 records) and redo the calculation (ADD the numbers to the existing numbers in the above cells); repeat up to three times. Additional Comments (if any) - 29