DATA QUALITY ASSESSMENT FOR NATIONAL NTD PROGRAMMES

Transcription

1 DATA QUALITY ASSESSMENT FOR NATIONAL NTD PROGRAMMES DR. MBABAZI, Pamela Sabina Medical Epidemiologist, WHO/HTM/NTD/PCT

2 Content 1. Background 2. Definition and Objectives of DQA 3. Process of development of DQA tool 4. Structure 5. Field testing for NTD programmes 6. Outcomes 7. Recommendations

3 Background* Many national programmes have been implemented for more than > 5 years, with little/no action at country level to validate reported data Integration has created new ways of working, and new challenges/concerns for data management Coverage surveys (routine & periodic) often not implemented at recommended regularity NTD roadmaps milestones 2015, 2020 measurements require robust, reliable and credible national programme data: WG M&E: reports from Regions present subjective information on data quality issues * Preventive chemotherapy focus

4 Definition and Objective Data quality assessment (DQA) is a process that aims to expose technical and implementation data issues in order to PLAN and DEPLOY data enrichment strategies for national NTD programmes. Two specific objectives: Quantitative: to determine the quality, accuracy, accessibility, timeliness and completeness of implementation data Qualitative: to determine the quality of the NTD information system

5 Process of drafting DQA for NTD programmes 1) Reviewed DQAs from EPI HIV/AIDS Tuberculosis MEASURE Global Fund PEPFAR 2) Compared statistical methods (Consultant): DQA vs. LQAs

6 Overall structure 1. Assesses the quality of reported NTD data for a given period ACCURACY RELIABILITY PRECISION COMPLETENESS TIMELINESS INTEGRITY CONFIDENTIALITY Valid, correct, minimal errors, measures what it is intended to Collected through consistent procedures and protocols Has sufficient and appropriate detail Inclusive, without omissions Reported by an established deadline Free from deliberate bias or manipulation throughout system Managed according to protection and use standards Recommended: 12 sites 2. Assesses the ability of the NTD data management systems to collect and report quality data

7 DQA protocol for NTDs KEY STEPS Assess NTD data management systems M&E unit Trace and verify results from intermediate aggregation site reports at National M&E unit Assess data aggregation and reporting systems at intermediate aggregation levels Trace and verify results from intermediate site reports at intermediate aggregation levels Assess data collection and reporting systems at Service Delivery Points (SDPs) Trace and verify results from source documents at SDPs reports.

8 DQA related tools

9 DQA for NTDs - Uganda

10 Field-testing: AFRO/Uganda 1. Proof of principle and pilot Country permission & meetings: at start, and debriefing Team composition: MoH, RTI, WHO-HQ Conducted in 4 randomly selected districts, 4 subcounties (1 per district), and 16 villages that served as service delivery points (included schools & communities) Cost $12,000 (provided by RTI) DQA experience very insightful, revealing, constructive Proved that DQA tool works well, minor improvements needed Poster presentation at ASTMH (on display)

11 DQA for NTDs - Nicaragua

12 Field-testing: AMRO/Nicaragua Country permission & meetings: at start, and debriefing Team composition: MoH, PAHO-NID, PAHO-EPI Regional office and PAHO-IST, WHO-HQ Conducted in 1 Department (Grenada) conveniently selected, 1 health facility (Diriomo), and 1 school DQA methodology is familiar to EPI health workers, yet was still revealing, constructive & insightful when applied to deworming data Proved that DQA tool works well, but will require adjustment 3-step adjustment in order to enable integration

13 Outcomes System quality index (QI) Uganda Nicaragua Data Management Assessment - Overall Average M&E Structure, Functions and Capabilities 3 Data Management Assessment - Overall Average M&E Structure, Functions and Capabilities 3 Links with National Reporting System Indicator Definitions and Reporting Guidelines Links with National Reporting System Indicator Definitions and Reporting Guidelines Data Management Processes Data-collection and Reporting Forms / Tools Data Management Processes Data-collection and Reporting Forms / Tools Radar graphs

14 Data accuracy: Verification factor Uganda Nicaragua Data Verifications - Overall Average by Indicator Data Verifications - Overall Average by Indicator 180% 110% 160% 140% 120% 100% 80% 60% 40% 100% 158% 100% 78% 63% 100% 90% 80% 70% 60% 50% 40% 30% 100% 99% 99% 101% 102% 20% 0% Number treated w ith IVM + ALB Number treated w ith PZQ Number treated w ith ZITH tabs Total Population Number of ZITH tablets used 20% 10% 0% Albendazol 2-4 yrs Albendazol 5-12 yrs MMR1 1 yr Vitamin A 1-<5 yrs Albendazol tablets

15 Global Dashboard Data Management Assessment - Overall Average 3.50 System Assessment Results by Level of the Reporting System Links with National Reporting System Data Management Processes M&E Structure, Functions and Capabilities Indicator Definitions and Reporting Guidelines Data-collection and Reporting Forms / Tools Service Site Average Sub-County Level Average District Level Average M&E Structure, Functions and Capabilities Indicator Definitions and Reporting Guidelines Data-collection and Reporting Forms / Tools Data Management Processes Links with National Reporting System 180% 160% 140% Data Verifications - Overall Average by Indicator 158% Data Verifications - Verification Factors by Level of the Reporting System % 100% 80% 60% 40% 100% 100% 78% 63% % 0% Number treated with IVM + ALB Number treated with PZQ Number treated Total Population Number of ZITH with ZITH tabs tablets used 0.0 Service Site Average Sub-County Level Average District Level Average Overall Average Number treated with IVM + ALB Number treated with PZQ Number treated with ZITH tabs Total Population Number of ZITH tablets used Reporting Performance - Overall Average Reporting Performance by Reporting Level 120% Completeness 50% 100% 80% 100% 100% 75% 75% 60% Timeliness 0% 40% 50% 20% 33% Availability 78% 0% 0% 0% 0% Sub-County Level Average District Level Average MoH NTD Program M&E Unit 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Availability Timeliness Completeness

16 Recommendations 1. DQA is very valuable M&E tool that should be used elucidate national NTD information system strengths, and determine country-specific data quality issues to be addressed. 2. Encourage and support countries to implement DQAs in order to ensure good robust quality data Apply current protocol in several more countries obtain feedback to improve & finalize tool before October Provide necessary resources: budget, training (in-service & selflearning), technical support etc note: it is a reporting requirement for some programmes with targeted donations (e.g GAVI)

17 Recommendation 3. DQA for NTDs should be integrated with assessments for other public health programmes implementing large-scale interventions, where feasible. 4. Extend DQA to intensified disease management/case management diseases a PAHO request 5. DQA should be conducted as a an integral part of programme evaluations be a pre-requisite to nationwide coverage evaluation surveys. Involve external evaluators

18 Recommendation Technical Implement DQA as a three part sequential process 1. Concordance* monitoring and Rapid Coverage Monitoring 2. Data Quality selfassessment (DQS) 3. Data Quality assessment (DQA) Timing Key features Remarks Within 1 week of PC At least once a year as part of support supervision Once every 3 5 years as part of programme and coverage evaluation. Data verification only (Rapid Coverage Monitoring of 20 conveniently selected households only) Data verification & systems assessment Convenient sampling District level personnel only Data verification & systems assessment Probabilistic sampling Includes national level personnel, and external assessors Findings used to ensure follow-up treatment of missed individuals & revise reports Findings CANNOT be used to modify reports already submitted Findings CANNOT be used to modify reports already submitted * Need to compare with CDTI community self-monitoring process

19

20 Acknowledgement Collaboration and support of RTI/ENVISION (helped accelerate activities) Ministries of Health for Nicaragua, Uganda National NTD programme staff at sub-national levels WHO Country Offices DPCs & NTD focal points WHO Regional offices: AFRO, AMRO

21 THANK YOU FOR YOUR ATTENTION