Terms of Reference for an External Data Quality Audit

Transcription

1 Terms of Reference for an External Data Quality Audit Introduction/Background: Save the Children (SC) is a leading independent child rights organization working for children's rights and delivers immediate and lasting improvement to children's lives worldwide. The organization is working on 5 core themes- Child Survival, Education, Child Protection, Humanitarian and Disaster and Risk Management in collaboration with the Government, civil society organizations, and the community across 14 states and has reached nearly 4 million children across India. In the last few years Save the Children has grown rapidly. In order to ensure that during this phase of rapid growth, the organisation continues to monitor its achievements; Save the Children has started putting in place standard systems for tracking key program parameters, including the number of children being reached by the organisation. Program data is being captured through SCINDIA INFO for key standard monitoring indicators identified for all thematic areas twice a year. Additionally, a regular MIS is also being maintained at project level which reports on in-depth program-specific indicators. Save the Children is committed to reach a 5 million number of children as per their strategy The number of children being reached through the efforts of the organisation are being collated and compiled on a quarterly basis through a standardized tool. At this stage of growth, Save the Children is undertaking a process of strengthening its monitoring systems. As part of these efforts, the organisation plans to integrate its various monitoring systems into a single standard system. This system will also have effective data quality mechanisms built in. In order to ensure that Save the Children designs appropriate data quality mechanisms, we need to examine the current systems in place for data quality as well as the actual quality of the monitoring data being currently collected through the project MIS, SC India Info as well as the reach data. As Save the Children implements most of its programs through a NGO partners, the role of these partners in ensuring data quality is critical and their role in maintaining data quality also needs to be examined. Rationale: In view of the above an external data quality audit is planned. This audit would examine the effectiveness of the current data quality systems in improving program effectiveness by ensuring the authenticity, reliability and validity of data being used for decision making and will help to identify areas in which data quality needs to be strengthened further. As this process involves beneficiaries and checking of registers/forms/ progress reports, the audit is critical and would provide important feedback for rectification and improvement of quality of reporting and recording. Overall this would help us to gain greater confidence on the system and increase beneficiaries, partners and donor satisfaction. Objectives: To review the existing data quality mechanisms and their effectiveness which are put in place at national, state and partner level during data collection, recording, analysis, usage, data storage, data updation and reporting of monitoring data viz. SC India Info, Project MIS and reach data; To assess the quality of routine data recorded and reported in light of authenticity, validity, reliability, system integrity and accuracy for different data collection mechanisms viz. REACH, SCINFO, PROJECT MIS; To triangulate the data reported through various monitoring mechanisms; To understand the methods used to collect, process, analyze and report the data (e.g. appropriateness of methods, possibilities of human error and delay in data collection and processing, application of technology at various levels, etc.); To assess the capacity of people who are responsible for collecting and processing data or for ensuring data quality at all levels;

2 To provide recommendations to Save the Children and its partner organizations for improvement and maintenance of data quality. The five steps that need to be followed for Data Quality audit are as follows: Reliability 1. Record verification at all levels (field level, NGO level, Save the Children level {state and national} for identified indicators. Validity 2. Authenticity of beneficiaries for identified indicators. 3. Timeliness: Are data collected and reported as regularly as relevant? Is reported data the most recent? Is the date of data collection/reporting clearly identified? 4. Feedback: Once the above two components are done; the audit team has to come back to the respective level and provide appropriate feedback to audited level in a prescribed format developed in consultation with Save the Children. 5. System Assessment: Capabilities of people who are responsible for data management and reporting, data management processes. Methodology: Study Area: Save the Children is currently implementing their programs in 14 states. For selecting the geographical area, it was suggested to have those states which have more projects or projects which are crucial for the organisation. Based on these criteria, West Bengal (17), Uttrakhand, Delhi (16) and Rajasthan (7) were found to be major states, where this study can be conducted to review the system and data quality. Sampling Frame and Study Tool: The bidders are encouraged to suggest methodologies and tools along with appropriate justification in their technical proposal. Data Collection Units: National Office State Office Save the Children Partner Level Field Level Partner Organisation Beneficiary Level Tools for the study: The selected organisation will develop the requisite tools for assessing the validity, reliability, system integrity, accuracy, information flow and feedback mechanism, utilization of data on a dayto-day basis and system assessment. Other than this, they will cover validity of data which will be used to interview the beneficiaries. The selected agency needs to develop tools that would help address all the objectives of the study. Study will have quantitative as well as qualitative approach. Processes/Activities The agency selected for conducting the study should follow the processes mentioned below: Discussion on the Terms of Reference and Methodology Drafting and pre-testing of data collection tools

3 Collection of data (includes assessment of data, interview of beneficiaries, interview of NGO partners, Discussion with local level groups, discussion with PCs and KM C and state manager etc.) Submit two pager note on update regularly to SAVE National team Data entry, cleaning and analysis plan (in consultation with SAVE) Submission of the draft and final reports Dissemination of findings at state level and national level Submission of the final reports Terms and Conditions: The consulting organization must ensure the following steps: The auditor should be fully proficient in the local language; The auditor should at least be a Post Graduate; The auditor should be fully equipped with quantitative database structures and logical reasoning and good in qualitative techniques; The auditor should have work experience of at least for two years in auditing the information in child rights area and should be well aware of the concerns related to sensitivity and confidentiality of beneficiary data; Save the Children reserves the right to terminate the contract in case the above terms are not fulfilled by the consulting organization. Deliverables: Comprehensive report on the study after in-depth analysis of collected data; Specific suggestions which would help Save the Children to strengthen the data quality aspects in each selected project and overall; Well-defined framework (including quality) describing how the current system is and how it could be in future; Analysis of the skills of staff, factors hampering MIS operations, etc.; Dissemination of findings. Timelines: Two days orientation on tools and other procedures Complete data collection by December 31 st, 2013 Presentation of findings to Save by January 20 th, 2013 Submission of draft report by January 31 st, 2013 Submission of final report by February 10 th, 2013 Proposal submission procedures and timelines Interested organizations are required to submit Technical and Financial proposals electronically or in two sealed envelopes separately latest by 20 th November, The technical proposal should consist the following: A letter of interest for undertaking this study Profile of the organization demonstrating their ability to undertake this work Past experience in similar studies conducted in the area of Data Quality Audit

4 Profiles of key research personnel to be involved in the study Detailed operational plan and time plan to execute the study The financial Proposal should consist of the detailed budget with all the necessary assumptions for executing this assignment. Save the Children reserves its right to accept or reject any proposal without assigning any reasons thereof whatsoever The Technical and Financial proposals should reach Save the Children before to: Save the Children-Bal Raksha Bharat, 3rd Floor, Vardhaman Trade Centre, Nehru Place, New Delhi Contact persons at Save the Children Neeraj Mishra, PhD Manager MIS Voice#

5 Table 1: State Delhi Rajasthan West Bengal Number of projects 9 Number of partner Number of district covered under these projects Name of districts which need to visit for EDQA Thematic area Annex: Number of projects need to review for data quality 1- East 4- West 2 Child Protection North 5- North West 6 3- North East 6- South Health & Nutrition Alwar 4- Churu Education Banswara 5- Dungarpur Child Protection Bharatpur 6- Tonk 1 2 Health & Nutrition 4- Chittorgarh parganas North 6- Kolkata Child Protection parganas South 7-Maldah Education Jalpaiguri Health & Nutrition 2 2 Disaster Risk Reduction 1 Uttrakhand Uttarkashi 4- Dehradun 2- Chamoli 5- Tehri Garhwal 3- RUDRAPRAYAG Disaster Risk Reduction 1 (Number of grants are there which combined in one project) Table 2: Thematic Area Education Health & Nutrition Humanitarian Indicator (These indicators are not final however there could be addition or deletion of indicators in future.) Number of Teachers who have received training session from Save The Children Number of functional SMC/VEC/ VKS/ WEC trained by Save Meeting minutes available for SMC Number of children initiated breast feeding within an hour of birth Number of newly born babies who were seen by a health worker within 48 hours of delivery (for those delivered at home) Number of Frontline workers trained (ANM,ASHA,ANM) Number of VHNSC (Village Health, Nutrition and Sanitation Committee)(ward committees in urban area) members trained under Save the Children intervention. Community support / mothers groups under Save the Children intervention Number of Pregnant and lactating women provided with supplementary nutrition by SC intervention Number of Child friendly Spaces set up and children accessed

6 Disaster Risk Reduction Child Protection Number of health camps organized and people benefitted Number of education kit distributed Number of children and adult trained in risk & hazard mapping and school safety Number of task force group set up and functional Number of people trained in task force group Number of CPC and CG functional Number of children removed from labour Number of families linked with social security scheme or given shelter Number of children facilitated for birth certificate; Legal; Counseling and Medical Table 3: In Uttrakhand, SAVE is working in following districts: 1. Tehri Garhwal; 2. Dehradun; 3. Rudraprayag; 4. Chamoli; 5. Uttarkashi Number of beneficiaries reached so Children All beneficiaries (including children) far (Humanitarian Indicators) 6,542 15,575 Note: Indicator wise and project wise number of beneficiaries details are given in separate XLS file, which can be used for beneficiaries selection.