DATA QUALITY ASSESSMENT TOOL INSTRUCTIONS FOR DATA PREPARATION 1
The data quality assessment (DQA) tool provides a stepwise method to assess the quality of health facility data for some key coverage indicators. Users are asked to prepare data from their routine facility reporting, also called the health management information system (HMIS) in a specific format. Each of the worksheets in the tool and their data requirements will be described in detail below. This DQA tool requires users to enter or paste the data and it will result in a Data Quality Report Card being created. There are a total of 19 worksheets in this tool, including the cover page, the report card and graph worksheets. The user has to input (enter/paste) information in 12 of these worksheets. The following instructions will describe each of the worksheets and the data that needs to be entered in the individual worksheets. The main indicators selected for this tool and for the Report Card are ANC1, deliveries, DTP1, DTP3 and OPD. Please note: Data in your HMIS database are kept in different tables. Sometimes your administrative units are ordered differently in the different tables. To make sure you are copying and pasting the correct information for each administrative unit, please ensure that the data is for the appropriate administrative unit. General information about the tool 1. In the data entry sheets which are tabbed blue, the following information applies: a. Cells in yellow are total cells and are write-protected b. Only cells in green allow data entry c. Cells in white are also write-protected as they are linked to other cells in the same or different worksheets d. When you copy and paste data, please use the paste special function and only paste "Values" e. The icon indicates a cell with a drop-down list 2. The red tabbed are write-protected. The user will not be able to change any information on these sheets. When you open the tool When you open the tool some dialog boxes might open. The type of dialog boxes are shown below. Please follow instructions as given when you see a dialog box. If you get the warning as shown above, please click on Enable content to proceed. 2
If the dialog box as shown above open, please click on Don t Update button. The preceding dialog box explains what the tool does. Please read the information and click on the Ok button. This box might appear multiple times. Please click the Ok button until this box disappears. Worksheet # 1: Title The first sheet is the Title page. It has the version number and the date of last update made to the tool. 3
Worksheet # 2: Input_basic_info There are 7 items/questions that need to be filled in this sheet. Instructions are given below for each of the items/questions separately. 1. Please use the drop-down list and select your country 2. This tool can analyse data quality for one year at time. Please select the year for which you are analysing data quality. If your reporting year stars at some other month besides January, e.g. June, please select the year in which the last month of reporting falls. If you want to look at data quality for another year, you have to open a copy of the blank file and input data for the new year. 3. Please use the drop-down list to select the reporting levels in your country. The examples, below a couple of different scenarios that might apply to your situation. 4
Example 1: The reporting levels in your country are Facilities District Province National 1. The first level is the facility and cannot be changed 2. In the 1st green drop-down box, please select District 3. In the 2 nd green drop-down box, please select Province 4. Since provinces report to the national level, which is the highest reporting level, please leave the last green drop-down box blank Example 2: With many countries instituting web-based HMIS, facilities are now directly reporting to the national level. In this case, please input the level that has administrative responsibilities for facilities. For example, if a province is responsible for all the facilities in within its border, then the reporting level would be Facilities Province National. 1. The first level is the facility and cannot be changed 2. In the 1st green drop-down box, please select Province 3. Since provinces report to the national level, which is the highest reporting level, please leave the last two green drop-down boxes blank 4. This question requires the administrative level for which data quality analysis is being conducted. Please ensure that the administrative level at which you are analysing data is one of the reporting levels specified in item/question 3. If the administrative level of analysis is a district, please select District from the dropdown list. The one main criterion for selecting the reporting level of analysis is that it is essential to have population estimates for the level of analysis. Example 1: The reporting levels in your country are Facilities District Province National. However, your facility catchment area population estimates are not very robust. You believe your district level estimates are better, as these are projections from the census/statistics bureau. 1. You will select District from the green drop-down box Example 2: In the example where there is a web-based HMIS system and facilities are directly reporting to the national level, examine the reporting levels and determine the levels for which you have population estimates. If the administrative levels in your country are Facility District Province National. However, you do not have population estimates for the facility or district level. You do have population estimates for the provincial level from census projections. 1. You will select Province from the green drop-down box. 5
5. In this green drop-down box, please select the level for which are inputting service output data. Service output data, here, means data ANC1, deliveries, DTP1, DTP3 and OPD. You can input service output data starting from the facility level (based on what information you have available to you). Example 1: In your country you have a web-based system where facilities directly report to the national level. You have facility level data and you would like to analyze with facility level data. 1. You will select Facility from the green drop-down box Example 2: The reporting levels in your country is Facility District Province National. However, as facilities submit paper forms to the district, and the districts submit only aggregate data to the province, the starting reporting level that you have information for is the district. 1. You will select District form the green drop-down box 6. Due to large sample sizes and costs, surveys often only provide population estimates at either the national, state/province/regional level. Please provide the aggregation level for the most recent population-based survey. If you have multiple surveys conducted within a short time frame from each other, please select the aggregation level used in the survey you are most interested in. 6
7. Please type in the names of the months starting with the first month of the reporting year. For example, if the reporting year starts in July and goes till June of the following year, the first row will have July and the last row will have June. 7
Worksheet # 3: Input_admin_units In this worksheet, the user will be entering details on the names of the administrative units for the level which they are inputting facility data and the corresponding reporting levels. Two examples are presented below. Col. 1 title No is the same in all worksheets. This column is used for numbering the administrative units that will appear in Column 2. When there are names of 100 administrative units in Col 2, please type 1-100 in Column 1. Please DO NOT type any additional numbers beyond the total number of administrative units that you are analysing. Columns 2, 3, 4, and 5 are filled based on selections that were made for the different items/questions in Input_basic_info worksheet. Example 1: The country reporting levels from item/question 3 in Input_basic_info _worksheet is Facility District Province (since province reports to the national level, the national levl is not entered in the drop-down box). Column 2 says District and is the administrative level for which you are inputting data. Even though facilities are the first administrative level, the user selected the level of data entry at the district level (answer to item/question 5 in the Input_basic_info worksheet). Hence, the administrative unit level that appears in Column 2 is District. Column 3 says Province and is the level to which Column 2 reports to (if it exists). Explanation of example above: 1. Enter 1-n for the total number of administrative units appearing in Column 2. In this example, 1-12 has been entered for the total number of districts in Column 2. 2. Enter names of all the administrative units in the administrative level for which you are inputting data. In example 1, please paste or type in the name of all the districts in your country. 8
3. Enter the names of the administrative units to which the administrative units in Column 2 report to. If there is no reporting level for Column 2, Column 3 will not have an administrative unit level title. In this example, Districts report to Provinces. Columns 4 and 5 are not populated as Province is the last reporting level before the national level. Example 2: The country reporting levels from item/question 3 in Input_basic_info _worksheet is Facility District Province. Your country has a web-based HMIS where facilities report directly to the national level. However, administratively, your facilities are managed by districts which in-turn report to provinces. These columns were populated based on the following selections made in the Input_basic_info worksheet. 1. Column 1: Since data is being inputted at the facility level (as shown by Column 2) and there are 30 facilities, please number 1-20 in Column 1. 2. Column 2: Enter the names of all the facilities in your country. 3. Column 3: Enter the names of the districts to which the facilities report to. 4. Column 4: please enter the names of provinces to which the districts report to. Column 5 is not populated as Province is the last reporting level before the national level. Please use "Paste special" and paste ONLY VALUES when you are copying and pasting information. 9
Worksheet # 4: Survey_mapping This worksheet will map the administrative unit level that is being analysed to aggregation level that is used in the most recent population-based surveys. For example, in a country the reporting levels for facility data is Facilities Districts Provinces National. However, it is rare for a survey to provide district level estimates for key indicators, especially in a country where there are many districts. Typically, survey estimates are for higher aggregation levels, such as province level. There are two examples given below that will explain a couple of different scenarios. Please note: Data should only be entered/pasted in the green colored cells. The cells in white and purple/mauve are column headings and are write-protected. User will not be able to enter data in the white or purple cells. Example 1 The country reporting levels from item/question 3 in Input_basic_info _worksheet is Facility District Province. However, the administrative unit of analysis (as selected in item/question 4) is District. 1. Column 1 title, District, is automatically populated by the response choice in item/question 4 in Input_basic_info worksheet. 2. The administrative unit names in Column 1, in this case the district names, are automatically populated as well from the information entered in Input_admin_units worksheet. [Columns that are white are write-protected]. 3. Column 2 title, Province, is automatically populated based on the selection made on item/question 6 in Input_basic_info worksheet. In the example above, the user 10
Example 2: needs to map the districts to the provinces as seen above. Districts A, B and C are in Provine M, Districts D and E are in Province O and so on. Sometimes the survey aggregation levels are not similar to existing administrative levels. Example 2 below gives an illustration of this situation. In example 2, the survey aggregation level selected in item/question 6 in Input_basic_info worksheet was Region. The country s administrative levels are Facilities Districts Provinces National. Region is not an adminsitrative level. However, the survey aggregation was done at a regional level. In this case, you would map the main unit of analysis (the District was selected in item/question 4 in Input_basic_info _worksheet) as best as possible to the Regions that were selected for the survey. In the figure below, Districts (the main unit of analysis) A, B, C, D, E, F, G, H, I have been mapped to regions. In example 2, the survey aggregation level selected in item/question 6 in Input_basic_info worksheet was Region. The country s administrative levels are Facilities Districts Provinces National. Region is not an adminsitrative level. 1. Column 1 title, District, is automatically populated by the response choice in item/question 4 in Input_basic_info worksheet. 2. The administrative unit names in Column 1, in this case the district names, are automatically populated as well from the informatin entered in Input_admin_units worksheet. [Columns that are white are write-protected]. 3. Column 2 title, Region, is automatically populated based on the selection made on item/question 6 in Input_basic_info worksheet. In this case, you would map the main unit of analysis, Districts to Regions that were selected for the survey. In the figure below, Districts (the main unit of analysis) A, B, C, D, E, F, G, H, I have been mapped to Regions 1 and 2. 11
Worksheets # 5 9: Input_ANC1, Input_Deliveries, Input_DTP1, Input_DTP3 and Input_OPD Tables 3-7 are found in the worksheets Input_ANC1, Input_Deliveries, Input_DTP1, Input_DTP3 and Input_OPD. Please enter the monthly service outputs for ANC1, deliveries, DTP1, DTP3 and OPD in the appropriate worksheet. Only 1 example table (for ANC1) is shown below as the table format is exactly the same in the above worksheets. Please note: Data should only be entered/pasted in the green colored cells. The cells in light yellow are protected and contain totals. The cells in white and purple/mauve are column headings and are also write-protected. The cells in white are administrative unit numbers and names. User will not be able to enter data in the white, purple or yellow cells. 1. Columns 1 5 are automatically populated based on information entered in the Input_admin_units 2. In Columns 6 17 please fill in the service outputs for ANC1, deliveries, DTP1, DTP3 and OPD in the respective worksheets. 3. Each cell in Column 18 is a sum of the service outputs entered in columns 6-17 for a specific row. 4. Column 19 counts if there are any missing values in the data entered. If the user only enters data for 6 months, the missing values will only be counted amongst the months for which data has been entered. For example, if the user has entered data for Jan-June and the remaining 6 columns are blank, the formula in this column will only search for missing values in the months Jan-June. 5. Column 20 counts all the zero values in the entered data. 12
Example for Worksheets # 5 9 Example for Worksheet # 10 13
Worksheet # 10: Input_trend In this worksheet, Table 7, please input (enter/paste) annual service outputs for ANC1, Deliveries, DTP3 and OPD for up to three years preceding the year of analysis for each admin unit. If you do not have data for all three years please enter the data for the years you have complete information. Please note: Data should only be entered/pasted in the green colored cells. The cells in light yellow are protected and contain totals. The cells in white and purple/mauve are column headings and are also write-protected. The cells in white are administrative unit numbers and names. User will not be able to enter data in the white, purple or yellow cells. 1. Columns 1 to 5 are automatically filled based on the information filled in the Input_basic_info and Input_admin_units worksheets. 2. ANC1, Deliveries, DTP3 and OPD each have four columns. The user can input historical service output data for up to three years. The last column for each indicator is the total service output for the data analysis year. This column in writeprotected and the totals are carried over from the information inputted in Input_ANC1, Input_Deliveries, Input_DTP3 and Input_OPD. 3. The user does not have to input data for all three past years. They can input data for available years. 4. In the example above, the data for one of the years for deliveries is missing. This is allowed. 5. The historical indicator data has to be for consecutive years. This information is automatically determined by the tool. For example, if the year of a data analysis is 2011, the required historical indicator information would be for 2008, 2009 and 2010. 14
Worksheet # 11: Input_population In this worksheet, Table 1, you will be entering information about the administrative unit of analysis such as their names, denominators for the indicators analysed in this tool and reporting completeness information by the administrative level that is being analysed. Please note: Data should only be entered/pasted in the green colored cells. The cells in light yellow are protected and contain totals. The cells in white and purple/mauve are column headings and are also write-protected. The cells in white are administrative unit numbers and names. User will not be able to enter data in the white, purple or yellow cells.. The following information is required for each of the Columns specified. 1. Col 1 is the number for the administrative unit of analysis. The numbering will be in ascending order from 1 to the total number of administrative units in your country that you are analysing in this exercise (based on answer to question 4 in Input_basic_info ). If your unit of analysis is district (from question 4 in ( Input_basic_info worksheet) and you have hundred districts in your country, then Col 1 will be numbered 1 to 100. 2. Col 2 lists the names of the administrative units in the administrative level that is being analysed. The administrative unit names are filled based on the answer to question 4 in Input_basic_info (on the administrative level of analysis) and names of administrative units that were entered in Input_admin_units worksheets. 3. Col 3 requires the user to enter the population (or population projection) for each administrative unit for the year of analysis. 15
4. Col 4 to Col 7 Please enter the total number of expected pregnancies, expected births, surviving infants and total population for each administrative unit. These totals are denominator information that will be used to calculate rates. Expected pregnancies will be used to calculate denominators for ANC1, expected births for deliveries, surviving infants for DTP3, and total population for outpatient visits (OPD). Please input the denominator information that is normally used to calculate coverage rates for these indicators. Worksheet # 12: Input_reports_received The information entered in this worksheet will be used to calculate reporting completeness. 1. Column 1 is the number for the administrative unit of analysis. The numbering will be pre-filled based on the total numbers of administrative units in the administrative level selected for anlaysis. For example, Column 1 has been pre-filled 1-12 as there are 12 districts in the country. District was the selected administrative level of analysis (from question 4 in Input_basic_info worksheet. 2. Column 2 lists the names of the administrative units in the administrative level that is being analysed and is pre-filled based on the information in Input_admin_units worksheet. 3. In Column 3 please fill in the total number of reports received from the administrative level selected for the data quality analysis. For example, if District A is supposed to submit a monthly report to the province and they submitted 12 16
reports in 2011, you should enter 12 in the cell. It is possible, especially in countries that have web-based reporting at the facility level, there might not be an reports received from an intermediate level. 4. In Column 4 please enter the total number of health facilities providing services in each administrative unit for the year of analysis (please enter the total number that is used to normally calculate completeness rate). 5. In Column 5 please enter the total number of reports received from health facilities in the administrative unit for the year of analysis. District A has 110 reporting facilities and a total of 1209 reports received in 2011. If all expected facilities are supposed to report every month, the total number of expected reports for Admin unit 1 would have been 204 reports. However, only 190 reports were received. This indicates that not all facilities submitted a report every month. At the top right corner of the worksheet, there are two other items that need to be filled. The first item asks for the reporting cycle for facilities. Please select the answer from the drop-down list. The second item asks for the reporting cycle at the administrative unit level. Please select from the drop-down list. 17
Worksheet # 13: External_data_sources In Table 11 below, the user will be entering coverage estimates from population-based surveys to compare with facility-based coverage rates. 2 1 1. In the row referred to as (1) please select your data source from the drop-down list. 2. In row referred to as (2) please select the year of the estimate from the drop-down list. 3. In Columns 3a and 3b, please enter the coverage estimates and the standard errors for ANC1. In row that has Overall estimates, for the aforementioned columns, please enter the overall (most often national) coverage estimate and standard error for ANC1. In subsequent rows please enter the survey coverage estimates for the administrative level or aggregation level specified in Survey_mapping worksheet. 4. Please do the same for Deliveries (Columns 4a and 4b), DTP1 (Columns 5a and 5b) and DTP3 (Columns 6a and 6b). 5. The columns for the survey estimates and the SE are all in "%" format. Please do not change this format. For example, if your survey report says the ANC estimate is 0.89, please enter 89 in the column and not 0.89. MICS and DHS reports have standard errors (SE) for a number of indicators in the annex of the survey report. Please see you can find the SE for the indicators listed below. If you find that the SE for DTP3 is 0.04, please enter 4 and not 0.04. If you do not find SE for the indicators listed 18
below, please enter 0 (zero) in the SE column. Please DO NOT leave the SE column blank. 6. Above Table 11, there is an item that says Total # of live births for the year of analysis and a white cell with a number. This number of live births is pre-filled from the World Prospects database (2010 version) from the United Nations Population Division. This number is pre-filled as soon as you select your country of analysis in the Input_basic_info worksheet. This number will be used to compare to the denominators for live births used for calculating some of the indicators. If you are doing sub-national analysis, the information in this cell will not be applicable to you. Worksheet # 14: Report_card In this worksheet, the Data Quality Report Card (DQRC) indicators will be calculated for the user based on the data that has been inputted. The DQRC examines and presents a data quality score for indicators spanning across four dimensions of data quality. These dimensions are: 1) completeness of reporting; 2) internal consistency of reported data; 3) external consistency of population data; 4) external consistency of coverage rates. The indicators are described in detail in the document titled Guide to the Health Facility Data Quality Report Card, which can be found at: (add link) 1 19
The Report Card worksheet provides results for the indicator at the national level and sub-national level. At the sub-national level, it also identifies administrative units that are performing poorly. For example, please see (1) in the figure above. The right most column says District C. Calculations are automatically done in the tool and sub-national units with less than 80% facility reporting rates are listed. In this example, District C has facility reporting rate of less than 80%. Worksheets # 15-19: Graphs Worksheets 15-19 provide graphs for some of the indicators. 1. Worksheet 15 Graphs_indicator_2b presents graphs for indicator 2b which looks at the consistency of the indicators over time. 2. Worksheet 16 Graphs_indicator_2c shows the consistency between ANC1 and DTP1. 3. Worksheet 17 Graphs_indicator_2d shows the consistency between DTP3 and DTP1. 4. Worksheet 18 Graphs_indicator_3b1_3b2 compares different population denominators 5. Worksheet 18 Graphs_indicator_4a-4c compare coverage estimates from facility data with coverage estimates from surveys for ANC1, deliveries and DTP3. 20