The Data Improvement Guide. A curriculum for improving the quality of data in PMTCT

The Data Improvement Guide A curriculum for improving the quality of data in PMTCT

TABLE OF CONTENTS Introduction... 3 Module #1: Introduction, forming a Data Improvement Team, and developing an Aim Statement... 6 Module #2: Understanding the system creating a data Process Map... 8 Module #3: Random data auditing... 10 Module #4: Data Validation, Indicator Calculation, and Monthly Summary Submission... 11 Module #5: Developing Process Measures and Run Charts... 13 Appendix 1: Aim statement worksheet... 18 Appendix 2: Run Chart Tool... 19 2

Introduction Purpose of the Data Improvement Guide Data is very important for creating a high quality PMTCT programme; however, much of the information that is collected from health care facilities is inaccurate or incomplete. Furthermore, when data is collected, it is rarely used to guide improvement. This short guide is intended to provide practical tools for facility managers and clinical staff to improve the quality of data they report. Improving data, we have found, will also drive improvement in clinical service delivery. At the end of this training, readers should be competent in collecting, presenting, and interpreting basic data from Primary Health Clinics, Community Health Centres, Labour Wards, and Hospitals. How to Use this Guide This guide is set up in a module format and is meant to be hands-on and taught order starting from module #1 and continuing through module #5. Each module teaches clinic staff a critical skill to improve the quality of data collection and its use. At the end of each module is a series of tasks for the clinic staff to perform during the action period before proceeding to the next module. You should not proceed to the next module until all of the tick boxes for the current module have been checked. Before moving on to a new module, readers should also review the previous module to ensure understanding before moving on to the new module. This data improvement guide can be taught by a trained data improvement advisor or if so motivated, facilitybased improvement teams may use this guide independently. What does data mean? Data is simply defined as a collection of facts from which conclusions may be drawn. We collect data in our everyday lives, from counting the amount of airtime on our cellular phones to measuring how much money is left in our banking accounts. We then use this information to make decisions like how much time we can spend talking to our friends or whether we can afford to buy a new t-shirt. Data that is collected in a health care facility is no different we collect information to understand what is happening inside the health system and we use this knowledge to make decisions. What is data in health care used for? 1) Clinical decision making: Taking care of the client in front of you, for example: Which clients should be referred for HAART initiation? Which clients need to have a change in regimen because of treatment failure? Which clients are becoming anaemic? 2) Understanding how the PMTCT programme is functioning: Using data to find the gaps in the system of care, for example: Are all of the HIV positive women getting a CD4 count? Are the women with CD4 < 200 referred for HAART? Are all of the HIV exposed babies getting PCR tested for HIV? 3) Quality Improvement: Using data to decide if improvements are working, for example: Did the change you implemented result in the improvements you predicted? Has implementing a change idea resulted in improved client/staff satisfaction? 3

4) Evaluating the performance of a facility: Allows managers to assess facilities, for example: Which clinics need more resources in terms of staff, infrastructure, medications, etc.? Which clinics should be commended and showcased as best performers? Which clinics are underperforming and may need assistance? Which clinics need assistance in collecting, reporting, and interpreting data? How is data collected? 1) Registers individual patient information is transcribed by facility staff into a hard-copy book that is kept at the facility level 2) Reporting templates monthly summaries with requested statistics are submitted to the district office. The data for the reporting template is captured from the registers. 3) Databases A district information officer at the district level is responsible for entering data from individual facilities into the District Health Information System (DHIS). This information is then submitted to the province and then to the National Department of Health. Getting started: getting leaders on board Your data improvement programme will need to be integrated with a comprehensive quality improvement project that also focuses on improving the quality of PMTCT clinical services. Any improvement plan will rely on the participation and support of leadership at multiple levels. Facilities will need the support of their operational managers, their sub-district coordinators and their district leaders. Quality Improvement teams will need to be empowered by these leaders through the creation of time and space to pursue improvement activities including data improvement. They will also need leaders who maintain an interest in the improvement work, providing a valuable source of accountability, not for the purpose of judgment but rather to check in on and celebrate successes as facilities progress. Leaders should also review the information that is submitted from the facilities and provide troubleshooting mentorship. Getting started: data improvement execution plan This curriculum will take between one and two months to complete, with each individual module taking between one and two weeks. Some facilities may complete the data improvement exercises faster than others, so staff should not feel discouraged if their progress is slower than anticipated. After participants have completed the five modules, it is imperative that the successes gained during the initial phase are maintained with refresher courses and constant reinforcement by the team leader. 4

Checklist of items that should be present in every facility by the end of the data improvement project: Each module in the data improvement guide highlights key tasks that are to be completed and celebrated in the clinics. The success of this training will be measured by how many of these items are present and understood by facility staff. Some items are permanent wall fixtures, such as Aim statements, process maps, and run charts. These should be visible to everyone and continually updated even after the training phase of the data improvement project is complete. Facility-based Improvement team is formed and posted in a patient space to highlight the honor of serving on the improvement team Publicly posted Aim statement with a clear target and timeframe Publicly posted Process Map detailing each step of data flow from the patient held record until it reaches the District Information System Publicly posted Run Chart plotting an important measure over time Completed random data audit Performed a data validation exercise on a monthly summary sheet Developed a plan (including who, when, where) for doing monthly validation prior to monthly summary sheet submission 5

Module #1: Introduction, forming a Data Improvement Team, and developing an Aim Statement By the end of this module, the facility should have a: 1. Facility-based Data Improvement Team 2. Filing cabinet or new file folder to store copies of data sheets 3. Standard official PMTCT registers 4. Publicly posted Aim statement Introduction: The project leader should begin the first meeting with an explanation of the data improvement project, how it started, and why it exists. Be sure to include some of the main objectives of the project: 1) To use information to guide clinical decision-making 2) To improve the completeness and accuracy of PMTCT data reported to the district 3) To streamline the data flow and collection process in the clinics 4) To help identify areas for systems improvement 5) To teach basic skills regarding the management and use of routine data 6) To improve the reporting, interpretation, and presentation of non-pmtct data Participants should commit to weekly meetings and be willing to carry on the lessons from the data improvement project long after the curriculum is complete. Forming the Data Improvement Team: Forming the Data Improvement Team is the first step to improving data in your facility. Members of the team can include anyone in the staff including data clerks, nurses, doctors, counselors, and lab personnel. The team leader is the main driver of the Data Improvement Team and should be able generate enthusiasm, maintain momentum, and run regular meetings with the rest of the team. The team leader will be responsible for ensuring participation and making sure the deliverables (see page 5) are accomplished. Many facilities will also have a quality improvement (QI) team, responsible for improving the quality of PMTCT services. The data improvement team can either be a part of the QI team or exist separately depending on your preference. Elements to consider when forming the Data Improvement Team: The team leader must be willing own the process and drive improvement If possible, the team should include different members of the clinical staff including nurses, doctors, counselors, and data clerks. Anyone who is responsible for collecting or submitting statistics can be a part of the Data Improvement Team Team members should be activated and interested in improving the quality of the data The Data Improvement Team should be as small as possible whilst still comprising all of the key personnel that handle information (no more than 7 people) Once the team is defined, discuss the logistics including a mutually convenient once-weekly meeting time. The data improvement meetings should take approximately one hour. Be sure to outline a standing agenda/procedure that you will adhere to in future visits. The team meetings should be a place of shared learning and understanding where each participant feels comfortable contributing their thoughts and feelings about the best ways to improve. The team leader will need to facilitate this atmosphere of openness. 6

Developing a Common Aim: The group Aim Statement provides the goal or target that the team is trying to achieve. In the case of data improvement, the Aim Statement should focus on improving the quality of some component of data collection or reporting. A good Aim Statement seeks to answer two fundamental questions: How Much and By When? How Much? or, how much will your facility change, refers to a quantitative or qualitative measure that is applied to a particular process step (or is a measure of the outcome desired) and will reveal whether the process is performing to a desired specification. By When? or, by when will your facility change, refers to a defined timeline by which the measurable outcome will be accomplished. Each process step will involve the development of an individual Aim Statement. These need not be complicated, often being as simple as stating, Our facility aims to submit all 10 PMTCT indicators at 95% reliability by (specify the date). Other aspects of a good aim statement include: Transformative the Aim would not be possible if the process continues to be performed the same way Ambitious the Aim should be meaningful to improving access to or quality of care received by the patient Specific the Aim seeks to focus in on one piece of the data flow pathway, addressing issues within the control of the improvement team to change Clear the Aim should be entirely clear to anyone who reads it (even those visiting from outside you facility), communicating exactly what it is your team is focused on trying to improve How to develop an Aim statement (see appendix for worksheet): A. Meet together as a team to review the overall Aim(s) of the improvement plan (usually to reduce MTCT to < 5% in the next 18 months) B. Review together what you know and do not know about your facilities current performance C. If possible, in one sentence, answer the two fundamental questions ( how much? and by when? ) D. Review the draft aim statement together a couple of times to make sure it is clear to the team and also keeping in mind that it should be clear to anyone who reads it. Example Aim Statements these might be used to structure your teams Aim Statements Ensure that 95% of CD4 test results received will be recorded in CD4 register by August of 2009. Ensure that 100% of requested data elements are submitted on time to the district office by January of 2010. Once you have developed at least one Aim Statement, write it in LARGE letters on a piece of paper, and post it in the room where the Data Improvement Team meets. Remember, this is your goal, and you will focus your improvement activities on achieving this goal. End of Module #1: Introduction, forming a Data Improvement Team, and developing an Aim Statement By the end of this module you should have completed the following deliverables (tick if complete): Formed an Improvement Team Developed an Aim Statement and posted it on the wall 7

Module #2: Understanding the system creating a data Process Map By the end of this module you should: 1. Know how to track where in your facility data is collected 2. Know how to follow where data goes after it is collected 3. Identify gaps in data flow that can be improved A process map is a way to graphically represent how information flows from an activity to registers to tally sheets to monthly reports and ultimately to the DHIS. A good process map should include these elements: Reflect current reality the process map will be a true picture of what each patient experiences when they visit your facility (as opposed to what should happen according to guidelines ) Input from many sources the process map will include input from caregivers at all levels, from clerks and counselors to nurses and doctors, and if possible might include input from patients Specific the process map will identify step by step what happens, separating steps in the care pathway if they are conducted by a different person, in a different space, or at different times, being careful to extract each individual event in the care process Big picture the process map will provide a clear picture for all stakeholders (from beginning to end) about what happens in the targeted process of care How to create a data Process Map (of your facility) A. Meet together as a team with the intention of mapping out how data moves around your facility. B. Using a flip chart paper identify the first place where data is collected (for example, when the patient checks in, her name is recorded), and where it ends (for example, at palpation when the sister writes in the patient-held record). Your process map should follow the patient as they go through the facility including data collection points, tools used at each step and personnel who record the data. Mark the beginning step as a box in the upper left hand corner and the ending box in the lower right hand corner (GRAPHIC below). Keep in mind that a process map can be as big or as small as you want it to be. C. Together, as a team, describe the source of data and who handles the data and what they do with the data step by step, from the starting point until the ending point a. Each step should be drawn as a box on the flip chart paper with Arrows leading from step to step. b. If a step requires a decision (yes/no for example) then arrows leading divisions in the pathway are created and subsequently followed to the end (see examples below) D. Once the Process Map is complete, it can be helpful to analyze the map; describing how long each step takes, who performs it, where it is performed, and where data can be lost or damaged, in an effort to begin identifying where improvements can be made. E. Decide which problem step should be addressed. Choose the area that you believe will have the largest impact on the process and that can be improved. 8

Process Mapping Basics (Figure 1): End of Module #2: Understanding the system creating a data Process Map By the end of this module you should have completed the following deliverables (tick if complete): Publicly posted Process Map detailing each step of data flow from the patient held record until it reaches the District Information System 9

Module #3: Random data auditing By the end of this visit the facility should have: 1. Performed a random data audit on at least THREE data elements In order to ensure that the data submitted to the DHIS are of good quality, random data audits should be performed by the Data Improvement Team leader and the data clerk. It is not necessary to check all data elements at once, but a random audit can give valuable information about the quality and the integrity of the data as it flows from source documents to the summaries. The purpose of this data audit is not to lay blame on a particular person, but rather to look for ways to improve the quality and accuracy of the data that is submitted to the DHIS so that errors do not continue in the future. How to perform a data audit A. The improvement team leader, data clerk, and another member of the improvement team should meet and review the plan for the data audit. B. The group should choose at least THREE data elements for the data audit. C. You should then gather the registers that are used to submit monthly summary statistics and the previous month s PMTCT monthly summary sheet. D. Using the three data elements you have selected, the team leader, data clerk, and additional team member should reconstruct the monthly summary counts from the sources (e.g. PMTCT registers or daily tally sheets). If the data are accurate it should be possible to get the same counts as previously submitted to the district. If there are errors, make note of them. What was the cause of this error? E. Did the three auditors agree? If no, why not? What processes should be changed to make this more efficient and more accurate? It is critical that any errors that you have found while doing the data audit are discussed and a plan is made to address them. If necessary, perform the data audit on data elements until you are confident you have discovered all of the errors and you have made a plan to address them. If you have found errors on your monthly summary sheet, please re-submit a corrected summary as errors can still be corrected in the DHIS even after the first submission. Once you have performed your first data audit, you can decide as a team how frequently you want to perform further data audits. Some teams perform daily data huddles where they meet frequently, while others meet weekly or monthly. You should definitely meet at least monthly to review your data prior to submission as you will learn about in the next section. End of Module #3: Random data auditing and monthly summary submission By the end of this module you should have completed the following deliverables (tick if complete): Completed random data audit 10

Module #4: Data Validation, Indicator Calculation, and Monthly Summary Submission By the end of this visit the facility should have: 1. Validated all data present in the monthly summary 2. Performed simple indicator sense checks 3. Made a plan for monthly PMTCT data validation Data validation By now, you have reviewed the data you have previously submitted to the district using just a few data elements. Remember, this data is entered into a database and submitted to province and then to the national department of health. The quality of this data is important for tracking how the PMTCT programme is functioning. If you submit more accurate data, the leadership will have a clearer picture of your performance. Now it is time to validate your monthly submission sheet in its entirety. You do not have to go back to the registers to validate every number; rather, you should look at the submission as a whole and ensure that all of the numbers make sense before submission. While variations in numbers of patients per week are inevitable and acceptable; figures that are extremely outside the norm should spike suspicion of a data error such as illustrated in the example below (Table 1). As you can see, the 1 st Antenatal clients on week 4 are way above the average number of clients that are seen every week; therefore this would not be a valid count. Table 1: Example of a weekly summary sheet with a data error where is the error? Data Element Week1 Week 2 Week 3 Week 4 TOTAL 1 st ANC Clients 30 26 22 203 281 ANC counseled for HIV 30 24 22 23 99 ANC tested for HIV 30 23 22 23 98 The next step in the validation process is to check if the data make clinical sense by looking at two or more data elements in relation to the other. For example the clinical process dictates that you get counseled for an HIV tested before the test is done, therefore there should never be more people that are tested than are counseled for the test. Another clinical and logical process is infants that are born to HIV positive women are given Nevirapine (NVP) Syrup, therefore the number of NVP doses should never exceed the number of HIV exposed infants; please refer to the example below. Table 2: Example of data errors on monthly submission sheet where are the errors? Data Element Week1 Week 2 Week 3 Week 4 TOTAL ANC counseled HIV 100 80 98 96 374 Tested for HIV 98 87 99 96 380 Live births to HIV pos women 105 200 189 170 664 Nevirapine given to infants 105 203 192 170 670 AZT prophylaxis given to infants for 7 days 98 145 190 168 601 AZT prophylaxis given to infants for 28 days 7 68 3 2 80 11

In the example, all the figures that are marked in bold represent data errors or poor quality of data. With your team discuss the quality of the data that is presented above. What are the possible causes for this picture and how would you fix problems above? There are more logical sense checks on the data than the ones listed above. You should discuss these with the Data Improvement Team at your weekly meeting. Once you are confident that there are no major errors, it is safe and appropriate to submit the monthly summary sheet to the district. The process of data validation should occur every month before submitting the monthly statistics. Before you complete the module, make sure you have a plan of who will perform the monthly data validation (usually the Data Improvement Team leader), when in the month the validation will occur (for example first Monday of the month), and who is response responsible for submitting the monthly summary sheet to the district. Of note, this process can be used for other data elements other than PMTCT, so once you are familiar with the process of data validation, please try this process with other forms of data. End of Module #4: Data Validation and Indicator Calculation By the end of this module you should have completed the following deliverables (tick if complete): Performed a data validation exercise on a monthly summary sheet Developed a plan (including who, when, where) for doing data validation prior to monthly summary sheet submission 12

Module #5: Developing Process Measures and Run Charts By the end of this module you should: 1. Understand the purpose of tracking longitudinal data in the form of a Run Chart 2. Have a publicly posted Run Chart tracking the part of the data process map that is the least reliable A run chart is a simple line graph which is used to track the performance of one (or more) steps in the process targeted for improvement across a defined period of time. Run charts allow you to see, at a glance, how a specific part of the PMTCT program is changing. Run charts typically graph out process measures, which are specific data elements that evaluate the performance of a step in a pathway. An example of a process measure is the percent of ANC 1 st bookings that have an HIV test (more examples are below). How to develop process measures A. Meet together as a team, coming to consensus on which steps in the process map are absolutely essential to reach the Aim List these B. For each essential step answer the following questions a. How will we know if each patient received this step in the care pathway? b. How many patients do we expect to receive this step in the care pathway? C. For both questions list any data elements (in registers, patient held records or other medical records) that will help to provide the answers to the questions above D. Identify what data your team needs for effective process measures but currently does not collect E. Develop a numerator and denominator (said another way, actual performance and target performance) for each essential step identified F. Where data does not yet exist, or is not currently collected, discuss how information for this step could be obtained Things to keep in mind with ideal performance (under real working conditions), the numerator and denominator will be equal to each other (indicating perfect performance). When they are not (this will be most of the time) there is a need for improvement. Example Process Measures % HIV tested = (HIV tested/hiv pre-test counseled) X 100 o Numerator = HIV tested (actual number tested) o Denominator = HIV pre-test counseled (expected number to test) % infants receiving DPT 1 immunization = (DPT1 immunized/total infants at EPI clinic) X 100 o Numerator = DPT1 immunized (actual number of infants receiving DPT1 immunization) o Denominator = total infants at EPI clinic awaiting first immunization at 6 weeks (expected number of infants to be immunized) % HIV positive screened for TB = (HIV positive TB screened/hiv positive) X 100 o Numerator = HIV positive TB screened (actual number of HIV positive patients submitting a sputum sample) o Denominator = HIV positive (total number of HIV positive patients identified by a clinic Run-charting Run charting is a powerful tool for understanding and analyzing data. It is also the most effective way to show changes over time and to visually see gaps in clinical care. Some major themes of run-charting: 13

Time is always on the HORIZONTAL axis going from right to left What you are measuring (the process measure) is always on the VERTICAL axis There are two types of lines that should be drawn in different colors o Target line: what you are aiming for o Trend line: what you current, actual data shows Run-charting Exercise: The following exercise will take you through the process of creating a run chart with hypothetical data. The purpose of this exercise is to help you gain experience with run-charting and in analyzing gaps in your system by using data. Using the following information from a hypothetical ANC clinic, plot the data on the attached run charting tool (Appendix 2). Remember to plot the numbers on the vertical axis and the months on the horizontal axis. For each month, connect the dots for each data element (ANC first bookings, counseled, tested) to show the trend over time. ANC First Bookings # Counseled for HIV # Tested for HIV January 2008 18 16 14 February 20 18 13 March 15 12 9 April 21 17 15 May 17 14 12 Jun 18 16 12 July 16 15 10 August 22 19 15 14

When you are done plotting the data, it should look like this: Questions to answer before moving on: What are the problems with this clinic? Where do you see the gaps on the run-chart? Are they performing as well as they could be? If they were performing well, what would the lines look like? Next, plot the following additional set of data on the SAME chart, continuing where you left off: ANC First Bookings # Counseled for HIV # Tested for HIV September 2008 18 18 17 October 20 20 19 November 15 14 14 December 21 21 19 January 2009 17 17 17 February 18 18 16 15

Your updated run chart should now look something like this: Questions to answer before moving on: Was there a change in this facility? What was the change? When did it occur? Can you draw an arrow on the chart showing where the change happened? How to develop and use a run chart in your facility using your own data A. One person from the improvement team should volunteer to physically update the run chart each day, week or month (depending on the measure and the frequency of the service at the clinic) B. This person should record, according to the time period, both the numerator and the denominator in a table below the run chart C. Directly above the table the team member should plot a dot where the numerator should be and the denominator should be on the graph (in different colors if possible) D. The team member should connect the dots, numerator to numerator and denominator to denominator, to slowly and methodically produce the run chart E. Whenever a change is introduced to the process of care the team member will make a notation on the run chart on the appropriate date. Marking the run chart with important events creates an annotated run chart. F. The team should review the progression of the run chart each time meet, looking for trends in the data (these indicate process performance, process improvement or emergent problems depending on if data is getting closer to what the team wants or farther away) G. Use the run chart tool that is attached (Appendix 2) 16

More Examples of Annotated Run Charts End of Module #5: Developing Process Measures and Run Charts By the end of this module you should have completed the following deliverables (tick if complete): Publicly posted Run Chart plotting an important measure over time 17

Appendix 1: Aim statement worksheet Adapted from URC s HCIP Project Presentation 18

What you are measuring Appendix 2: Run Chart Tool Title: Measure 1: Measure 2: Measure 3: TIME 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31