Guide to Ensuring Data Quality in Clinical Audits

Transcription

1 Guide to Ensuring Data Quality in Clinical Audits Nancy Dixon and Mary Pearce Healthcare Quality Quest Clinical audit tool to promote quality for better health services

2

3 Contents 1 Introduction Who this guide is for How the guide is intended to help 1 2 What s involved in data quality What data quality means What s involved in achieving data quality for clinical audits 3 3 How to ensure that the purpose of the clinical audit is right 4 4 How to ensure that the right cases are selected for a clinical audit Cases to be included and excluded How to confirm that cases identified for a clinical audit are the right cases How to decide on the number of cases to include in a clinical audit and how 8 they will be selected How to decide to include a population or a sample and the type of sample Representative sampling techniques Non-representative sampling techniques How to decide on sample size What to do if cases selected for the audit don t work out 13 5 How to check on the validity of clinical audit standards What validity of clinical audit standards means How to test the validity of clinical audit standards The efficiency of clinical audit standards in identifying good and not so good 17 quality of care 5.4 How to test if clinical audit standards are sensitive and specific 18 6 How to check if data needed for a clinical audit can be found 19 7 How to ensure that data collection processes produce reliable data How to ensure that key terms are defined and instructions for making decisions 21 are specified 7.2 How to design and test data collection tools or systems How to develop and test a protocol for data collection How to select and prepare data collectors for a clinical audit How to test the degree of inter-rater reliability How to pilot test data collection 30 8 How to validate data collection and data collation How to monitor adherence to case selection and the data collection protocol 31 and process 8.2 How to prevent threats to data quality during collection and collation Testing data Tracking data Transferring data Tidying up data Triangulating data How to act to resolve issues in data collection and collation 35 Guide to Ensuring Data Quality in Clinical Audits i of 46

4 9 How to avoid pitfalls in data collection for clinical audits Pitfalls related to people and organisations Pitfalls related to data How to make arrangements for sharing data for clinical audit and ensure 39 that information governance requirements are met 10.1 Agreeing on and following arrangements for sharing clinical audit data Ensuring that information governance requirements are met 39 References 40 Acknowledgements 44 Appendix. Table for selecting sample size for a clinical audit and formulas for 45 calculating sample size for a clinical audit ii of 46 Guide to Ensuring Data Quality in Clinical Audits

5 1 Introduction 1.1 Who this guide is for This guide is for leads, managers and staff carrying out or supporting participation in national clinical audits and for the following people who are involved with clinical audits in individual healthcare organisations: Clinical audit or clinical governance managers and staff Clinical audit leads Clinical audit committee chairs and members. 1.2 How the guide is intended to help Clinical groups are sometimes expected to make changes in patient care, based on findings of clinical audits. Clinical groups need to have confidence in clinical audit data in order to agree to change their practices. Retrieving data from electronic or paper health records for clinical audits is inherently more complex than clinicians may imagine. 1 Factors such as imprecisely worded directions for making decisions about the quality of care, vague definitions of key terms, poorly designed data collection tools, inappropriate interpretation by data collectors, and poor or missing recording of data in data sources may compromise data quality. 1 This guide describes how a clinician or group carrying out a clinical audit can ensure the quality of data collected for the audit. It includes: what data quality means what s involved in achieving quality data for clinical audits how to ensure that the purpose or objective of a clinical audit is so clear that it identifies the nature of the data needed for the audit how to ensure that the cases selected to be included in or excluded from a clinical audit are the right cases and that cases selected won t produce biased results how to test the validity of clinical audit standards how to check if data collection processes are producing reliable data how to select data collectors for a clinical audit and ensure they are doing the right job how to quality control data collection and data entry how to avoid pitfalls in data collection how to make arrangements for sharing data for clinical audit purposes across healthcare organisations and ensure that information governance requirements related to clinical audit data are being met. Examples relating to data quality for clinical audits are provided in the guide. Guide to Ensuring Data Quality in Clinical Audits 1 of 46

6 2 What s involved in data quality 2.1 What data quality means Data quality has been recognised as an issue in the NHS 2 6 and NHS organisations are implementing strategies to audit and improve the quality of data produced. 7 Data quality has been defined by dimensions or characteristics including accuracy, availability, completeness, relevance, reliability, timeliness and validity. 3, 8 14 In addition, some informatics experts define data quality as data that are fit for purpose. 11 The key terms related to data quality are defined 3, 8 14 in the box, particularly as they apply to data about patient care. Characteristics of data quality and their meanings Characteristic Accurate Available or accessible Complete Fit for purpose Relevant Reliable Timely Valid Meaning Data are correctly input and reflect exactly patient care transactions. There are no mistakes in the data in comparison to data in an original data source or to what actually happened. Data enable identifying exact patients or events correctly and can be retrieved relatively rapidly when needed. All the elements of information needed are present in the designated data source and no elements of needed information are missing. Data are suitable for their intended purpose. Data are of interest to data users to enable them to meet the intended purpose of the data. Data are the same no matter who collects the data or when a person collects the data. Original data are recorded contemporaneously with the provision of patient care or service and are available in time to make safe decisions about the quality of patient care or service. Data meaningfully represent exactly what they are intended to represent. Patient care data also must be secure and confidential. 2 of 46 Guide to Ensuring Data Quality in Clinical Audits

7 2.2 What s involved in achieving data quality for clinical audits Data quality in a clinical audit is not simply about data collection. Achieving data quality is embedded in all the stages in a clinical audit. Key questions about data quality should be asked at each stage of a clinical audit. The questions are in the box, along with an explanation of what s involved in each stage and the related characteristic of data quality. To provide clinical groups with a true picture of the quality of patient care that justifies changing current clinical practice, the answers to all the questions have to be yes. In addition to the characteristics of data quality described in section 2.1, there are other characteristics of standards used in clinical audits and these are also included in the box. Stages of a clinical audit that involve data quality and questions about data quality at each stage, what s involved and the characteristic of data quality involved Clinical audit stage and key question What s involved Being sure that: Data quality characteristic Design Is the purpose or objective of the clinical audit right? The purpose, aim or objective of a clinical audit is explicit about confirming current good practice or improving current practice. Fit for purpose Design, especially case selection Are the right cases selected to be included in the audit? There is no bias in the selection of the intended or actual cases included in an audit. The cases to be included in and excluded from an audit and the directions for making the inclusion and exclusion decisions are explicit. All the intended cases are retrieved for an audit and there are no missing cases. Unbiased and complete (for case selection) Development of standards Are the right things being measured about quality? The objective(s) of an audit is(are) translated into specific aspects of the care to be measured in the audit. Valid Relevant Development of standards Are the measures right for the things being measured? The measures, such as standards, developed for a clinical audit are capable of pinpointing instances of good and not so good patient care consistently and efficiently. Sensitive and specific (see page 18 for definitions) Development of standards and data collection Can data be retrieved for the things that are being measured? The data needed to make explicit decisions about whether or not there is compliance with agreed standards exist or are capable of being gathered relatively efficiently. Depending on what is being measured in an audit, data also can be collected contemporaneously with the provision of care. Available or accessible Timely Guide to Ensuring Data Quality in Clinical Audits 3 of 46

8 Stages of a clinical audit that involve data quality and questions about data quality at each stage, what s involved and the characteristic of data quality involved Clinical audit stage and key question What s involved Being sure that: Data quality characteristic Data collection Are the data collection processes right for what is being measured? The way data are collected for a clinical audit produces data that enable clinical groups to be confident that the data are consistent no matter who collected the data or when the data were collected. Reliable Data collation and validation Are the data being checked prior to analysis? Data collected for a clinical audit are correct and complete. Accurate Complete (for clinical audit standards) Data analysis Are the data being analysed the right way? The analysis and presentation of clinical audit data enable clinical staff to see easily if current patient care is or isn t consistent with the audit standards used to measure quality of care. Fit for purpose The following sections of the guide suggest how to ensure data quality at each of the clinical audit stages described in the box. 3 How to ensure that the purpose of the clinical audit is right The purpose of a clinical audit is to: confirm that the current quality of care is consistent with best practice or demonstrate that the quality of care is improved by acting on shortcomings shown in current care and repeating data collection to show the effect of actions taken. An objective for a clinical audit has to be clear about the aspect(s) of the quality of care that is(are) to be measured in comparison to best practice. A model for writing an objective for a clinical audit is in the box. 15 Objective model Verb The intention for doing the clinical audit exactly how the audit relates to confirming or improving quality Quality focus + + The feature(s) of quality to be measured by the audit what the audit will focus on Subject The specific care or service the audit is about what the clinical subject of the audit is Enables fit for purpose Indicates data to be collected Indicates cases to be included and excluded 4 of 46 Guide to Ensuring Data Quality in Clinical Audits

9 The quality focus part of the objective identifies the nature of the quality of care to be measured, and therefore, the data to be collected in an audit. An example is in the box. Objective for a clinical audit on chest drains and how the objective relates to audit standards and data Background A clinical group in an acute hospital wants to carry out a clinical audit on chest drains. Objective Following discussion on what the group wants to achieve, the group members agree that the objective of the clinical audit is to ensure that chest drain insertion and management are carried out effectively, that is, consistent with the British Thoracic Society (BTS) guidelines for the insertion of a chest drain. 16 What has to be measured According to the objective, the clinical audit has to measure the effectiveness of chest drain insertion and management. The effectiveness of care is as defined in the BTS guidelines that describe the right way to insert a chest drain and manage a patient with a chest drain. 16 Audit standards and data that have to be collected The audit standards will specify the exact process to be followed in inserting and managing a chest drain. The data to be collected are about whether or not the stages in the process of inserting a chest drain and managing a patient with a chest drain have been followed. The group s objective does not require any other data to be collected. 4 How to ensure that the right cases are selected for a clinical audit There are several decisions involved in selecting the right cases for a clinical audit including: what exactly are the cases to be included in and excluded from the audit how the selection of the right cases for the audit can be assured if it is feasible to include all the cases specified for the audit in a given time period if it is not feasible to include all the cases specified for the audit, how a subset of cases will be selected for inclusion in the audit what to do if cases selected for the audit don t work out for some reason, including that some cases are missing. 4.1 Cases to be included and excluded Specify exactly the patients, cases, situations, circumstances or events to be included in a clinical audit. Consider all of the following when specifying patients or cases to be included: Guide to Ensuring Data Quality in Clinical Audits 5 of 46

10 the specific diagnosis, condition, surgical procedure or special procedure, if the subject of the audit is care provided to patients with a diagnosis, condition, surgical or special procedure. Agree on the codes to be used to retrieve the cases from your organisation s information system. If previous history of the diagnosis or condition is relevant, specify the history in detail. For example, an audit may be focused on patients who have had a stroke for the first time or it may be focused on patients who have had a repeat stroke. the age range of patients to be included, if age is important to the subject of the audit. For infants and children, specify age in days or months, depending on the subject of the audit. the specific referrals by reason, condition, source of referral or time period, if referrals relate to the subject of the audit the specific visit by reason for visit, diagnosis, number of visits or time period, if GP or clinic visits relate to the subject of the audit the exact events or circumstances and how they will be identified, if events or circumstances, such as patient falls, relate to the subject of the audit. Examples of special situations that relate to defining cases for a clinical audit are in the box. Special situations for defining cases to be included in a clinical audit Clinical audit subject depends on identifying procedures Finding cases for a clinical audit depends on the accuracy and completeness of coding Finding cases for a clinical audit relies on verifying cases and collecting data concurrent with the delivery of the care involved Finding cases for a clinical audit requires screening of possible cases A clinical audit on the effectiveness of completion of the consent process depends on being able to specify the procedures for which consent is required and then finding a perfect list of patients who have had one of the procedures for which consent is needed. For an audit on the effectiveness of use and management of chest drains, finding the patients to include depends on whether or not chest drain insertion has been recognised and coded correctly in the organisation s information system. If the cases are not coded or are not coded consistently, a clinical group needs to work out the best way to identify cases. A clinical audit on the effectiveness of providing special clinical diets to hospital inpatients may require collecting data as meals are delivered, if the exact meals actually delivered to patients aren t routinely recorded for patients for whom a special clinical diet is requested. For some clinical audits, it may be relatively easy to specify the patients to be included in the audit but difficult to actually identify the exact patients to be included. For example, a clinical audit on the effectiveness of implementation of the Mental Capacity Act would include patients who lack mental capacity temporarily or continuously. However, specific indicators would be needed to identify the patients whose care should be measured for the clinical audit and patient records would have to be manually screened to identify those patients who meet the indicators for lacking mental capacity. 6 of 46 Guide to Ensuring Data Quality in Clinical Audits

11 If the description of the patients or cases to be included isn t comprehensive, it may be necessary to also specify the patients or cases to be excluded, for example, patients with co-morbidities, patients in a particular age group for which the audit subject is not intended or non-nhs patients. An example is in the box. 17 Example of specifying patients or cases to be included in or excluded from the national clinical audit on chronic obstructive pulmonary disease (COPD) Diagnosis To ensure accurate diagnosis of COPD exacerbation, the lead clinician at each unit was encouraged to review the medical notes of patients included in the audit to check for any evidence of misdiagnosis. Any patients whom the lead clinician considered to have been misdiagnosed (i.e. diagnosis appeared to be COPD on admission but later deemed incorrect) were to be excluded from the audit. Also excluded were any patients where the diagnosis was changed to exacerbation of COPD from another presenting condition, as this would have affected the patient s early management in hospital. Sites were asked to include only the index admission for patients having more than one admission within the data collection period. A very small number of patients with repeat admissions were identified within sites and these were excluded. Admission For the purposes of the National COPD Audit 2008, an admission is defined as an episode in which a patient with an acute COPD exacerbation is admitted to a ward and stayed in hospital for 4 hours or more (this includes Emergency Medicine Centres and Medical Admission or similar units but excludes Accident and Emergency Departments) prior to discharge or acceptance to an early discharge scheme. A stay in hospital of less than 4 hours is a non-admission and is not included. Early discharge scheme Early discharge schemes have a variety of names, including hospital at home, or may be known by local acronyms. Sites were asked to include in the audit those patients who presented to hospital with COPD exacerbation and were then accepted onto an early discharge or hospital at home scheme, so reducing length of stay. Patients seen at home by such schemes but not presenting to hospital were excluded from the audit. 4.2 How to confirm that cases identified for a clinical audit are the right cases For every clinical audit, those carrying out data collection or data validation should confirm that the cases intended for inclusion in an audit are actually included. The method for confirming that the cases identified are the right cases may depend on the sources used to identify cases. Examples are in the box. Approaches to confirming that the right cases for a clinical audit are included For clinical audits that involve patients with a specific diagnosis or having had a specific procedure, for example, for a national clinical audit For cases for a particular diagnosis or procedure, compare the number of cases identified as needed according to the data source agreed for the clinical audit and the time period over which the cases occurred with the number of cases that are recorded by Hospital Episode Statistics (HES) for the same time period. Errors in coding can contribute to errors in HES data Use robust and reliable organisational records such as clinic or therapy lists, perfusion registers or pathology reports to validate that all eligible cases for a clinical audit are being identified Guide to Ensuring Data Quality in Clinical Audits 7 of 46

12 Approaches to confirming that the right cases for a clinical audit are included For other local clinical audits For clinical audits that rely on electronic systems in general practices Before collecting data for each case, check that the case meets completely the description of cases to be included in the audit. A person overseeing data collection or another data collector can check the cases. If cases to be included are missing from the data sources to be abstracted for the audit, make every effort to locate the missing cases. Substituting cases by using patient records that are readily available can produce biased results. Use Data Quality Probes, which involves posing a query in a clinical information system where the result can be used as a measure of the performance of that system, that is, there is strict concordance of the association between one data item and another. 23 An example is that all patients identified as having diabetes have an estimation of HbA1c recorded in the system. 4.3 How to decide on the number of cases to include in a clinical audit and how they will be selected Clinical audit staff members sometimes recommend that local clinical audits do not need to include more than 30 or 50 cases. The basis for this recommendation is that if care is not being provided in accordance with clinical audit standards in 30 or 50 cases, there is no need to look at more cases. Clinical groups need to take action to improve compliance with the standards. This approach may be suitable for clinical audit subjects that are limited to reasonably small numbers of patients handled by a reasonably consistent clinical group. Where performance may vary by clinical service, location, primary care centre, staff shift pattern, or time of year, a single sample of 30 or 50 cases may produce biased results, particularly when the cases are consecutive. A systematic approach is needed to decide on the number of cases to include in a clinical audit and how to select the cases. There are several considerations that will affect these decisions, such as those in the box. 15 Considerations that can affect decisions about the number of cases to include in a clinical audit 1. How many cases of what you want to include in the audit are there in a given time period, such as a week, a month or a year? Suppose an emergency department group is interested in the effectiveness of assessment of patients who come to the emergency department with symptoms of substance abuse, including alcohol or drugs, as a subject for a clinical audit. If the group estimates that about 100 patients a day with such symptoms come to the department, and the group recognises the shift patterns of staff working in the department and possible seasonal variations in patient presentations as well as staffing, the group will have to decide how to get an unbiased sample for the audit. A sample could include a designated number of cases, that is, a cohort, by week or month, for example. 8 of 46 Guide to Ensuring Data Quality in Clinical Audits

13 Considerations that can affect decisions about the number of cases to include in a clinical audit 2. How difficult is it to find cases that would be eligible for inclusion in the audit? If identifying cases eligible for inclusion for an audit, such as people who lack mental capacity in an acute hospital, is itself time-consuming, the audit may have to include a relatively small sample. 3. How difficult is it find the exact information needed for the audit for each case that is to be included? For example, suppose an objective of an audit concerns an aspect of quality that may not routinely be documented, such as the provision of special clinical diets to the right patients. If data are to be collected concurrently, staff need to be available to collect data when meals are delivered. Therefore, the number of cases and/or time period for data collection may be affected by staff availability for data collection. 4. Does a clinician or a group want to be able to generalise the audit findings to other cases? If a clinical group wants to generalise the findings to other cases, then sampling has to done the right way to enable the right conclusions to be made. Normally, the sample will need to be representative. 5. What is the level of statistical confidence a clinician or a group wants to have that the findings of the audit will be representative of what happens to all similar cases? For example, if a change in practice has significant financial or clinical implications, clinical groups and managers may not be convinced about making the change on a sample that is smaller than a 90%, 95% or 99% statistical confidence level would provide. 6. How much time is available to collect data for the audit? If time is limited, a small number of cases may have to be selected. A group could use small cohorts of cases over a longer time period, for example, 5 cases per week for 10 weeks, with the cases selected using a random sampling technique How to decide to include a population or a sample and the type of sample It is important to be clear about the difference between a population and a sample of cases for a clinical audit and types of samples. The terms and their meanings are in the box. 15 Sampling terms and their meanings Population All, the entire collection of, the patients, events or things in which you are interested. A population can range from a relatively small number to a large but finite number to an infinite number, depending on the time period you refer to or the number of clinical services or healthcare organisations included. Examples of populations are: all the patients who had a chest drain inserted in hospital X last week; all the patients who had a chest drain inserted in the UK last year; or all the people in the UK who have had a chest drain inserted sometime in their lives. Guide to Ensuring Data Quality in Clinical Audits 9 of 46

14 Sampling terms and their meanings Sample Representative (or probability) sample Some, a specific collection, of the patients, events or things that are drawn from a population in which you are interested. Samples can be representative or non-representative of the population. A sample that attempts to ensure that the sample contains cases that represent the population An example of a representative sample is every 5th patient who had a chest drain inserted in hospital X in the last month from a list of all patients arranged in date sequence of a drain insertion. Non-representative (or non-probability) sample A sample that does not attempt to ensure that the sample contains cases that represent the population. A non-representative sample is used when it is not feasible, desirable or economical to use a representative sample. An example of a non-representative sample is the first 10 patients in each of the hospitals in the South who had a chest drain inserted last year. There may be bias in the first 10 patients, for example, if they were all cases having the drains inserted in the emergency department. Clinical audit findings could be biased if the decision on including a population or a sample of cases for a clinical audit is not made carefully. The questions in the box can help in making the decision on using a population or a sample. 15 If you want to be able to say that the audit findings from a sample of cases can apply to all cases, you have to select a representative sample. How to decide on a population or a sample for a clinical audit Specify the patients, cases, events or circumstances to be included in and excluded from the audit, and/or the intended time period for including the cases. Then consider the following. 1. Can you find with certainty all the patients, cases, events or situations needed for the audit, that is, can you get a perfect list? 2. Do you need to include all the patients, cases, events or situations in the audit? 3. Is there time or resources to include all the patients, cases, events or situations in the audit? If Yes, go on to question 2. If No, use a non-representative sample because every patient does not have an equal chance of being included in the audit. If Yes, go on to question 3. If No, do you want a sample that attempts to represent the population? If Yes, use a representative sample. If No, use a non-representative sample. If Yes, use the population. If No, do you want a sample that attempts to represent the population? If Yes, use a representative sample. If No, use a non-representative sample. 10 of 46 Guide to Ensuring Data Quality in Clinical Audits

15 Some clinical audit staff think of representative sampling as any group of cases that are likely to be typical and that can be chosen from any convenient data source. However, representative sampling involves giving each case eligible for inclusion in the audit an equal chance of being selected for inclusion in the audit. Representative sampling requires having a perfect list of all eligible cases and selecting cases for the audit from the perfect list in accordance with the rules for random sampling. Key ideas about representative sampling are in the box. 15 An explanation of representative sampling What it is Why use it When to use it How to use it A representative sample has the best chance of reproducing in the sample the key characteristics of the population in the same proportion as they occur in the population. Use representative sampling when you want to draw inferences about what is happening to a population based on a sample. You can use representative sampling only if you can identify all cases in the population (to give each case an equal chance of being in the sample). Use a random sampling technique Representative sampling techniques Some types of representative sampling techniques are described in the box. 15 Representative sampling techniques and their meanings Sampling technique Meaning When to use Simple random sampling A given number of people, events or things is selected from a complete list of people, events or things eligible for inclusion (the population) in such a way that each has an equal chance of being included in the sample. When the population is more or less the same or highly similar for the characteristics that are key to the objective of the audit and every person, event or thing in the population can be identified Stratified random sampling All people, events or things eligible for inclusion (the population) are divided into groups or strata on the basis of certain characteristics they share such as age, diagnosis, medication, clinic or day of the week. Then a random sample is selected from each group. When the population is not the same or highly similar for the characteristics that are key to the objective of the audit and every person, event or thing in the population can be identified Guide to Ensuring Data Quality in Clinical Audits 11 of 46

16 Representative sampling techniques and their meanings Sampling technique Meaning When to use Systematic (interval) random sampling A fixed interval is specified. The people, events or things are arranged in a sensible order such as by date of receipt of referral. The first person, event or thing eligible for inclusion is selected at random and then every person, event or thing that falls at the fixed interval thereafter is selected for the sample. When the population is more or less the same or highly similar for the characteristics that are key to the objective of the audit; people, events or things can be arranged in a natural sequence; it is believed that there is no underlying bias in selecting every nth person, event or thing; and every person, event or thing in the population can be identified Non-representative sampling techniques You could use a non-representative sampling technique if you are not interested in making inferences about a whole population with any degree of statistical confidence or you are unable to identify every person, event or case in the population, but you are interested in using an audit to understand a situation or a problem. Some non-representative sampling techniques are described in the box. 15 Non-representative sampling techniques and their meanings Sampling technique Meaning When to use Purposive sampling People, events or things for inclusion in the sample are selected for specific purposes, particularly to provide data related to the purposes. When the population is known, a small sample will suffice and you want to exercise judgement in selecting the sample and not leave selection to chance Convenience sampling Quota sampling People, events or things for inclusion in the sample are selected because you can get them relatively easily. Subgroups or strata of a population are identified and a desired number of people, events or things from each subgroup is set for inclusion in the sample. Then, people, events or things are sought until the quota for each subgroup is achieved. When you don t want to generalise the findings to a population and you want a manageable sampling method When: a list of the eligible population from which to draw a random sample is not available the data need to be collected faster and cheaper than random sampling methods would allow just knowing something about each group is sufficient, even if the findings may not be representative 12 of 46 Guide to Ensuring Data Quality in Clinical Audits

17 Clinical audit staff sometimes refer to snapshot sampling by which they appear to mean a small number of cases designed to give a quick picture of patient care. Snapshot sampling, therefore, can be an example of convenience sampling as described in the box. 4.4 How to decide on sample size Use the questions in the box to consider how many cases should be included for a sample. 15 How to decide on the size of a sample for a clinical audit How many patients, events or situations will a clinician or a group want to include in order to be willing to act on the findings of the audit? Will a clinical group want to generalise the findings of an audit from a sample of cases to a population? If a clinical group wants to draw inferences from a sample to a population, the sample has to be representative, that is, drawn using a representative sampling technique, and sufficiently large to enable the clinician or the group to be confident in the trueness of the data. Use a statistical formula to determine what a sample size should be. Then when the audit findings are collated, the clinical group is able to state how sure it is that the true population value falls within a confidence interval. For example, for an audit finding of 84% compliance with a clinical audit standard, using a sample size sufficient for a 95% level of confidence and a 5% range of accuracy, you could say, I am 95% sure that the true value is 84%±5% or that the true value lies between 79% and 89%. In other words, you can say that you are 95% confident that the compliance with the audit standard in the entire population would be between 79% and 89%. See the table in the appendix to decide on the number of cases for an audit when a clinical group wants to be reasonably confident that the audit findings from a sample can be generalised to a population. The formulas for determining the number of cases needed for different confidence levels for any population are also in the appendix. 4.5 What to do if cases selected for the audit don t work out The protocol for the clinical audit should instruct the person retrieving cases for inclusion in the clinical audit about how to handle all of the following: missing cases from the list of cases intended for inclusion in a clinical audit cases needed for a clinical audit cannot be made available for data abstraction in the time available for data collection for the audit cases in the list intended for the clinical audit that do not meet the inclusion description the needed number of cases cannot be achieved. Clinical audit staff sometimes add extra cases to those to be retrieved for data collection to deal with problems related to finding cases. However, this approach may produce biased findings for an audit. For example, if an audit includes only cases for which paper patient records could be retrieved on the first attempt, there is the possibility that the patients are not representative of all patients eligible for inclusion in the audit. One or two extra cases may be acceptable to add but adding more extra cases may result in biased findings. Guide to Ensuring Data Quality in Clinical Audits 13 of 46

18 5 How to check on the validity of clinical audit standards 5.1 What validity of clinical audit standards means Validity has been defined in several different ways, depending on whether the activity described as having validity is a research study, an examination or test, or a measurement tool. Clinical audit standards are intended as tools to measure the quality of patient care. The term validity applied to clinical audit standards is defined in the box. 15 The term validity applied to clinical audit standards Validity The extent to which clinical audit standards have the capability to give a true picture of what is being audited. Validity relates to the confidence that clinical staff have that they will draw the right conclusions about the quality of patient care based on the standards used in the audit. Validity is about the relevance of the standards being used in the audit in relation to the objective(s) of the audit. Four types of validity theoretically could apply to clinical audit standards, which are: content face criterion-related construct. The terms, as they apply to clinical audit standards, are defined in the box. 15 Types of validity and their meanings Content validity The clinical audit standards selected include all the key aspects of clinical practice that relate to the objective(s) of the audit. Content validity of clinical audit standards can be demonstrated by showing that the standards cover all the key aspects of quality for the audit objective and don t omit any key aspects that are capable of measurement. Face validity The clinical audit standards relate to the aspect(s) of quality in the audit objective in the opinion of the clinical group members, that is, on the face of it clinical staff think there is a direct relationship between the clinical audit objective(s) and the audit standards. 24 Face validity also includes consideration of whether or not the audit standards are good measures of the aspects of quality that the audit is about as stated in the objective(s). Face validity can be very closely related to content validity where there is an evidence base available to identify standards for a clinical audit and when clinicians are familiar with and believe in the evidence base. 14 of 46 Guide to Ensuring Data Quality in Clinical Audits

19 Types of validity and their meanings Criterion-related validity The correlation between a result or outcome of an aspect of the quality of patient care and specific clinical audit standards that are believed to represent that aspect of quality. Criterion-related validity of clinical audit standards can be demonstrated by correlating outcomes of care with clinical audit standards that are considered to provide a direct measure of an aspect of the quality of care. Criterion-related validity can be predictive or concurrent. Predictive validity indicates the extent to which a future level of performance on outcomes can be predicted from prior or current performance. Concurrent validity indicates the extent to which outcomes estimate present performance in relation to the standards. Predictive or concurrent criterion-related validity underpins tests that assess an individual s suitability for a job, for example. Construct validity When a clinical group is interested in measuring an aspect of quality that is not easy to define operationally and therefore measure, such as quality of life, the attributes that are thought to be involved in that aspect of quality are identified and measured. Construct validity is demonstrated by using a tool that measures each of the individual attributes and then measuring the degree to which the individual attributes identified account for overall results. Examples of how the types of validity could apply to clinical audit standards are in the box. Examples of types of validity applied to a clinical audit on the effectiveness of the use and management of chest drains Content validity The British Thoracic Society guidelines for the insertion of a chest drain describe the following aspects of clinical practice that relate to the effective process of inserting and managing a chest drain: 16 getting the patient s written consent assessing the need for antibiotic prophylaxis and prescribing accordingly having the right equipment available selecting the size of the drain giving premedication, particularly analgesia positioning the patient confirming the site of drain insertion using image guidance to insert a drain for fluid using aseptic technique providing local anaesthetic inserting the chest tube securing the drain managing the drainage system managing the chest drain. There is a detailed list of equipment and materials that are required to insert a chest drain. A clinical group may agree that it is unlikely that there will be any record kept of the availability of all the items on the list for the insertion of a chest drain for an individual patient. Guide to Ensuring Data Quality in Clinical Audits 15 of 46

20 Examples of types of validity applied to a clinical audit on the effectiveness of the use and management of chest drains The standards for a clinical audit on the insertion and management of a chest drain could be said to have content validity if there are audit standards for each one of the aspects of inserting and managing a chest drain, except for the presence of the equipment listed in the guideline. All the other aspects of inserting and managing a chest drain are needed to ensure that chest drains are being inserted effectively, which is the objective of the clinical audit. Face validity Criterion-related validity Suppose a clinical group was interested in the safety of chest drain insertion. A clinical audit standard stating that the chest drain was inserted by a doctor who has completed formal training in chest drain insertion could be said to have face validity. A clinical group may want to know if patients whose drains are inserted and monitored according to the guidelines were also the patients who did not develop complications or infections. Suppose a group decided to collect data on the outcomes of the patients who have chest drains inserted. The clinical audit standards on the effectiveness of insertion and management of a chest drain could be said to have criterion-related validity if there was a strong positive correlation between compliance with these standards and the absence of complications and infections. The criterion-related validity could be predictive if the group wished to assert that future positive outcomes are associated with the clinical audit standards on the effectiveness of the use and management of chest drains. The criterion-related validity could be concurrent if the audit showed a correlation between compliance with the effectiveness standards and outcomes related to complications and infections in the same patient group. Construct validity Suppose a clinical group wanted to extend the objectives of the clinical audit to include satisfaction of patients who have chest drains inserted. The group would have to identify attributes of patient satisfaction, for example, relating to pain relief, explanation of the need for the chest drain and the procedure by clinical staff, doctor and nurse courtesy, relief of symptoms, and so on. The standards for a clinical audit on patient satisfaction for patients having a chest drain could be said to have construct validity if each of the attributes of patient satisfaction were strongly positively correlated with the overall results of patient satisfaction measurement. 16 of 46 Guide to Ensuring Data Quality in Clinical Audits

21 5.2 How to test the validity of clinical audit standards If a clinician or a group is concerned about the validity of clinical audit standards, any of the types of validity described in the previous section can be tested, using accepted statistical tests developed for this purpose. Practical ways to check content and face validity of proposed clinical audit standards are in the box. 15 Ways to check content and face validity of proposed clinical audit standards Content validity Compare each proposed clinical audit standard with the evidence base related to the subject and the objective(s) of the clinical audit. Check the following: Is each standard completely consistent with the wording in the relevant evidence base and consistent with the objective(s) of the audit? Is there a standard for each important aspect of care referred to in the evidence, if it is feasible to measure compliance with the standard? Are there complete definitions for words or terms used in the audit standards? Are there complete and unambiguous directions for how to decide if each standard is complied with or not and how to record decisions? Call to the clinical group s attention any standard for which the answer to the questions above is not unequivocally yes. Ask for clarification on the clinical audit standards for which the answer is not yes. Face validity Submit the clinical audit standards to clinical staff that know about the audit subject. Ask the clinical staff if the standards appear on the face of it to be true measures of what the clinician or the group is interested in. You can give the people who are asked to judge face validity a questionnaire with yes no questions or a rating scale for each standard. You can collate and analyse the responses to determine the number or percentage of staff that agreed that the standards are valid and the findings of a clinical audit using the standards can be acted on accordingly. 5.3 The efficiency of clinical audit standards in identifying good and not so good quality of care Clinical staff members sometimes observe that clinical audit standards don t screen or filter cases included in the audit in order to identify the occasions in which patient care should or could have been better. On other occasions, clinical groups can be concerned that clinical audit standards are identifying a large number of cases in which the quality of care has not been consistent with best practice. They may question if the standards used in the audit were appropriate for measuring the quality of care. In either of these circumstances, a clinical group could be assured that the clinical audit standards are sensitive and specific. These terms as they apply to clinical audit standards are defined in the box on the next page. 15 Guide to Ensuring Data Quality in Clinical Audits 17 of 46

22 Terms related to clinical audit standards and their meanings Sensitivity The likelihood that a case will be identified, through data collection using a clinical audit standard, as representing poor care and the case really is poor care A clinical audit standard is sensitive if it flags all or almost all cases in the audit for which there is a problem about the quality of care provided and doesn t miss cases in which care was poor. Specificity The likelihood that truly good care will be identified, through data collection using a clinical audit standard, that is, that a case identified as representing good care really is good care A clinical audit standard is specific if it doesn t flag cases or flags few cases for review when the care provided is clinically acceptable. 5.4 How to test if clinical audit standards are sensitive and specific The process for testing the sensitivity and specificity of a clinical audit standard is in the box. 15 How to test sensitivity and specificity of a clinical audit standard 1. Carry out data collection in accordance with the clinical audit standard and related definitions and instructions for data collection. 2. For any cases that are found not to be consistent with the audit standard, ask one or more clinicians to review the cases and make a decision on whether or not the case represents acceptable or unacceptable quality. 3. Ask one or more clinicians who have not been involved in the case review to review all of the cases that have already been screened according to the audit standard and make decisions independently (without knowing the results of screening against the audit standard) about whether or not the cases represent acceptable quality. 4. Compare the cases flagged by the audit standard and the cases that did not represent acceptable quality as judged by clinicians. 5. Display the figures in a table to show the sensitivity and specificity of the audit standard, such as the one in the box on the next page. 6. Draw conclusions about the sensitivity and specificity of an audit standard as follows. A clinical audit standard is a sensitive measure if it identifies most of the cases that represent a problem about quality and perhaps a few that did not (true-positive cases and false-negative cases). A clinical audit standard is a specific measure if it identifies most of the cases that did not represent a problem about quality and perhaps a few that did (true-negative cases and false-positive cases). 18 of 46 Guide to Ensuring Data Quality in Clinical Audits