Data quality checks performed on SUS and HES data

Transcription

1 Data quality checks performed on SUS and HES data Author: HES Data Quality Team Date: 24 th February Copyright 2013, Health and Social Care Information Centre.

2 Version Control Version Date Author Reason for Change /02/2014 HES DQ Team First release /08/2015 HES DQ Team The role of system integrator from BT has moved to the HSCIC. 2 Copyright 2013, Health and Social Care Information Centre.

3 Contents Contents 3 Pre-submission checks 4 Commissioning Data Set 4 CDS Data Dictionary 4 Figure 1 Overview of the processing and data quality checks undertaken on SUS, PbR and HES data 5 On submission 6 XML Validation 6 SUS Business Rules 6 Post submission SUS Processing 7 SUS Tracker 7 SUS Data Quality Report (DQR) 7 Payment by Results (PbR) 9 Post submission HSCIC checks 10 Payment by Results (PbR) data 10 Data quality checks performed on PbR data include: 10 Assurance of SUS release 12 SUS Data Quality Dashboards and KPIs 12 Hospital Episode Statistics (HES) Data Quality 12 Overview of HES Process 12 Figure 2 Overview of the data flows from healthcare providers into the HES dataset 13 Data quality processes for the HES Data 13 Data quality during the processing of the HES dataset 13 Provider Mapping 13 Removal of Duplicate Data 14 Automatic data cleaning and derivation rules 14 Post-processing data quality checks 15 Data Coverage 15 Review of Fields 15 Other data quality checks 15 Data Quality Reporting 16 3 Copyright 2013, Health and Social Care Information Centre.

4 This document aims to summarise and catalogue all data validation, cleans and assurance checks applied to Secondary Uses Service data and subsequently Hospital Episodes Statistics (HES) and Payment by Results (PbR) data. Figure 1 provides an overview of the data validation and assurance checks carried out on the data and when in the process these checks are undertaken. Pre-submission checks The Health and Social Care Information Centre (HSCIC) prescribes a uniform data structure that is to be used by providers submitting data into the central data repository: Secondary Uses Service (SUS). These tools allow for consistency and improved quality of data. These tools are: Commissioning Data Set The Commissioning Data Set (CDS) is the Information Standard Board approved structure used for the submission of commissioning data into SUS and is designed to be capable of individually conveying many different CDS structures including amongst others: Accident and Emergency (A&E) Attendances; Outpatient (OP) Attendances; Future Attendances; Admitted Patient Care (APC) and Elective Admission List data. The primary purpose of national data sets is to enable conformant health information to be generated across the country, independent of the organisation or system that maintains it. In achieving this, the HSCIC enables healthcare professionals to measure and compare the delivery and quality of care provided and to support them in sharing information with other health professionals and organisations. Further details about the Standardisation Committee for Care Information can be found on this link: CDS Data Dictionary In addition to defining the CDS structure, the HSCIC also describes the permissible/ preferred values that should be submitted for each CDS attribute. The NHS Data Dictionary is there to help maintain consistency and quality of data. The data dictionary is available on this link: _menu.asp 4 Copyright 2013, Health and Social Care Information Centre.

5 Figure 1 Overview of the processing and data quality checks undertaken on SUS, PbR and HES data Acute Providers Commissioning Datasets APC Outpatient A&E Undefined Format XML XML Brokers Translation into XML EDT Client Check Data Rejection PbR Managed Service Extract HSCIC HES Extract HES DQ SUS Ops PbR SUS Users PbR DQ Landing SUS Business Rules Staging Data Warehouse Providers Commissioners Tracker SUS DQ Data Rejection Reference Data/PDS SEM Role Based Access Control Processing and Validation Data Marts 5 Copyright 2013, Health and Social Care Information Centre.

6 On submission XML Validation CDS data is submitted into SUS in a XML schema form and as part of the submission process there are a number of XML validation checks applied. If a submission fails one or more of these checks then the whole set of data is rejected. Data only enters SUS if all XML validation rules are met. Examples of the types of checks applied at this submission stage are: Data items to not contain values that exceed the prescribed limit of characters. For common and more mature data items, ensure the values are those that are prescribed in the NHS CDS Data Dictionary. Ensure date fields only include date values. Check that mandatory data items are populated. Further details of the SUS schema are available on the data dictionary website: The latest version of the SUS schema (version 6.2) is available on this link: SUS Business Rules After the submitted data has passed the XML validation checks, some data elements are subject to additional business rules. There are 14 business rules applied, including: S13 Data Elements reporting a DATE (which is not Date of Birth) must use dates between 01/01/1900 and 31/12/2999 in order to pass validation. S14 For Data Elements reporting a TIME, the hour portion must be between 00 and 23 inclusive in order to pass validation. Details of each business rule can be found on the NHS Data Dictionary: business_rules.asp?shownav=1 If the submitted data passes the XML validation checks, but fails one or more of the business rules, this will be shown in SUS Tracker (see below) with a status of rejected. A message is sent to the provider directly or through XML middleware supplier to inform them that their submission failed one of the business rules providing details of the reason for the rejection. 6 Copyright 2013, Health and Social Care Information Centre.

7 Post submission SUS Processing SUS Tracker Once the data has passed through initial XML validation, the SUS Tracker can then be accessed via the SUS Portal. It provides details of the live status of interchanges into SUS and whether they have been processed and made available in the SUS Extract Mart (SEM) and PbR marts. Users can then drill down into these interchanges and view Data Quality Reports (DQR) for the data within them. SUS Data Quality Report (DQR) Within SUS Tracker, SUS performs a number of data quality checks for each interchange. These are performed during the load staging processing of SUS. No data is changed or rejected as these checks are for information only. The results of these checks are presented as Data Quality Reports, displayed as a percentage of missing/invalid/other giving users the opportunity to investigate and resubmit data if they wish. There are a number of DQR rules and the three specific checks applied for each rule are: Missing field is missing when expected to be populated Invalid the field has passed XML validation (to make it into DQR), but is still not valid. Note that not every field is checked for validity in XML, some are only checked on format. E.g. an historic Commissioner Code passes XML but will be flagged in the DQR. Other the field is not invalid but there is something unusual noting about it. E.g. NHS Number where all the digits are the same ( , ) Users also have the ability to drill down at record level for a particular rule. 7 Copyright 2013, Health and Social Care Information Centre.

8 Examples of the DQR rules used for performing these checks are: Rule Data Item Dataset Description Other Invalid Missing E4* Ethnic Category E5 E1 Primary Diagnosis Commissioner Code APC APC APC The first character present of the ethnic category is tested in each record. The first four characters of Primary Diagnosis code in each record excluding Home Delivery and Home Birth records is tested against the ICD 10 codes table Tested against the Organisation codes table. E7 Postcode APC The Postcode in each record is tested Records where ethnic category is equal to 99 or Z (not stated) Records with a Primary Diagnosis code equal to R69? (where? = any character ) Records with a Commissioner code that could be matched on the organisation code table but where at the time of the CDS record the organisation code was determined to be closed. Records that have a valid postcode structure but cannot be matched on the postcode file Records that have a correctly formatted ethnic category that does not equal A-H, J-N, P, R, S, Z or 99 Records with first four characters of Primary Diagnosis code (not equal to R69?) that could not be matched on the ICD 10 code tables Records with a Commissioner code that could not be matched on the organisation code tables or could be matched but was dated before the organisation start date Records with a Postcode that could not be matched on the Postcode tables (excluding Other ) Records where Ethnic Category is missing Records with no Primary Diagnosis code Records with no Commissioner code Records with no Postcode The full list of Data Quality Rules is available on the HSCIC website: 8 Copyright 2013, Health and Social Care Information Centre.

9 Payment by Results (PbR) There are specific business rules that check the validity and quality of data that is submitted into SUS and used for PbR. Failing these checks can lead to activity not being tariffed under PbR policy. A spell of activity can result in a non- tariff if one of the following data quality rules are not met: Additional information on data quality checks and error codes specific to PBR data are available on the HSCIC website: Guide/pdf/R11_SUS_PbR_Indicators_Errors_Data_Quality_Guide_v1_2.pdf 9 Copyright 2013, Health and Social Care Information Centre.

10 Post submission HSCIC checks The HSCIC receives both a SUS PbR Extract and a HES extract sourced from SEM. Both of which are processed separately by the HSCIC for data quality. Payment by Results (PbR) data SUS PbR data is loaded into a staging database and then a number of data quality checks are performed before the data is loaded into production databases for APC, OP and AE. Data quality checks performed on PbR data include: Completeness check: The first assurance check counts the number of NULL values within each data item. Volumes of NULL values are then compared against the previous three months and the average of the last 12 months to identify any unexpected variations that may indicate issues with the data. An unexpected quantity of NULLs in any of the fields is investigated. The following PbR data sets are checked for coverage: Admitted Patient Care Spells, Admitted Patient Care Episodes, Outpatients, Accident & Emergency, Multi Births, Readmissions, Critical Care, Best Practice Cataracts and Ward Stay. Coverage and validity: For a small number of commonly used data items, a test is performed looking at the specific values submitted, highlighting any new or invalid values. This test also looks at the distribution of activity for each valid value comparing the counts to the last three months and the average of the previous 12 months identifying any unexpected variation. Coverage and validity checks are performed only for a small number of fields within each of the following extracts: Admitted Patient Care Spells, Admitted Patient Care Episodes, Outpatients and Accident & Emergency. Provider Submissions: A report showing a breakdown of activity by provider and by month is produced to help identify any significant changes in provider submission volumes, highlighting potential duplicate records and/or missing data. Admitted Patient Care Episodes, Outpatients and Accident & Emergency data sets are analysed at provider level. Application of default codes: SUS PbR data is unchanged from how it is supplied to the HSCIC, except a few minor cleans applied. These are: Data item Admission Method in the Admitted Patient Care data set has contained small volumes of records with the value ##, this has be replaced to the NHS data dictionary default code of 99, for consistency and ease of use. HSCIC also provide an additional qualifying flag to add value for analysis purposes. The Excel spread sheets which record the results above are currently for internal use only but wider dissemination is under active consideration as we do recognise this information could be of significant value to wider stakeholders. 10 Copyright 2013, Health and Social Care Information Centre.

11 FEB 1112 MAR 1112 APR 1213 MAY 1213 JUN 1213 JUL 1213 AUG 1213 SEP 1213 OCT 1213 NOV 1213 DEC 1213 JAN 1213 FEB 1213 MAR 1213 APR 1314 Data quality checks performed on SUS and HES data Some examples from the Excel spread sheets are shown below: Top 10 of up shift in providers supplied records between March and April Sum of % Shift Provider Total Previous Month Current Month RCB 45.69% RBV 10.56% RY8 9.60% RR1 8.94% RA3 6.66% RLT 6.63% R1F 5.75% RPC 5.16% RXH 4.87% RCF 4.63% RQX 4.61% RD1 4.26% RC1 3.35% RM3 3.09% RAP 3.06% NOTE: This summary excludes independent sector providers. Count of episodes over time for RCB: The upshift in episodes for provider RCB (York) in April was caused by the provider merging with RCC (Scarborough) and all episodes submitted under RCB for April whereas they were split between RCB and RCC prior to April. 11 Copyright 2013, Health and Social Care Information Centre.

12 Assurance of SUS release Further to the data quality and validation checks, each new SUS release is complemented with an assurance exercise, which ensures any new or amended functionality is delivered as expected and that there is no adverse impact on exiting data or data processing. SUS Data Quality Dashboards and KPIs SEM is the source for HES data and before it is cleaned for HES, this data is used to provide two sets of tools with which to measure and monitor data quality: the Key Performance Indicator (KPI) Reports and Data Quality Dashboards (DQD). These tools are available to help monitor and drive improvements in the quality and completeness of SUS data. These tools are published at provider level and allow organisations to assess their own data in SUS to ensure that it is comprehensive and compliant with data standards. There are four KPI reports showing information such as Comprehensive coding (operation and diagnosis codes); Trusts submitting data using the Net and Bulk Protocol; Referral to treatment (RTT) data quality and high level PBR Indicators. There are also four DQDs which report on the coverage and quality of a number of key fields for APC, Outpatient, A&E, Critical Care and Maternity data. Overall coverage such as activity levels and number of duplicate records are also reported. There are currently over 1500 users registered with the SUS DQDs. Hospital Episode Statistics (HES) Data Quality Overview of HES Process Each month, at pre-arranged dates during the year, SUS takes an extract from the SEM database and sends it to HES. Data on SUS will continue to change, with providers able to update or replace data, but HES data is fixed as it was when that particular extract was taken. This is why there are likely to be differences between analyses on SUS data and those on HES data. Each monthly HES extract contains provisional year to date data that providers can update in later submissions to SUS. The Health and Social Care Information Centre (HSCIC) then validates and cleans the extract, before deriving new items and making the information available in the data warehouse. Data quality reports and checks are completed at various stages in the cleaning and processing cycle. 12 Copyright 2013, Health and Social Care Information Centre.

13 Figure 2 Overview of the data flows from healthcare providers into the HES dataset Data quality processes for the HES Data Data quality during the processing of the HES dataset The data quality team cleans the data, derives new variables and prepares the HES Output Schema for analysis and publication. Details of the main types of cleaning rules applied to the HES data are detailed below. Provider Mapping Reference data based on the information on Organisation Data Services (ODS) is used to update old or invalid provider organisation codes to the correct provider code. Where no appropriate provider code can be found, this data is deleted. For example: Provider A merged with Provider B to form Provider C in When data is submitted for Provider A or B after 2011, these are mapped to Provider C. The data quality team will automatically map data to the most appropriate code in the monthly provisional HES datasets and make providers aware that this work has been undertaken; giving them an opportunity to correct their submissions in future months. Before the final end of year HES dataset is produced, the data quality team will seek the providers 13 Copyright 2013, Health and Social Care Information Centre.

14 permission to apply this change to the data. If a provider does not give consent for the HSCIC to change the provider codes, the data is deleted from the end of year dataset. Removal of Duplicate Data Procedures are run each month to identify records that have been submitted multiple times by a provider. Duplicates are identified by looking for matching data in a number of key fields and when identified duplicate records are automatically deleted from the monthly provisional data. Details of all deleted records are shared with trusts so that if necessary they can request further data deletions from SUS to help ensure SUS remains accurate. Once the duplicated records have been removed from SUS, they will also be removed from HES when the next data extract is received. Before the final end of year HES dataset is produced the data quality team will contact all providers who continue to have duplicated data. If the providers give consent, the duplicates are removed from the dataset before publication of the end of year dataset. If a provider does not give consent for the data to be removed, the duplicated data will remain within the dataset. Automatic data cleaning and derivation rules Data cleaning rules for HES have been developed over time and continue to evolve to enhance the dataset. These rules have three main purposes: 1. To clean common and obvious data quality errors. 2. To derive additional data items to populate the HES dataset. 3. Remove activity data outside the relevant date range for the current HES extract. At present there are 120 rules applied to the APC dataset, 57 rules applied to the OP dataset and 43rules applied to the A&E dataset. The types of data cleaning that is undertaken include: When sex is submitted as M or F change to 1 or 2; when sex is invalid change to 0. Set dates of birth before 1/1/1885 as invalid. Update invalid ethnicity codes to X. Ensure that date and time formats are correct and if invalid set a default value. Example of a correction rule: Rule #0150 looks for evidence where a Birth Episode (CDS type 120) has been incorrectly submitted to SUS as a General Episode (CDS type 130). If evidence is found then the Episode Type of the record is altered to reflect this. Without this clean, the number of birth records (Episode Type 3) in HES would tend to be lower than the actual number of births taking place. 14 Copyright 2013, Health and Social Care Information Centre.

15 Example of a validation rule: Rule #0090 validates Main Speciality codes. If the main specialty code is null or contains an invalid entry, it is overwritten with the appropriate value for not known &. The clean makes it easy to filter out values that are not known from any analysis for this field. Without the clean the user would need to either list all the recognised values or search for all the different incorrect values that were submitted. Full details of all the cleaning rules applied to the datasets are provided on the HSCIC website: Post-processing data quality checks The data quality team carries out a number of checks on the HES dataset to ensure that it is fit for purpose, using both formal checks that are undertaken each month and AdHoc analyses to investigate specific issues. For example, we ensure that the total amount of data in each dataset falls within expected ranges. We also ensure that we are receiving data from the expected number of providers, both NHS and Independent providers. We also look at the number and percentage of records submitted by NHS and Independent providers to ensure they fall within expected ranges. Data Coverage We track providers with missing of data each month as well as those with increases or decreases in the amount of data submitted compared with the amount of data submitted in previous months and in the same period in the previous year. Providers are informed of any coverage issues identified and are requested to review their SUS submissions and update their data as appropriate. Details of any data omissions are published within the monthly data quality note. Review of Fields We analyse the population and coverage of all the fields submitted to HES each month, but at present we do not analyse any of the derived fields. Currently we analyse 247 fields in the APC dataset, 101 fields in the OP dataset and 98 fields in the A&E dataset. The aim is to highlight any issues in the population of fields at a national level, rather than to validate or quality assure the data items that have been submitted within the fields. Time series analyses are used to check for unexpected changes in field completion and to monitor changes over time. Other data quality checks The team undertakes AdHoc data quality checks and investigations based on information supplied by providers or from feedback from HES data users. Any data quality issues that we find are published in the data quality notes each month. The data quality team will then work with the providers in question to make them aware of any issues and help them to resolve any problems they are experiencing. 15 Copyright 2013, Health and Social Care Information Centre.

16 Data Quality Reporting Data quality information for each year to date HES dataset is published each month in the HES data quality notes for each CDS. These are published at the same time as the provisional year to date HES data. Examples of the data quality notes are available on this link: al+monthly+hospital+episode+statistics%22&sort=relevance&size=10&page=1#top Details of data quality issues that relate to more than one month of provisional HES data are published on the Processing cycle and HES data quality page of the HSCIC website: 16 Copyright 2013, Health and Social Care Information Centre.