SUS Data Quality Dashboards Survey September 2012

Transcription

1 File Name: Document Reference No: Version: 1.0 Status: Final Issue Date: January 2013 Author: J. Clough SUS Data Quality Dashboards Survey September 2012 For more information on the status of this document, please contact: HSCIC Contact Centre Tel: Fax: Internet: Date of Issue Reference

2 Author: J. Clough Directorate: Version: 1.0 Status: Final Date: January 2013 DOCUMENT MANAGEMENT REVISION HISTORY Version Date Summary of Changes APPROVED BY Name Signature Title Date of Issue Version REVIEW DETAILS Review Date: Reviewer Version: 1.0 Page ii

3 CONTENTS 1 INTRODUCTION PURPOSE OF DOCUMENT SCOPE OF DOCUMENT BACKGROUND AND METHOD ANALYSIS OF RESULTS RESPONDENT DEMOGRAPHICS USEFULNESS OF THE SUS DATA QUALITY DASHBOARDS DESIGN AND USABILITY OF THE SUS DATA QUALITY DASHBOARDS HELP AND EXTRA INFORMATION ON THE SUS DATA QUALITY DASHBOARDS SPECIFIC INDICATORS ON THE SUS DATA QUALITY DASHBOARDS BENCHMARKING ON THE SUS DATA QUALITY DASHBOARDS OUT OF SCOPE APPENDIX 1 SURVEY RESULTS RESPONDENT DEMOGRAPHICS USEFULNESS OF THE SUS DASHBOARDS DESIGN AND USABILITY OF THE SUS DASHBOARDS HELP AND EXTRA INFORMATION ON THE SUS DASHBOARDS... 9 Version: 1.0 Page iii

4 1 INTRODUCTION 1.1 Purpose of Document The purpose of this document is to provide a summary of the feedback received from the recent SUS Data Quality Dashboard Survey together with an agreed action plan of improvements to address some of the feedback raised. 1.2 Scope of Document This document addresses key issues raised by the SUS Data Quality Dashboard Survey. Completion of some of the improvements identified in this document may be dependent on the availability of funds and resource within the programme. This document should, therefore, be read as a statement of intent. 1.3 Background and Method The HSCIC Data Quality and Operational Support team develop, maintain and produce the monthly SUS Data Quality Dashboards. These Dashboards are intended to help Senders of data into SUS, to report and continue to improve data quality for their organisation(s). This team conducted a survey around the SUS Data Quality (DQ) Dashboards between 29 August 2012 and 21 September The purpose of the survey was to obtain views from users of the SUS DQ Dashboards, especially around the new functionality that was added at the start of the 2012/13 financial year. Invitations to complete the survey were issued to registered users for the SUS DQ Dashboards. The survey covered 4 main topic areas and the questions were split between: closed questions asking stakeholders to rate specific elements of the dashboards and open questions inviting respondents to provide textual comments regarding the dashboards The responses to the Closed Questions are reproduced in Appendix 1 of this document. Where Stakeholders felt that a question was not relevant, they were given the option to not answer that question. This explains the variable number of respondents to each question. A manual process was undertaken to review all of the open questions. The feedback for each topic is summarised in the next section together with intended actions and areas of activity to improve the stakeholder experience. Version: 1.0 Page 1

5 2 ANALYSIS OF RESULTS 2.1 Respondent Demographics A total of 125 stakeholders responded to the survey. Over 70% of respondents were from NHS Provider organisations, followed by 11% of respondents who were from NHS Commissioner organisations. Respondents commented that the Dashboards are easy to use, and that the wording around each indicator is useful for providing a quick look at targeting areas of concern. Respondents also commented that it is useful to have the raw data available and that the Dashboards are being used in local DQ meetings. 2.2 Usefulness of the SUS Data Quality Dashboards 77% of respondents found the Main SUS DQD (Data Quality Dashboard) either very useful or frequently useful out of those who used it (i.e. excluding the 8 respondents who do not use it). 51% of respondents found the SUS Critical Care DQD either very useful or frequently useful out of those who used it (i.e. excluding the 36 respondents who do not use it). 58% of respondents found the SUS Maternity DQD either very useful or frequently useful out of those who used it (i.e. excluding the 38 respondents who do not use it). 63% of respondents found the SUS Coverage DQD either very useful or frequently useful out of those who used it (i.e. excluding the 24 respondents who do not use it). Respondents queries and comments: Why is the data 2 months old by the time it is published? The deadline (inclusion date) for users to submit CDS data into SUS is 3 weeks after the end of CDS activity month. This data is then processed by BT who sends the data to our HES (Hospital Episodes Statistics) supplier. The Data Quality Team at the HSCIC receives the data early in the following month and standard checks are run on the data as soon as it is received. After this point the data is made available for processing the Dashboards. It takes between one and two weeks from receiving the data at the HSCIC to publishing the first two Dashboards. The other Dashboards are published slightly later to stagger the work over the month. 2.3 Design and Usability of the SUS Data Quality Dashboards 90% of respondents found the provider comparison within each SHA either very useful or of some use. 97% of respondents found the time series chart (main DQ Dashboard only) either very useful or of some use out of those who used it (i.e. excluding the 7 respondents who do not use it). 90% of respondents found the changes v s previous month indicator arrow symbols (main DQ Dashboard only) either very useful or of some use out of those who used it (i.e. excluding the 9 respondents who do not use it). Version: 1.0 Page 2

6 Respondents queries and comments: Why has the dial been replaced by the % Change traffic lights? This enables users to see the RAG traffic lights for all fields together and free up room for the more info on data item text box. Why does the Time Series graph start from April? There is no point in showing future months as they re all zero. We have taken this comment on board and are planning to change this graph for 2013/14 to display a rolling 12 month period, with the latest month being the last bar. The Time Series percentage doesn t change as we are usually in the high 90s. The y-axis percentage is dynamic and as such its scale changes according to the values. The previous resolution will fix this because the y-axis will start from the lowest validation percentage instead of from 0 (corresponding to the empty future months which will no longer be present). It would be good if the printing of these dashboards was simpler. I convert them to PDFs in order to do so currently. To ensure that the Dashboards print in the correct format, first go to 'File > Page Setup...' and select the Landscape orientation, and then Print. I have to scroll down to see all of the data items. It s frustrating that we can t see all the items in one view. On some occasions, particularly where the dashboards are viewed using laptop computers, re-sizing of tables may occur, which means that a user must navigate using scroll bars to view all the available data (see Fig 1). The dashboards are best viewed using a screen resolution of 1024x768 or higher, and with no more than 2 toolbars visible within your internet browser (see Fig 2). This guidance is available within the help button on the top left of each DQD and also from page 3 of the SUS DQD Frequently Asked Questions. Version: 1.0 Page 3

7 I seem to have difficulty moving around the dashboards as there is not a return to the main home page button. When opening the Dashboard directly from the Sharepoint site users can return to the dashboard home page by clicking on the back button in their web browser (see below). If users close the Dashboard this will close the whole Sharepoint page. 2.4 Help and Extra Information on the SUS Data Quality Dashboards 97% of respondents found the more info for data item text box either very useful or of some use. Respondents queries and comments: It would be useful to have a direct link to the appropriate page of the data dictionary where applicable. We agree with this suggestion and are in the process of adding active hyperlinks to guidance and the NHS data dictionary. It would be much better if you could provide the logical measure description. We also agree that more detailed logical explanations would help users understand the validation derivations. We are looking into adding these for the 2013/14 Dashboards. 2.5 Specific Indicators on the SUS Data Quality Dashboards There were no comments or queries around any of the specific indicators on the SUS Data Quality Dashboards. 2.6 Benchmarking on the SUS Data Quality Dashboards Respondents queries and comments: It would be useful to have a filter that allows you to select specific treatment function codes, admission methods and patient class which would help identify specific areas within the hospital. Users are unable to filter directly from the Dashboard. We have investigated this but the software is unable to take the scope of variations to the data. The dashboard is an initial tool to highlight any data quality validation issues. We encourage users to use these high level results to investigate any trends and filters locally. It would be helpful if more of the data items covered by the IG Toolkit were included in the DQ Dashboard for the APC and OP data sets. We are working with the Department of Health in the hope of amending/adding validation to the Dashboards to match with the IG Toolkit requirement 507 and vice versa. Our intention is that users would directly be able to obtain their validation percentages directly from the DQ Dashboard. However we cannot guarantee that will happen, nor when this will happen. Version: 1.0 Page 4

8 It would be helpful to be able to view best performing trusts and run peer comparisons We agree that the ability to do peer comparisons would be useful. Unfortunately it is not possible to identity the appropriate groups for the Dashboards. In addition we are unable to create this as a totally dynamic attribute (i.e. users to select any selection of organisations to compare against) with the software available. 2.7 Out of Scope Respondents queries and comments: It would be helpful if the Elective Admissions List (EAL) data set could be included. EAL is an optional CDS Type and whilst coverage has increased over the last couple of years, it is still low. We would consider reporting on EAL if this was to become mandated to flow. But since it is not a nationally mandated CDS Type it is not a priority. We are also working with the Department of Health in the hope of removing this CDS type from the IG Toolkit requirement 507. Some specific Mental Health (MH) indicators would be useful. E.g. cluster, care programme and other fields relevant to MHMDS. Dashboards report on SUS data only. MHMDS is not part of SUS, as such this data is not included in the SUS DQ Dashboards. However guidance is available from the Mental Health Minimum Data Set webpage on the HSCIC website It would be useful to be able to drill down to patient/record level. This is something we recognise that users would find very useful. The Dashboard software limitations don t make this currently possible. However we are capturing this in requirements for the SUS replacement ODP (Open Data Platform) project for Version: 1.0 Page 5

9 3 APPENDIX 1 SURVEY RESULTS 3.1 Respondent Demographics Figure 1: What type of organisation do you represent? 3.2 Usefulness of the SUS Dashboards Figure 2: What best describes your opinion of the new set of SUS dashboards as a data quality aid? Version: 1.0 Page 6

10 3.3 Design and Usability of the SUS Dashboards Figure 3: If you used the dashboards prior to 2012/13, please indicate how you feel the new SUS design and functionality has affected your overall user experience Figure 4: Front screen tab Provider comparison within SHA Version: 1.0 Page 7

11 Figure 5: Individual CDS tabs Time series chart (main DQ dashboard only) Figure 6: Individual CDS tabs Change v s previous month indicator (arrow symbols) on data validity tables (main DQ dashboard only) Version: 1.0 Page 8

12 Figure 7: Individual CDS tabs Comparison with national average (circular symbols) on data validity tables) 3.4 Help and Extra Information on the SUS Dashboards Figure 8: Individual CDS tabs More info for data item text box Version: 1.0 Page 9