Internal Audit Review of Waiting Times Management A FINAL REPORT

Transcription

1 Internal Audit Review of Waiting Times Management A FINAL REPORT Audit Completed: 7 th vember 2012 Discussed at Audit Committee: 27 th vember 2012 Auditors: Julie Watters - Chief Internal Auditor, NHS Dumfries and Galloway Sandra Abbott - Internal Auditor, NHS Dumfries and Galloway Distribution: John Matheson - Director for Health Finance and Information, Scottish Government

2 CONTENTS Summary of Audit Findings 3 1. Audit Scope and Objectives 3 2. Background and Overview 4 3. Approach National Level Local Level 5 4. Audit Findings Objective Objective Objective Conclusion 9 6. Acknowledgements Glossary of Terms Management Action Plan 11 Appendix 1 Summary of core data queries and findings 17 Page Page 2 of 21

3 Summary of Audit Findings The table below summarises the grades of audit recommendations as they sit against each of the audit objectives. Audit Objective 1. To ensure that individual patient records are accurate and that systems are in place to ensure that the patient management system cannot be inappropriately changed 2. To ensure that reporting on waiting times is accurate and consistent at every level in the organisation up to and including the Board 3. To ensure that the local guidance is consistent with national guidance and that its implementation is both valid and reliable (i.e. not open to different interpretation in use) A Low risk Recommendations B C D Medium High Very High risk risk Risk Level of assurance Significant INTRODUCTION 1. Audit Scope and Objective The audit objectives were set out in the Terms of Reference document from the Scottish Government of 3 May There are three main objectives of this audit, which are to ensure that: 1. Individual patient records are accurate and that systems are in place to ensure that the patient management system cannot be inappropriately changed; 2. Reporting on waiting times is accurate and consistent at every level in the organisation up to and including the Board; and 3. The local guidance is consistent with national guidance and that its implementation is both valid and reliable (i.e. not open to different interpretation in use). These objectives were requested to be achieved through the following audit activities: 1. Undertake a comprehensive review of waiting times reporting to Executive Management, relevant Committees of Governance, the Board and the Scottish Government. This will include tracing the content of these reports back to the waiting times system, and through intermediate systems if relevant. 2. Trace a sample of waiting times data from input, through amendment/updating within systems, to output within the various reports presented to Management, relevant Committees and the Scottish Government, through to publication to ensure consistency through every level of reporting. Page 3 of 21

4 3. Investigate and report any variations, unusual matters or obvious omissions identified in relation to paragraphs 1 and 2 above. 4. Review the Board's local guidance for completeness and consistency with the SGHSCD guidance on waiting times management. In particular, this will include an assessment of accessibility, availability and applicability of that guidance in the waiting times process. 5. Review the systems and process controls that exist and the operation of those controls for data input, processing data through the waiting times system and final reporting, through sample checking. The existing systems, processes and controls should be fully documented to allow a transparent review of documented and actual performance. 6. Assess completeness of recording for New Ways data fields, including reasons for amendments to patient records. Analyse core data to identify key issues including, but not restricted to, trends and adjustments to periods of unavailability and other adjustments of the patient s waiting time clock, making use of all relevant data available including local data and nationally available data from ISD. 7. Interview a sample of staff involved in the waiting times management process at all levels of the organisation, including clinicians, managers and data entry staff, to provide a further dimension to the assessment of data, controls and processes. 2. Background and Overview This audit was undertaken to provide independent assurance that the systems and processes for Waiting Times Management within the Board are sufficient to ensure that all patients waiting times are accurately recorded and reported upon. This request follows an issue identified within another NHS Board in relation to inappropriate use of unavailability codes which have distorted the figures reported to the Scottish Government. Pricewaterhouse Coopers LLP (PwC) undertook a review into Waiting Times management at NHS Lothian, and this was formally reported to the Scottish Government on 19 th March The audit was undertaken in addition to internal reviews of processes by management within the Board and an Audit Scotland review of the use of patient unavailability codes in the management of NHS waiting times. Internal Audit have been mindful of the risk of duplication with other pieces of work but have maintained the achievement of the specific objectives within the scope set out by the Scottish Government as a priority so that the final report to the Scottish Government can be treated as a stand alone document which clearly fulfils the initial request. All key dates in relation to this audit are detailed below. Date December 2007 October 2011 February 2012 tes New Ways waiting times management guidance published Sunday Times report that NHS Lothian patients being offered treatment in England at short notice which they are unlikely to accept PwC were engaged to undertake review of Waiting Times management and practices in NHS Lothian 19 March 2012 PwC report issued to John Connaghan Scottish Government 21 March from Derek Feeley to all NHS Board Chief Executives regarding PwC report on Lothian waiting time management and practices. Page 4 of 21

5 Informed that Boards Internal Audit Plans for 2012/13 will require an audit on Waiting Times management 3 May 2012 Scope of Internal Audit issued to Board Chief Executives 31 May 2012 Initial workshop attended by representatives of all Boards Internal Audit providers, Audit Scotland, Scottish Government Finance and ISD. Decision to procure data analysis support for Internal Audit and Audit Scotland reviews This was subsequently awarded to PwC to commence on 23 rd July August 2012 PwC request data from Boards with a deadline of 14 August September 2012 Data provided to PwC 12 September 2012 Data packs consisting of agreed core query information provided to Internal Audit These were reviewed with the Information Services at which time access to ToPAS was requested and subsequently provided on 22 October vember 2012 Preliminary report issued 22 vember 2012 Management Response received 23 vember 2012 Final report issued with Audit Committee papers 27 vember 2012 Audit Committee 3. Approach 3.1 National Level Throughout the course of this audit there have been frequent workshops and conference calls undertaken to enable us to raise queries with the Scottish Government and other Internal Audit providers. This has ensured that whilst each internal audit will be independent and will represent the issues and findings from the relevant NHS Board, the final reporting to the Scottish Government will have a consistency in approach and format. It was during early discussions between Boards, that the requirement for a consistency in data analysis was identified to ensure that all Boards were considering the same information when looking at changes made to patient records on their Patient Access System (PAS). PwC were procured by the Scottish Government to undertake data analysis on behalf of Boards. Following discussion, 17 core data sets were identified for the internal audit functions to base testing on and to be used to identify information for further sampling. PwC visited each board and identified the information that they would require from our own system ToPAS. This information was requested on 7 th August 2012 and sent back to PwC with full Caldicott approval on 6 th September Following analysis by PwC this information was made available to Internal Audit on 12 th September 2012 at which time data and trend analysis could commence. This data did not include any patient identifiable information and was used for sampling purposes. Once samples were identified three subsequent data requests were made to PwC to obtain CHI numbers from which we could interrogate the system in a more focussed way. This testing was completed once access was provided to the ToPAS system. These timescales have varied for each NHS Board, as there are a number of systems being used and extraction and analysis had to be tailored to the system(s) holding the data. 3.2 Local Level The audit scope was set out by the Scottish Government with very clear objectives and outcomes identified. We have had to tailor our normal audit approach which would have been to scope the audit based on our perception of key risk areas following discussions with management and staff. Page 5 of 21

6 Prior to receipt of the scope we undertook early discussions with colleagues within the Board to identify our own processes for managing waiting times. Early discussions were focussed around the specific system that we use, how the patient journey is recorded within this and how information from the system is reported to the Scottish Government. We have undertaken a full systems review and we have mapped the processes around management of waiting times within NHS Dumfries and Galloway. Following the receipt of the data sets from PwC we analysed the data received to identify trends which may indicate unusual activity. We have had access to the ToPAS system so that we can drill into specific information to see what has been recorded against individual patients records. Appendix 1 summarises the core data queries that were used as a basis for testing and identifies any findings or observations raised during the course of this testing. AUDIT FINDINGS 4. Audit Findings These are presented below to address the key objectives indicated within the audit terms of reference. Reference is made to the relevant recommendation which is contained in the Management Action Plan at the end of the report. 4.1 Objective 1 - Individual patient records are accurate and that systems are in place to ensure that the patient management system cannot be inappropriately changed. Our review of the process has found that access to the referral management system is only given to employees with the appropriate access along with the relevant level of training commensurate to their role. There are adequate levels of local documentation that support users in the day to day operation of their roles. Our discussions and information received from an anonymous survey found that those staff that took part agreed in general with this finding, although we could not identify a formal process for monitoring the total number of users on the Referral Management System to ensure they are still current and appropriate (Recommendation 1) As a direct result of mapping the process we identified the following gaps: There is a potential for a delay in updating patient referral data in ToPAS due to the use of the Waiting List Referral Card, which is a manual element to the otherwise electronic process. There is a risk that the document could go missing or be held back awaiting additional information which ultimately may create an unnecessary breach. (Recommendation 2) The Theatre Team update the Sapphire system with all relevant outcomes from their daily procedures, but this is not uploaded back to ToPAS creating the requirement for nurses to ring the Patient Access Team (PAT) so that this can be updated into ToPAS at a later date/time. This may result in a delay that could affect the accuracy of patient records and may create an unnecessary breach. (Recommendation 3) Changes to demographic information on some patients records were creating an amendment to historic periods of unavailability, that was later confirmed to be due to a system bug once it was reported to Cambric, the system provider. (Recommendation 4) Page 6 of 21

7 There was a lack of clarity surrounding the actual target/standard dates primarily concerning when the referral related to an intervention procedure. The impact of this is more challenging where a more specialised resource is required to complete the procedure. Confirmation was required to ensure there was sufficient lead time to arrange a procedure within a specialised field to meet 7 week rota and holiday commitments. (Recommendation 5) When we reviewed removals from the waiting list we noted that a large percentage of the reason codes used to complete a referral pathway were summarised as Treatment Longer Required, whereby the Detailed Reason was not specified. In addition we could not identify a process that incorporated a review of the outcomes input to the Referral Management System either to validate the inclusion on the list or to confirm that the final outcome coding is accurate. (Recommendation 6) Regular meetings are in place at a local management level to review potential breachers commonly six weeks on advance with a view to implementing measures to avoid this occurrence. This may include more focussed booking activity and putting on extra clinics, but we have no reason or evidence to suggest that pressure is brought on staff to amend records to minimise or reduce potential breaching referrals. Verification of information contained within the data sets against records within ToPAS has not flagged up any cause for concern. 4.2 Objective 2 - Reporting on waiting times is accurate and consistent at every level in the organisation up to and including the Board. The data on which Board reporting is based is taken from the raw data downloaded from the system and is the same base data as that which is uploaded to ISD on a monthly basis for validation. When the validated data is made available to us by ISD it presents within a report that denotes the quality of the data submitted by attaching a flag to each entry ranging from 0 to 3. The flag number indicates to us the level of quality with 0 being the standard and 1-3 denoting an omission of pieces of data in grades of seriousness. If a flag of 2 or 3 is attached the omission of data has created a potential breach. All of these numbered entries are investigated and the required corrective action is taken. further action is taken on this report. It is not passed back to inform the Waiting Times Management of any quality issues which can be improved through retraining or discussion with inputting staff. (Recommendation 7) Once validation of information is completed internally the data is used to populate spreadsheets used for analytical statistics used in reporting. In this respect the process is consistent. The production of the report presented to the Board is very formulaic in that it uses a standard template which sets out each section in the same way. Reports are created using extracts of the validated data and results or anomalies are followed up with the respective data managers and included as a commentary to the data. In this respect the reporting process is consistent, however this is reported at a specific point in time and only relates to the current position. Both the reporting format and process could be improved by including a more detailed trend analysis highlighting improvements or slippages in figures against target. This is compounded by alternate submissions between Performance Committee and the Board, which does not facilitate consistency of review. (Recommendation 8) The content of the report focuses heavily on waiting times which is consistent with the policy/strategy implications identified on the Monitoring Form, however other performance and efficiency target measures are also included which are not reflected in the monitoring form. In the majority of reports between January and June 2012 (5 out of 6) the Risk Assessment field was completed as t Applicable or N/A. This is inconsistent with the content of the reports and the Page 7 of 21

8 fact that there are risks attached with how we manage our patients expectations and ultimately not meeting Scottish Government targets. We traced a sample of waiting times information from the raw data in ToPAS through the downloading and validation process to the statistical analysis undertaken on a monthly basis. We confirmed the process by which this data is extracted and provided to the report writer for inclusion in the monthly reports submitted to Performance Committee and/or Board. We confirmed that the figures reported in the statistical analysis agreed to those in the Board report, however identified a formula error which created an underreporting of patient/day case totals in excess of 1000 patients. This had gone unnoticed since August 2011 which was when the originating error occurred. Whilst this was only an internal reporting discrepancy it should be of concern that this type of error was not picked up through monitoring or evaluation of accuracy of data reported. (Recommendations 9 and 10) In addition to internal reporting, we are required as a Board to submit a monthly return to the Scottish Government. The data used to populate this return is taken from the same statistical dataset mentioned above. We tested the figures reported to the Scottish Government in June 2012 and found all figures agreed between the return and the dataset with the exception of one figure which was merely a keying error. This was highlighted to the relevant staff and confirmed as an error and the return was resubmitted with the correct information. If the outturn was linked to the statistical dataset this process could be completed electronically. (Recommendation 11) 4.3 Objective 3 - The local guidance is consistent with national guidance and that its implementation is both valid and reliable (i.e. not open to different interpretation in use). From our discussions with key line management we can see that they are fully aware of the guidance and how this relates to the various roles within the referral management process. We were provided with a range of local procedures and guidance notes that support staff in their day to day processing, which were relevant to the roles that they performed. In addition to the documentation, focused training is provided to ensure that staff are fully conversant with their day to day work requirements. This ranges from induction for new starters through to more detailed training as their job role develops. Refresher training is also provided where this is required when there are local enhancements or updates to national guidance. We were directed to higher level policy documents which are referred to by staff at a local level. We could not confirm that these documents had been formally approved. For example, the Patient Access Policy was attached to the Performance Report presented to October Board to allow scrutiny by Board members. This had an effective date of October 2012, but there is no record of approval of this policy through the standard approval route. (Recommendation 12) In relation to completeness of recording for New Ways data fields this testing was carried out by reviewing how data is uploaded to ISD from the referral management system and viewing and analysing the data returned. The data is returned with validation flags which denote the accuracy of data and identify those which will impact on potential breachers. We have confirmed that all inaccurate data relating to potential breachers is rectified in a controlled and evidenced manner, however we are unclear that this information is shared for data quality purposes (see Recommendation 7). All discussions with staff are fully documented in our working papers. These discussions were undertaken with a range of staff involved in the referral management process including nonexecutives, clinicians, system administrators, IT staff and members of operational teams. To supplement this we issued a survey to provide a platform for staff using the system between Page 8 of 21

9 January and June 2012 to comment anonymously on various aspects of their role. The survey included questions on understanding of guidance, adequacy of training provided in relation to procedural and system responsibilities, availability of local procedures, workload management and resourcing issues. From the 150 users identified, 44 responded to the survey. Whilst this may appear a low number we found that responses were for the most part answered in a positive manner and the opportunity to include freeform comments provided some very constructive observations. Separate feedback has been provided on the results of this to appropriate management to draw out the good practice identified. In March 2012, following the PwC report on NHS Lothian, this Board implemented an enhancement to the local procedure for inputting retrospective periods of unavailability. It can be seen from the graph below that our instances of retrospective creations were comparatively low prior to the Lothian issue being raised publicly and the reinforcement of the procedure has reduced these instances even further Number of instances Total Inpatient Outpatient Unknown 0 Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Graph Reduction in retrospective unavailability use 5. Conclusion At the time of reporting, and based on audit testing undertaken, we can provide reasonable assurance that waiting lists are being managed appropriately within NHS Dumfries and Galloway. This assurance cannot be absolute due to the vast quantity of data that would need to be reviewed and the limited timeframe of six months as defined by the Scottish Government request. We can conclude, however that the scale of inappropriate use of unavailability codes reported in NHS Lothian is not evident within this Board from the data provided and the period reviewed. The focus of attention within the reporting framework is weighted towards the statistics within the referral process which is figures led. This does not capture the process as a whole in relation to the complete patient journey, and excludes items such as return waiting lists. If reporting was more process and outcome led the Board would achieve a more rounded view of the referral to treatment pathway. This will become increasingly important with the implementation of Treatment Time Guarantee obligations. We can conclude that local guidance is consistent with national guidance and there is sufficient documentation to support procedural requirements. If quality measures were introduced this would enhance processes and provide the tools to give relevant feedback to staff at a local level. This would also minimise future errors in patient records and provide a sound platform for meaningful reporting at all levels. We have made 12 recommendations based on our findings during the course of this audit. These are detailed in the Management Action plan in Section 8 at the end of this report. Page 9 of 21

10 6. Acknowledgements We would like to acknowledge with thanks the help and co-operation of all staff during the course of this audit. 7. Glossary of Terms The following details the abbreviations and associated terms encountered throughout the course of this audit report. Abbreviation SGHSCD NHS D&G PAS ToPAS ISD CHI PAT MMI PwC Term Scottish Government Health and Social Care Directorate NHS Dumfries and Galloway Patient Administration System NHS Dumfries and Galloway s patient administration system Information Services Division Community Health Index Patient Access Team Monthly Monitoring Information Waiting Times outturn to Scottish Government Pricewaterhouse Coopers LLP Page 10 of 21

11 8. Management Action Plan Audit Findings and Recommendation Management Response Key Risk / Control Recommendation Grade Management Action weakness Responsible We could not identify a Management should create a Agreed. Andrew formal process for process for managing ToPAS users Our IT internal processes already cater for Turner, monitoring the total number that encompasses the whole the transitioning of users being added and Information of users on the Referral master file of users by confirming removed from our systems. We have a Security Management System to their access is still required and A compliance problem where often, the IT ensure they are still current appropriate staff are not made aware of user change and appropriate roles / leave. We are commencing a program of work in line with ISO27001 There is a potential for a delay in updating patient referral data in ToPAS due to the use of the Waiting List Referral Card, which is a manual document. There is a risk that the document could go missing or be held back awaiting additional information which ultimately may create an unnecessary breach. The Theatre Team update the Sapphire system with all relevant outcomes from their daily procedures but this is not fully uploaded back to ToPAS, creating the requirement for nurses to ring PAT team so that this can be updated into ToPAS at a later date/time. This may result in a delay that could affect their Management should investigate the potential to include this process within the referral management system to use the controls built into the system. The potential to fully link these systems should be investigated to minimise the delay in updating the patients records. A A principles which will improve this situation. We are currently looking for a small number of clinicians to trial direct electronic recording in clinics. It is recognised that this is a good process to reduce risk of potential loss, however there are likely to be very high resistance from clinicians who will see this as spending more time with administration and less time with patients. This requires to be led by the Waiting List General Management. Understand concerns, however there is an assumption here that the Sapphire outcomes are sufficient to meet the needs of waiting list management. Sapphire outcomes relate to the outcome of the patient on the theatre list only. e.g. Theatre only record that a patient DNA d (Did t Attend), they do not record whether or not that patient is to be given another theatre slot or if they are to be removed from the waiting list altogether. We are currently exploring an option of Nicole Hamlet, General Nicole Hamlet, General Target Date Jun 13 Mar 13 (pilot) Jun 13 Page 11 of 21

12 4 Audit Findings and Recommendation Management Response Key Risk / Control Recommendation Grade Management Action weakness Responsible waiting time limits enhancing the theatre module within TOPAS so that all theatre data is recorded on the one system. This would make it far easier to ensure that there is a feedback mechanism to the waiting list module. It should be noted that not every elective patient has an operation in theatre. Also emergency patients often require follow up either as an outpatient or an elective procedure. We are currently investigating a new process where EVERY inpatient/day Case has an Inpatient Outcome recorded in the same manner as we currently do for outpatients. As a result of this, all discharge letters will only be allowed to be sent when an outcome has been recorded. This will address this issue in full. We identified that changes to demographic information on some patient s records were creating an amendment to historic periods of unavailability that was later confirmed to be due to a system bug. Management should ensure that assurances are gained from the software provider that this has been rectified. A The identified bug did not change the unavailability dates, but it did effect the Last Edited Date. The supplier has acknowledged the bug and this will be fixed in the next software upgrade. Stewart Cully, Information Services Target Date Jan 13 5 There was a lack of clarity surrounding target/standard dates primarily concerning when the referral related to an intervention procedure. Confirmation was required to ensure there was sufficient lead time to arrange a procedure within Waiting Times Management Teams should ensure Consultants are fully appraised of changes to guidance, legislation and how this affects target or standard dates of 9, 12, 18 weeks and so forth. A Agreed, this is partially due to ever changing waiting list targets / regimes. An education program will be delivered via SMT meetings. This will include reinforcement of annual leave policy. Nicole Hamlet, General Mar 13 Page 12 of 21

13 6 7 Audit Findings and Recommendation Management Response Key Risk / Control Recommendation Grade Management Action weakness Responsible a specialised field to meet 7 week rota and holiday commitments. We noted that a large percentage of the reason codes used to complete a referral pathway were summarised as Treatment Longer Required, whereby the detailed reason was not specified. In addition we could not identify a process that incorporated a review of the outcomes input to the Referral Management System to confirm that removal codings were supported and correct. Validated data reports returned by ISD following our monthly submission of data showed a number of quality issues whereby key data had been omitted from the referral input or as a result of system anomalies. Whilst corrections are made to the errors affecting breaching no additional quality assessment or corrective action is taken on the reasons for the administrative omissions. This failure to highlight these issues with inputting Management should introduce a review process whereby outcomes and removals from the list are analysed for accuracy and appropriateness. Management should introduce a quality assessment of the errors created due to omission or input of inaccurate data to the Referral Management System. This should include an analysis and summarisation of the key areas of concern and the introduction of a reporting and retraining programme. B B Agreed, Codes to be reviewed at least annually for continued relevance. Suggest the addition of a free text box for NLR s ( Longer Required) so that reasons can be recorded and therefore easily auditable. All of the ISD code 3 errors are already managed item by item to ensure that potential breachers are picked up and addressed. It is however recognised that data quality is an ongoing challenge and requires to be formally addressed going forward. Stewart Cully, Information Services Stewart Cully, Information Services Target Date Feb 13 Apr 13 Page 13 of 21

14 8 Audit Findings and Recommendation Management Response Key Risk / Control Recommendation Grade Management Action weakness Responsible staff is creating additional pressures and delays in the validation process and provides no meaningful feedback to our staff. There is a risk that patient records are incomplete. Our review of performance reporting highlighted a number of areas where the presentation and content did not provide a meaningful month-onmonth trend analysis that was consistently reported to one scrutinising committee or Board. Management should review the content of the Performance Report. This should include a full description of why each element of the report has been included, where the targets originate and how updates are captured, where the supporting data comes from and assurance on its accuracy, and a more detailed historical trend analysis. This process of review should encompass as wide a range of views regarding readers expectations of the report including non-executives to ensure the content is appropriate. B The content of the performance report is currently under review. The points highlighted will be picked up as part of this review. Julie White, Chief Operating Officer Target Date Mar 13 9 A review of the Performance Report identified an inaccurate reporting of the Inpatient/Day Case total for the period. The process of creating meaningful statistics from the month end Patient Access information involved the use of the previous Information Services Management should ensure that the macros and pivot table information used to calculate the figures reported internally for inclusion in the Board monthly reports are subject to an accuracy check to confirm that there is no corruption has taken place. Rather than use a previous month s spreadsheet they should set up a Master Template that B A Master template would not have avoided the error that occurred here. The error was introduced as the description of under 4 was changed to 0_4 in the underlying data extract. In excel pivot table terms, excel took the extra code and added an extra column at the end of the table. There was a formula =sum(3 columns) that made up the under 6 column that was transposed to the board report. This formulae carried on summing Stewart Cully, Information Services complete Page 14 of 21

15 10 Audit Findings and Recommendation Management Response Key Risk / Control weakness Recommendation Grade Management Action month s spreadsheet and should be protected. up 3 columns, but the under 4 was refreshing the data. This blank. The formula did not adjust to take has led to an error in a in the extra column at the end. See no 11 macros or formula which for suggested solution. went unnoticed from August The Inpatient/Day case totals within the performance report have been inaccurate since August Chief Operating Officer should restate Appendix 1 figures of the Monthly Performance Reports relating to Inpatients and Day Cases to reflect the accurate position of the Board from August B Yes. Suggest that we also introduce a separate sense check to be carried out by someone other than the person completing the board report. Responsible Stewart Cully, Information Services Target Date Immediate A review of the MMI Waiting Times Outturn found an inputting error resulting in an underreporting of one figure within the specialty detailed analysis. Whilst this was a human error there should be a final sense check completed to confirm that all information submitted is accurate. There is a risk that unapproved policies are difficult to enforce as they have not been through the standard approval process which is contrary to Information Services Management should ensure that figures reported within the MMI return are subject to a final accuracy check prior to submission to Government. They should consider linking the return to the various cells of the spreadsheets used to populate this return. Management should submit all policy documents through the Standard Approval route. B B The format of the MMI tends to change annually depending on SG s new priorities. The use of reference formulae is therefore likely to prove difficult to audit when layout changes. This is actually a very similar issue as no 9. We need to change the way both reports are produced and rely far less on Excel formulae, by using SQL data tables and possibly Qlikview. Programming will take longer but will ensure consistency. The fact that the programming will take longer may compromise our ability to meet initial SG submission target. There is often little time to respond to SG deadlines which are out with our control. Agreed, this requires to be actioned. Stewart Cully, Information Services Julie White, Chief Operating Officer Inline with changes to performance report / SG requests. Mar 13 Page 15 of 21

16 Audit Findings and Recommendation Management Response Key Risk / Control Recommendation Grade Management Action weakness Responsible Standing Orders. We could not evidence approval for the Policy for Hospital-led Cancellations, the Patient Access Policy or DNA Policy. Target Date Signature of Date Page 16 of 21

17 Appendix 1 Summary of core data queries and findings Query Description of data provided Action taken Findings or Observations 1 Pattern of periods of unavailability Analysed and reviewed From the patterns reviewed by hour, week and speciality being created Testing to validate the process has there appear to be no unusual patterns in Q1 been undertaken in Query 11 Procedures for creating referrals and unavailability have been discussed and found to be adequate Numerous guides, procedures and some policies that have yet to be ratified. Trend is downward 2 Pattern of periods of unavailability being created retrospectively 3 Pattern of amendments to period of unavailability 4 Pattern of bookings cancelled by the hospital, where a period of unavailability has been created within 5 days after the date that the booking was cancelled 5 Pattern of patients suspended within 5 days after refusing and offer where the offer has been Analysed and reviewed Testing to validate the process has been undertaken in Query 8 Looked at examples from before and after procedure was enhanced for retrospective unavailability inputting to confirm data is accurate. Analysed and reviewed link to Q9 and Q12 Further data was requested from PwC to complete focussed testing The query was analysed and reviewed and samples were tested. CHI numbers were requested from PwC to allow further testing data could be provided from the TOPAS system to inform this data set therefore our testing has been Q2 show low levels of retrospective changes decreasing after March Change of authorisation process internally explains this Decrease would suggest lack of awareness in coding rather than deliberate manipulation of figures Received further detailed analysis information further to Request 2, which includes user ID. Majority of entries in this query relate to universal amendments There were 34 examples identified where a period of unavailability was created within 5 days after the date that the booking was cancelled Testing found one common reason for this activity which was the hospital changing the original appointment time and the patient was unavailable for the proposed appointment time. This is not considered unusual activity and gives no cause for concern. Page 17 of 21

18 Query Description of data provided Action taken Findings or Observations refused due to the treatment being outside the NHS Board 6 Deletion of periods of unavailability 7 Patients removed from the waiting list 8 This is testing in relation to Q2 - Pattern of periods of unavailability being created retrospectively undertaken from other queries and local access to the system This data was analysed and following review further data was requested to allow additional testing by hour, day, week, month and user ID There were a total of 1661 deletions of periods of unavailability within this query. From the data we could see that 1236 (74%) of these deletions were created by the system. This is appropriate and not an indicator of manipulation. The remaining 26% of deletions were created manually, the reasons for which were appropriate and supported although some of the commentary could have been more complete to enhance the patient s record. Analyse and review link to Q14 A review of the sample of removals found that the commentary supported the reasons for the removal. We also noted that of the total number of 6670 removals between January and June % of these related to Treatment no longer required. This was further analysed and identified that 29% were removed with the agreement of the consultant and 12% related to advice letters raised by GPs. The patient outcomes are recorded on the Clinical Outcome Sheet which when completed is printed off and filed for between 3-6 months. This is not scanned or saved electronically and cannot be accessed by consultants or GPs. This will be resolved when the e-case notes system is fully operational. Select a representative sample before and after 31 st March when the procedure was amended. Relatively low levels of retrospective amendments were observed. Comparison of the percentages of overall to retrospective creations found that the percentages of creations made retrospectively were 9% in Jan & Apr, 11% in Feb & Mar, 5% in May and 3% in Jun averaging 8% in total. The highest percentages were in the specialities with the highest creations observed in Q1. Review of the data found that the highest number of retrospectives processed was in week 9 which had increased in line with changes in Outpatients as opposed to Inpatients. The lowest weeks were from week 20 onwards in line with Page 18 of 21

19 Query Description of data provided Action taken Findings or Observations the revised instruction. We observed significant peaks in weeks 4 and 9 with a significant trough in week 15 (10 th April) which was Easter week none of which coincided with quarter ends. There is a significant drop in retrospectives since the revised instruction in March This is testing in relation to Q3 Select a period and speciality of patients where the application of unavailable has been amended in the period Our review of the amendments made to periods of unavailability found no unusual patterns that had been made as a result of manual manipulation. The majority of entries made within this query were made by the system itself and analysis of the remainder identified no cause for concern. Our testing did identify a bug within the system that created amendments to periods of unavailability when users were inputting updates to demographic information through the Patient Details Screen, which had not been previously noted. This has been raised with the system provider who will investigate the matter with a view to making a correction to the system. 10 The highest number of offers made per hour 11 The highest number of periods of unavailability input per hour Requested further information regarding frequency from PwC as some data was missing and limited within the Excel format This query could not be fully analysed from he data provided due to the information being date stamped as opposed to time stamped. It was observed that the data is weighted by the system matching patients with clinic slots on a daily basis which may or may not subsequently be accepted. A review of the data that showed the breakdown of the offers made by the system users was found to be in line with the busiest specialities and corresponded to the information being provided in other queries. There appeared to be no unusual or remarkable trends which would indicate a cause for concern with a consistent level of input across all users on a daily, weekly and monthly basis. Analysis of data undertaken. From our review of the highest number of periods of input in an hour we can say that there appears to be no cause for concern. The data provided included a system user that has been set up to run daily tickets within the system. Once this Page 19 of 21

20 Query Description of data provided Action taken Findings or Observations 12 The highest number of amendments made to periods of unavailability made per hour 13 Profile of current waiting list (depending on period of data extracted) 14 For 5 cases removed from the waiting list because they have been Transferred to Another Board, trace them through to the health records of the other Board This query links to Q s 9&3 and presents similar findings although in more detail, focussing on the hourly input. This was reviewed by interrogating ToPAS to review patients pathways. A sample was identified and passed to PwC user had been removed from the statistics there was a clear reflection of a number of high users within certain specialities. When these users data was reviewed on a daily and hourly basis there was no unusual pattern determined on this test. See results for Query 9 Review of patients pathways has not revealed any unusual activity. The validation of the waiting list has been reviewed in full as the data within this query was consolidated within all other testing. From the sample of 5 we increased this to 10 removals which were confirmed on other Boards waiting lists. 15 Profile of the number of instances where the referral date has been moved by more than 5 days 16 Offer and appointment dates that are the same (or within 3 days of each other We obtained a more detailed set of data from PWC which was received on 2 nd October. This data: Did not reflect the user ID Continued to have a large number of entries that were NULL and the only reference was the Patient ID number which could not be used to access anything on ToPAS. Completed a sample of the remaining entries and the majority appeared to be for Urgent cases, or speculative chances with In the sample of patient pathways reviewed in other queries we found no evidence to suggest that referral dates are being moved. The data provided in this query shows referral information that could have been changed within any of the fields of the referral table not specifically a movement of the referral date. From the information the data has shown: 187 Users have created these entries 25,917 entries have been created in the 6 month period Data is included on Cancellations Large number of NULL entries, which related to New to Return and/or Return to Return appointments which may be completed on a much shorter turnaround Page 20 of 21

21 Query Description of data provided Action taken Findings or Observations unreasonable offers, some of which were accepted and others were declined. 17 Offers declined per day Requested information relating to 5 top users and requested focussed testing for 3 of top users by speciality with highest number of declined offers Initially the data provided did not reflect the dates on which the cancellations had taken place, however when a further request was met this information proved more useful to meet the objective of the query. Our analytical review of the data as a whole found no significantly unusual patterns. This work found 13 users had processed in excess of 100 offers over the period however when this was reviewed individually this was not considered excessive and was found to be in line with the higher volume of activity within the specialities themselves. Our more detailed review highlighted some areas within which more focussed testing was performed. Each subsequent test found that the offers declined were supported fully and were mostly the direct result of a written appointment offers being unsuitable for the patients concerned. In most occasions the original declined offer was followed up by between 1 and 4 further offers which the patient then accepted. The remaining small percentage had been cancelled by the patient. Finally a small number of declined offers did not have a cancellation date attached to the information provided by PwC. This was confined to one user who manages the Endoscope and Colonoscopy appointments. Review of ToPAS found that only elements of the records were reflected in ToPAS, however there was more than sufficient evidence to support the activity in this area. Page 21 of 21