Data Quality Review Newport City Council. Audit year: Issued: January 2016 Document reference: 703A2015

Transcription

1 Data Quality Review Newport City Council Audit year: Issued: January 2016 Document reference: 703A2015

2 Status of report This document has been prepared as part of work performed in accordance with statutory functions. In the event of receiving a request for information to which this document may be relevant, attention is drawn to the Code of Practice issued under section 45 of the Freedom of Information Act The section 45 Code sets out the practice in the handling of requests that is expected of public authorities, including consultation with relevant third parties. In relation to this document, the Auditor General for Wales and the Wales Audit Office are relevant third parties. Any enquiries regarding disclosure or re-use of this document should be sent to the Wales Audit Office at info.officer@audit.wales. Page 2 of 14 - Data Quality Review - Newport City Council

3 Contents Summary report Background 4 Review findings and conclusions 5 The Council has addressed our recommendation to collect and publish data on Final Special Educational Needs Statements issued within 26 weeks (EDU/015a) in line with the national prescribed definition Although the Council has made improvements, there are still weaknesses in its data quality arrangements 6 6 Proposals for improvement 9 Appendix Summary of the results of our review 10 Page 3 of 14 - Data Quality Review - Newport City Council

4 Summary report Background 1. As part of our Data Quality Review in , we made the following recommendation in relation to the Council s data quality processes: R1 The Council must ensure that all National Strategic Indicator (NSI) data is collected and published in accordance with the Welsh Government definitions; particularly NSI. EDU/015a - Final Special Educational Needs statements issued in 26 weeks. 2. We also made the following proposal for improvement as part of our Data Quality Review in : P1 The Council must ensure that it has robust data collection arrangements for its own self-defined PIs that includes ensuring that: the Council s corporate arrangements for performance management are being rigorously implemented by service departments; the Council s own internal information systems (VIEWS) reconcile with the final validated data; and a complete audit trail of the Council s own internal processes for validating PI data is retained. 3. During November 2015 we carried out our data quality review and examined a sample of six of the Council s performance measures and operational data systems. The main objective of the review was to assess whether the Council has adequately addressed our recommendation and proposals for improvement to improve its data quality assurance arrangements. 4. This report provides a summary of the conclusions from our recent review. It does not provide a conclusion on the accuracy of all of the out-turn figures included in the Council s published performance report. This is because the existence of sound data systems reduces, but does not eliminate, the possibility of error in reported data. 5. A summary of the results of our review is set out in Appendix 1. Descriptors are based on the extent to which the Council has put in place, and is operating, effective processes and controls over the data systems that support performance measures. Page 4 of 14 - Data Quality Review - Newport City Council

5 Review findings and conclusions 6. We examined six performance measures and data systems in our latest data quality review. Two were Public Accountability Measures (PAMs), one was a National Strategic Indicator (NSI) and three were local performance measures. The first four of these, as shown below, are linked to one of the Council s Improvement Objectives (IO): Number of Social Services hospital discharge assessments completed per annum (Local indicator) IO1: Supporting Older People Leaving Hospital Carers offered assessment or review (PAM) IO2: Improving Care and Support Services for Adults Percentage of victims satisfied with the service they received from Youth Offending Service/Team around the Family (YOS/TAF) team (Local indicator) IO8: Improving Outcomes for Youth Justice Number of people engaged in employment programmes (Local indicator) IO3: Developing and Regenerating the City The percentage of final statements of special education needs issued within 26 weeks including exceptions (NSI) The number of working days/shifts per full time equivalent (FTE) local authority employees lost due to sickness (PAM) 7. The fifth indicator above, (the percentage of final statements of education needs), was included in the sample as it relates to the recommendation in our Data Quality Review. The final indicator above (sickness absence) relates to recommendation 4 from our Corporate Assessment follow-up report, which included the requirement for the Council to put in place a key performance indicator and a challenging target for sickness absence. 8. We found that the Council has addressed our statutory recommendation and is continuing to improve its data quality arrangements. However, there are still weaknesses in these arrangements, which need to be addressed to ensure that its published performance data is accurate. Our main findings are set out below. Page 5 of 14 - Data Quality Review - Newport City Council

6 The Council has addressed our recommendation to collect and publish data on Final Special Education Needs Statements issued within 26 weeks (EDU/015a) in line with the national prescribed definition 9. The Council has responded positively to the recommendation from our review last year and is now using the correct, nationally prescribed definition for EDU/015a the percentage of final statements of special educational needs issued within 26 weeks, including exceptions. The key change is that the Council is now starting to count the period of time from when it is made aware of the request for a statement. This is in contrast to its previous approach when it used a date later in the process. Officers have put in place sound arrangements to ensure that this data is now collected correctly. The officers have reviewed and checked all cases during to ensure that the outturn figure of 27.8 per cent is accurate. Although the Council has made improvements, there are still weaknesses in its data quality arrangements The Council has made some further improvements to its data quality arrangements 10. There is an increased level of corporate support for service departments to help them to understand the corporate arrangements around data quality. The Council has developed a Frequently Asked Questions document. This document is regularly updated to provide guidance on issues such as definitions, timescales for data returns and the use of VIEWS. In addition, officers responsible for the collation and/or performance indicator data attend regular meetings of the Operational Performance Network (OPN).The Network is proving to be a useful source of support for these officers as it provides them with an opportunity to discuss operational issues and concerns. 11. There have been improvements in the corporate arrangements and processes designed to improve the quality of the Council s performance data. The central performance team has clarified and enhanced the Council s self-assessment form, completed for all indicators by the compilers/owners. In addition, the central performance team has relaunched and updated the pro-forma used to set out definitions for local indicators. Our review of the six indicators set out above (three of which are local indicators) indicated that relevant officers are completing these forms and officers understand their purpose and importance. The central performance team ensures that these forms are returned. The team then stores them in a central e-library along with a list of the compilers and reviewers of all the Council s indicators. Page 6 of 14 - Data Quality Review - Newport City Council

7 12. The central performance team has emphasised the importance of the consistent use of VIEWS, the Council s performance management system. Training has been provided to relevant officers responsible for compiling and reviewing performance indicators. Our review indicated that this training has resulted in an increased awareness of, and confidence in using, VIEWS. Our work also found that in all cases except one (see paragraph 15) the data on VIEWS reconciled with the data submitted to the Welsh Government. 13. One of the specific improvements initiated by the central performance team since our review last year is the provision of training to relevant officers aimed at improving the audit trail for individual indicators. Our review found sound arrangements in place to record the source data for five of the six indicators tested and good evidence of appropriate tools to produce regular reports for managers. 14. Internal Audit play a clear role in the Council s data quality arrangements. Internal Audit staff liaise closely with the central performance team to discuss areas of risk within the suite of indicators and agree which to audit in more detail. Our review found that the indicator compilers and reviewers are clear about the role of Internal Audit and the importance of detailed checks on the accuracy of data. There are still weaknesses in some of the Council s data quality arrangements which need to be addressed to ensure that its published performance data is accurate 15. Our review of the local indicator measuring the level of satisfaction of victims with the service they receive from the Council s Youth Offending Service found that the Council s published outturn data (59.8 per cent) was incorrect. This error was the result of a combination of a lack of understanding of the process for collating and reporting performance data by the compiler and poor oversight of the process. HMI Probation also identified inadequate management oversight as a weakness in the service in its report published in September The service was able to provide the correct outturn figure (63.2 per cent) to us during our review. Following discussions with the recently appointed service manager, we are assured that more robust arrangements are now in place to collect and report the accurate performance for this indicator. 16. The Council measures sickness absence in two different ways, using both the PAM indicator and a local measure. The local measure uses an average full time equivalent figure, which the Council considers more accurate as a monitoring tool for managers and members. The Council records this local measure in VIEWS. The PAM indicator is calculated by the Council s HR team at the end of the financial year in accordance with the nationally prescribed definition and submitted to the Welsh Government. The Council does not record the PAM indicator in VIEWS. The Council submitted the correct figure for the PAM indicator. The Council published the correct figure for the PAM indicator in its performance report. 17. We acknowledge that the Council may wish to record its sickness data on VIEWS using its own calculation. However, the Council should also record the nationally prescribed indicator on VIEWS to ensure completeness and transparency. Officers Page 7 of 14 - Data Quality Review - Newport City Council

8 should then explain any differences between the two figures in reports to senior managers and elected members. 18. The Council does not use formal criteria in the risk assessment process to determine those PIs that will be subject to detailed testing by Internal Audit. Criteria may usefully include the consideration of recent inspections and audit findings as well as percentage variance against prior year performance. The Council may have identified the Youth Offending Service indicator as a risk if it had employed such criteria. The Council also does not formally record the risk assessment process. This means that it is difficult for the Council to track the risks identified from year to year or to identify any trends. 19. It is positive that the Council is keen to ensure that there is ownership of data quality within the services rather than it being seen solely as a central performance team concern. The compiler for each measure completes self-assessment and local definition forms. The forms are then checked by a Reviewer and signed off by the Head of Service. Whilst the sample of indicators tested has been small, the fact that we have found errors in two of the six indicators suggests that the Council s data quality arrangements are not yet mature enough to provide full assurance of the arrangements. The Council would benefit from sample checking completed selfassessment and local definition forms to ensure that its data quality arrangements are being used effectively. 20. Although elected members at the Council regularly receive performance reports, they are provided with limited information on the Council s data quality arrangements. For example, Cabinet considered our Data Quality report. Conversely, Audit Committee did not receive Internal Audit s findings on data testing. This means that elected members are not able to scrutinise fully the Council s data quality arrangements. 21. Finally, our review identified two issues that indicate that levels of proof-reading and checking need to be improved. Firstly, the indicator title relating to the number of people engaged in employment programmes has been wrongly included twice in the Council s published Improvement Plan. The additional inclusion was in place of the indicator measuring the number of people supported into employment. The Council has corrected this error in subsequent monitoring reports. Secondly, the title of the indicator The number of people engaged in employment programmes changed slightly in but the completed self-assessment form did not reflect this change. We previously raised the issue of proof reading with the Council in order to avoid unnecessary errors and omissions in relation to its Improvement Plan Page 8 of 14 - Data Quality Review - Newport City Council

9 Proposals for improvement 22. The proposals for improvement set out below supersedes those that we have made in previous data quality reports. P1 The Council needs to focus on the accuracy of its performance data by: ensuring all nationally set indicators are recorded in VIEWS; checking, on a sample basis, the quality and accuracy of information provided in the self-assessment forms; formalising and recording the risk assessment approach used to identify which indicators should be reviewed by Internal Audit; providing members with the results of Internal Audit s review of the Council s data quality arrangements, thereby enabling members to challenge the arrangements; and improving the proof-reading of the Council s performance reports. Page 9 of 14 - Data Quality Review - Newport City Council

10 Appendix 1 Summary of the results of our review Descriptor The data system is fit for purpose and effectively run. The data system is adequate but some improvements could be made. Indicators we reviewed that received this score Number of Social Services hospital discharge assessments completed per annum (Local) Carers offered assessment or review (PAM) The percentage of final statements of special education needs issued within 26 weeks including exceptions (NSI) Number of people engaged in employment programmes (Local) Comments N/A N/A N/A In general, the arrangements for collecting and reporting this indicator are sound. The service has completed the local definition form accurately. There are clear arrangements for data collection, input and storage. There are good quality checks of the data. The service has set up a reporting tool to produce sound management information. However, the indicator name was repeated erroneously in the Improvement Plan. The name of the indicator has changed slightly since but the self-assessment forms were not updated to reflect this. The Council should ensure that its documents are properly proof-read before publication and it reflects changes in indicator titles in its internal documentation. Page 10 of 14 - Data Quality Review - Newport City Council

11 Descriptor The data system has some weaknesses which the Council is addressing. Indicators we reviewed that received this score Percentage of victims satisfied with the service they received from YOS/TAF team (Local) Comments The data for this indicator was not collated and reported correctly in This led to the wrong outturn figure (59.8 per cent) being published in the Council s Performance Report in October This was due in part to a lack of understanding of the process for collating and reporting performance data by the compiler. There was also a lack of management oversight of the process. HMI Probation identified significant problems with the Council s Youth Offending Service in its report published in September 2014 by HMI Probation, including a lack of management oversight. During a review of the arrangements to collect data for this indicator, the new service manager realised that there were mistakes in the process. The service has since corrected the outturn figure. The service manager also worked with the compiler and the Council s central team to improve processes. This has included the provision of a clearer definition for the indicator. We are assured that more robust arrangements are now in place. Page 11 of 14 - Data Quality Review - Newport City Council

12 Descriptor The data system has some weaknesses which the Council must address. No system has been put in place to establish performance against the measure. Indicators we reviewed that received this score The number of working days/shifts per full time equivalent (FTE) local authority employees lost due to sickness (PAM) None Comments The Council outturn figure recorded on VIEWS for the PAM indicator measuring the level of sickness at the Council was not the same as the figure submitted to the Welsh Government. This discrepancy is due to VIEWS being set up to measure a local indicator which measures the number of full-time equivalent staff at the Council in a way that is not the same as the methodology set out in the national guidance. The service uploads the number of Full time equivalents (FTEs) at the Council at the end of each month into VIEWS. The system then calculates the average of the twelve monthly figures. HR calculates the PAM indicator at the end of the financial year in accordance with the national prescribed definition. The statutory guidance states that the figure should be calculated by adding together the number of FTEs at the beginning of the year with the number at the end and then dividing this by two. The figure submitted to the Welsh Government and published in the Council s performance report for was correct We acknowledge that the Council may wish to report its sickness data on VIEWS using its own calculation. However, the Council should also record the PAM indicator on VIEWS. Officers should explain any differences in reports to senior managers and elected members. N/A Page 12 of 14 - Data Quality Review - Newport City Council

13

14