Data Quality Spot Checks. Thanet District Council Audit 2008/09 October 2009

Transcription

1 Data Quality Spot Checks Thanet District Council Audit 2008/09 October 2009

2 Contents Overview 3 Detailed findings 5 Main conclusion 7 Appendix 1 Approach 8 Appendix 2 Action Plan 9 Status of our reports The Statement of Responsibilities of Auditors and Audited Bodies issued by the Audit Commission explains the respective responsibilities of auditors and of the audited body. Reports prepared by appointed auditors are addressed to non-executive directors/members or officers. They are prepared for the sole use of the audited body. Auditors accept no responsibility to: any director/member or officer in their individual capacity; or any third party.

3 Overview Overview Introduction 1 This summary report sets out the findings of our data quality spot checks of selected performance indicators (PIs) for 2008/09. The report supplements our overall use of resources feedback to the Council by providing detailed spot check findings in order to highlight to officers any weaknesses or areas for improvement we have identified. Background 2 Use of resources Key Line of Enquiry ('KLOE') 2.2 focuses predominantly on arrangements for using fit-for-purpose information and securing data quality. We undertake spot checks of selected data, based on our knowledge of local risks, to provide evidence to support our KLOE judgement. This integrated approach to use of resources and data quality replaces the previous three-stage approach to data quality work which was undertaken separately. 3 Undertaking spot checks supports our judgement on KLOE 2.2 by confirming whether an authority s arrangements are working in practice and are applied consistently. The spot checks therefore consider whether data is fit-for-purpose, by considering the arrangements to produce the data and testing a small sample of supporting records. The purpose of the spot checks is not to comment specifically on the published value of an indicator, as was previously the case for PI testing to support CPA service scores. 4 The spot checks may be of performance indicators from the National Indicator Set, local performance indicators or any other performance measure which lends itself to this style of testing. Scope and selection of PIs 5 The number and selection of PIs for spot checking depends on local data quality arrangements, local priorities and any risks we have identified. Based on our assessment, we selected the following PIs for spot checking: NI number of affordable homes delivered; NI number of households living in temporary accommodation; LI hits on the Council's tourism website. 3 Thanet District Council

4 Overview 6 In addition to the above spot checks which were selected based on local priorities and risks, for 2008/09 the Audit Commission specified data quality work on housing and council tax benefits data as part of the integrated HB COUNT approach. We are now required to undertake one integrated piece of work on benefits, covering claim certification, data quality and some audit opinion requirements, using a set of workbooks developed by the Commission. The work supports auditors use of resources assessments and the Commission s benefits inspection risk assessments. Our detailed findings from this integrated piece of benefits work will be reported to officers in a separate Housing and Council Tax Benefits report. Spot check approach 7 Our spot checks of PIs consist of the following. Applying a set of service-specific management arrangements questions. Understanding the system used to collect and process the data in accordance with the PI definition. Testing the underlying data against the six data quality dimensions (accuracy, validity, reliability, timeliness, relevance and completeness) as applicable. 8 Further details of the spot check approach we have applied are set out in Appendix 1. Thanet District Council 4

5 Detailed findings Detailed findings 9 Our detailed findings are set out below. Table 1 Spot check findings Performance indicator NI 155: Number of affordable homes delivered NI 156: Number of households living in temporary accommodation LI 109: Hits on the Council's tourism website Comment Our testing found that the PI had been prepared in accordance with the PI definition and underlying data. Although our testing identified no discrepancies with supporting documentation, it was noted that the method of collecting the data does not include all sources that are detailed within CLG guidance, but only returns supplied by the Council's four preferred partners. Our testing found that the PI had been prepared in accordance with the PI definition and underlying data. Although our testing identified no discrepancies, it was noted that the outturn reported in the Annual Performance Report differed from that reported to CLG. Although the impact was minor, it is important that outturns reported by the Council to different recipients are consistent. We also noted that the spreadsheet used for recording data for the return to CLG is an integral document that is also used for a number of other activities. There however no procedure notes for the completion of the spreadsheet and knowledge of how it should be used is currently limited to one individual officer. Our testing found that the PI had been prepared in accordance with the set definition and underlying data. Although our testing identified no discrepancies, it was noted that the monitoring of controls in relation to the compilation and reporting of Local Indicators is currently limited. There are, for example, a number of Local Indicators where the central Policy team is not aware of the basis of calculation and crucially whether there has been any change from the previous year. Given the Council currently maintains a large number of Local Indicators, there is risk that some of the resultant performance data is not fit for purpose and hence unreliable for decision making purposes. 5 Thanet District Council

6 Detailed findings Recommendations R1 NI Capture all relevant sources as set out within CLG guidance when establishing reported outturns. R2 Ensure PI outturns reported to different recipients are consistent or, if unavoidable, any differences are fully explained. R3 NI Document the process for the completion of the spreadsheet and share with at least one other officer. R4 Consider strengthening the monitoring of controls in relation to Local Indicators to ensure that the processes involved are sound and that their calculation is fit for purpose. Thanet District Council 6

7 Main conclusion Main conclusion 10 For all three of the PIs we selected for spot checking, we found no discrepancies in the recording and processing of underlying data. However, in the case of NI 155 (Number of affordable homes delivered) we identified that the reported data was not complete as it did not capture all relevant sources as set out in CLG guidance. 11 It was noted that the monitoring of controls in relation to the compilation and reporting of Local Indicators is currently limited and as such there is risk that some of the resultant performance data is not fit for purpose and hence unreliable for decision making purposes. 12 Our findings were broadly consistent with our overall assessment under KLOE 2.2 that the Council has adequate arrangements to produce relevant and reliable data and information to support decision making and manage performance. Our part of our work next year we will monitor progress against the recommendations set out in Appendix 2. 7 Thanet District Council

8 Appendix 1 Approach Appendix 1 Approach 1. Management arrangements Answer service-specific management arrangements questions, after checking that they are all relevant for this performance indicator. 2. Systems Obtain an understanding of the system used to collect and process the data in accordance with the performance indicator definition. 3. Testing Test a small sample of supporting records using the follow test guidelines. Test 1 - Test the accuracy of the PI calculation Refer to the performance indicator definition, and confirm correct numerator and denominator are used. Check all figures back to systems reports or compilation documents and re-perform the calculation. Test 2 - Test the completeness and accuracy of the numerator Check supporting figures back to system via the audit trails. Where intermediate calculations or adjustments have been performed to arrive at the required figure, check the arithmetic and review for reasonableness. Select a sample of items and ensure: the data is complete for the whole year; the data is complete for all methods of collection (for example where data is collected from different sites); the data has been recorded accurately and is in accordance with the PI definition; the data is valid, for example the correct data has been included or excluded; and where samples have been used to calculate the numerator, these are reasonable and follow relevant requirements of the PI definition. Test 3 - Test the completeness and accuracy of the denominator Check supporting figures and totals back to source. Where intermediate calculations or adjustments have been performed to arrive at the required figure, check the arithmetic and review for reasonableness and accordance with the PI definition. Consider the need for any further detailed testing. Overall conclusion - Does the reported PI comply with the Audit Commission's data quality criteria? Does the reported PI comply fairly with the Audit Commission s data quality criteria (accuracy, validity, reliability, timeliness, relevance and completeness) and are there any specific data quality concerns? Thanet District Council 8

9 Appendix 2 Action Plan Appendix 2 Action Plan Page no. Recommendation 6 R1 NI Capture all relevant sources as set out within CLG guidance when establishing reported outturns. 6 R2 Ensure PI outturns reported to different recipients are consistent or, if unavoidable, any differences are fully explained. Priority 1 = Low 2 = Med 3 = High Responsibility Agreed Comments Date 3 Stephen Tebbett Yes Stephen will work with Ashley Stacey to establish a Thanet Specific Performance Indicator Definition to ensure that Thanet is compliant with the CLG guidance when reporting this indicator. 2 Stephen Tebbett Yes As part of the work to identify Performance Indicator Definition the reporting of information to different recipients will be captured where applicable and an approach to ensure consistency of reporting will be agreed for each indicator where this is relevant. High risk indicators will be reviewed by Christmas December 2009 December R3 NI Document the process for the completion of the spreadsheet and share with at least one other officer. Other indicators will be reviewed as part of the service planning/corporate plan refresh process and will be completed by end of April Stephen Tebbett Yes Stephen will work with Vicky Harley to establish clear Performance Indicator Definition which will clarify the process of calculation so that other officers could pick it up in her absence. April 2010 December Thanet District Council

10 Appendix 2 Action Plan Page no. Recommendation 6 R4 Consider strengthening the monitoring of controls in relation to Local Indicators to ensure that the processes involved are sound and that their calculation is fit for purpose. Priority 1 = Low 2 = Med 3 = High Responsibility Agreed Comments Date 2 Stephen Tebbett Yes Stephen will roll out a process for the identification of comprehensive Performance Indicator Definition for both national and local indicators. As in R2 Thanet District Council 10

11 The Audit Commission The Audit Commission is an independent watchdog, driving economy, efficiency and effectiveness in local public services to deliver better outcomes for everyone. Our work across local government, health, housing, community safety and fire and rescue services means that we have a unique perspective. We promote value for money for taxpayers, auditing the 200 billion spent by 11,000 local public bodies. As a force for improvement, we work in partnership to assess local public services and make practical recommendations for promoting a better quality of life for local people. Copies of this report If you require further copies of this report, or a copy in large print, in Braille, on tape, or in a language other than English, please call Audit Commission 2009 For further information on the work of the Commission please contact: Audit Commission, 1st Floor, Millbank Tower, Millbank, London SW1P 4HQ Tel: , Fax: , Textphone (minicom):