Review of Data Quality. Guildford Borough Council Audit 2008/09 January 2009

Transcription

1 Review of Data Quality Guildford Borough Council Audit 2008/09 January 2009

2 Contents Introduction 3 Main conclusions 4 Audit approach 6 Management arrangements 7 Analytical review 10 Data Quality spot checks 11 Appendix 1 Action plan 13 Status of our reports The Statement of Responsibilities of Auditors and Audited Bodies issued by the Audit Commission explains the respective responsibilities of auditors and of the audited body. Reports prepared by appointed auditors are addressed to non-executive directors/members or officers. They are prepared for the sole use of the audited body. Auditors accept no responsibility to: any director/member or officer in their individual capacity; or any third party.

3 Introduction Introduction 1 The Audit Commission applies a three stage approach to the review of data quality. Stage 1: a review of management arrangements. Stage 2: analytical review of performance indicator information. Stage 3: substantive testing of a sample of published performance indicators. 2 In 2007/08 the Audit Commission introduced a new approach to its stage 3 testing of performance indicators, with a particular focus on the performance indicators for speed of processing of Housing Benefits new claims and changes of circumstances. The approach is designed to integrate our work on data quality with our work on the certification of the Housing and Council tax benefit subsidy claim. It also enables the Audit Commission to obtain a consistent and comparable picture of data quality for this indicator across all of England and Wales. 3 Guildford Borough Council agreed to be an early implementation site for this new approach. The early implementation work began in July 2008 and involved the Director of Policy from the Audit Commission's central Directorate, who spent several days on site in July and September. 4 By carrying out this work at an early stage in Guildford, the Audit Commission was able to trial the new approach and to learn lessons which helped formed the basis of later advice and training to auditors elsewhere in the country. 5 We would like to record our thanks to staff in the Revenues and Benefits section for agreeing to be part of this early implementation and for their assistance during this audit. 6 The expected impact of our work on data quality is that it will drive improvement in the quality of local government performance information, leading to greater confidence in the supporting data on which performance assessments are based. 7 Auditors work on data quality and performance information supports the Commission s reliance on performance indicators in its service assessments for comprehensive performance assessment (CPA). This delivers the commitment to reduce significantly the level of service inspection required. 8 Introducing the comprehensive area assessment (CAA) framework from 2009 will make reliable performance information more important. The CAA will place greater emphasis on assessments that are proportional to risk. Councils will also be required to use information to reshape services, and to account to the public for performance. 3 Guildford Borough Council

4 Main conclusions Main conclusions 9 The overall corporate arrangements for supporting data quality are good. Guildford Borough Council (the Council) has set a comprehensive strategic framework for its performance management and data quality. The Council has addressed most of the recommendations from last year's data quality review and has taken steps to embed data quality into corporate risk management. 10 The arrangements for supporting data quality at service level are inconsistent. Our tests on 2007/08 Best Value Performance Indicators (BVPIs) highlighted weaknesses in the reliability, accuracy and completeness of data. It is important for the Council to get procedures right within services, not only for securing data quality but also in helping to manage and improve service delivery. Stage 1 - management arrangements 11 The Council has further strengthened its corporate arrangements for data quality through the integration of the corporate plan, data quality policy and the corporate risk management register onto the performance management system (Prime). This has enabled an increased awareness of data quality issues for both senior officers and members. 12 The Council has made slow progress in updating departmental guidance and procedures for performance indicators. We continue to find weaknesses in our testing of performance indicators. Updating departmental guidance and procedures and involving staff in the process should enable the Council to improve the accuracy, relevance, timeliness and completeness of its data. Stage 2 - analytical review 13 Our analytical review at stage 2 identified five performance indicators with values that were outside the expected range. One was assessed in stage 3 and we conclude that the remainder were all fairly stated. Stage 3 - data quality spot checks 14 We carried out a review of three performance indicators. Two of these (BVPIs 78a and b) are being tested at all councils nationally and the review of BV184a was a follow up of last year's audit. BVPI 78a Housing Benefit (HB) / Council Tax (CT) benefit time to process new claims and BV 78 b HB/CT benefits' time to process changes of circumstance. We carried out detailed testing of 31 new claims and 31 changes of circumstance claims. BVPI 184a Proportion of LA homes which were non decent (previously tested in 2006/07) and 184b Percentage change in the proportion of non decent homes (associated with 184a). We qualified this PI last year and have reviewed it this year to follow up on last years issues. Guildford Borough Council 4

5 Main conclusions 15 Two out of the three indicators tested in stage 3 were fairly stated. However there were a number of issues with the indicator BVPI 78b ("time to process benefits changes of circumstance"). The Council misinterpreted the 2007/08 definition, which led to a number of errors. The indicator required notification of changes to be physically received. The Council misinterpreted the definition by including notification of changes by telephone. This had the effect of understating the time taken to process changes of circumstances. We note, however, that the definition has changed for 2008/09 and telephone notifications can now be included. The Council has not acted on last year's recommendation to update departmental guidance and procedures for performance indicators. There were no written departmental procedures and guidance for Housing Benefit /Council Tax benefit PIs and this was reflected in the quality of data and misinterpretation of definitions. The Council uses a calculation based on a random sample of 500 cases, which it follows in order to compensate for system problems and inputting errors. We found a number of inputting errors when testing the source data. An intelligent review of the current system and staff training on data quality will help the Council reduce the errors and inaccurate data being reported. 16 The detailed results of stage 3 are included in 'data quality spot checks' found in the main section of this report. The way forward 17 An action plan to address issues arising from this review is attached at Appendix 1. 5 Guildford Borough Council

6 Audit approach Audit approach 18 The Audit Commission has developed a three-stage approach to the review of data quality comprising the following. Table 1 Stage 1 Stage 2 Stage 3 Management arrangements A review to assess the corporate management arrangements for data quality and whether these are being applied in practice. The findings contribute to the auditor's conclusion under the Code of Audit Practice on the council's arrangements to secure value for money (the VFM conclusion). Analytical review An analytical review of 2007/08 BVPI data and selection of a sample for testing based on risk assessment. Data quality spot checks In-depth review of a sample of 2007/08 PIs, most of which come from a list of specified BVPIs used in CPA, to determine whether arrangements to secure data quality are delivering accurate, timely and accessible information in practice. 19 All three stages of the review have been carried out at this Council. Guildford Borough Council 6

7 Management arrangements Management arrangements 20 Data quality has become more integrated into the Councils planning, monitoring and reporting processes and the Council has demonstrated a commitment to improve its data quality arrangements at a corporate level. 21 Good quality data is not consistent throughout the organisation and the Council should provide members with an assurance that each department has reliable arrangements to support the accurate recording of data, and that relevant staff are trained in the six dimensions of data quality (accuracy, validity, reliability, timeliness, relevance and completeness). Governance and leadership 22 The Council has clear data quality objectives which are documented in a policy and held electronically on the performance information system (Prime). The Council has now integrated risk management, the corporate plan, performance management and data quality onto the prime system. This enables staff and Members to understand the relevance of data quality within corporate objectives. 23 The Council responded to the recommendations reported in our 2006/07 Data Quality review and in 2007/08 it made some improvements to its monitoring and review of performance indicators. Policies 24 Clear corporate objectives have been set by the Council for improving performance management and data quality. An organisational policy for data quality has been in place for two years and is supported by guidance on the Prime system. 25 The Council does not have comprehensive and up to date departmental procedures and guidance notes. This was evident last year and in this year's testing of three performance indicators. The Council needs to produce clear guidance at service level to improve the accuracy of performance indicators and to enable staff to understand local and national requirements. Recommendation R1 Update departmental guidance and procedures to support the reliability, accuracy and completeness of data. Systems and processes 26 The Council has introduced a system for integrating performance information into the wider business planning and management processes. The Council has adequate controls to enable the performance information system to produce data that is fit for purpose. 7 Guildford Borough Council

8 Management arrangements 27 The Council has arrangements to regularly test its business critical performance information systems to satisfy itself that processes are secure. Business recovery reports are made to both management and members eg through the Members T-Government Working Group, and there are plans to address any identified weaknesses. 28 The Council has developed a standard phrase on data quality to be incorporated into any new contract with third parties. The two main sources of data received from third parties are from the Surrey Community Safety Unit and the Spectrum leisure centre. An information sharing protocol from Surrey Community Safety Unit helps facilitate the exchange of information between them and the Surrey local authorities. 29 In 2007/08 Internal Audit has begun a rolling programme of reviews of data quality supporting performance indicators. Over time this should help identify weaknesses and drive improvement at a service level. People and skills 30 The Council has put in place some corporate level arrangements to improve awareness among staff of their roles in achieving data quality. The Council has introduced a standard clause on all job descriptions, which specifies timely and accurate data. There has also been an increased awareness of the importance of data quality through the PI testing carried out by Internal Audit during 2007/ The Council has not provided specific training at service level on data quality in 2007/08. This may be contributing to the inaccuracies found in our testing in our sample of performance indicators. A significant key to ensuring accurate and reliable data is to provide staff with up to date procedures and guidance and train them in their use. Recommendation R2 Develop training to support staff in understanding the six dimensions of data quality (accuracy, validity, reliability, timeliness, relevance and completeness), and to improve their understanding of departmental procedures and guidance. Data use and reporting 32 The Council continues to employ arrangements for senior management to routinely and actively use data supporting performance information to plan services and allocate resources. The Prime system has enabled the council to actively manage performance data. There is evidence that management action is taken to address service delivery issues identified by data returns and performance information reports. Members have available to them high level information with which they can assess delivery of services against agreed milestones. Guildford Borough Council 8

9 Management arrangements 33 The Council relies on service level validation procedures to support the accuracy of data used in reported performance indicators. The Council reported a mismatch between the central return from the CLG (Department for Communities and Local Government) and the Prime figures for BVPI's 109a-c speed of processing planning applications. The Council correctly identified the difference through their validation procedures. However, we found the error was at service level and this was again due to a misinterpretation of the PI definition. Although the only corrective action needed was on the Prime system, the Council should update guidance for staff to improve data quality. 9 Guildford Borough Council

10 Analytical review Analytical review 34 This analytical review is the second part of a three-stage approach developed by the Audit Commission. After BVPI data is submitted we carry out a preliminary analysis on the PIs. This preliminary analysis includes: a variance analysis against previous years' data; and a reasonableness test which examines whether submitted data falls within expected ranges of values. 35 The analytical review of 2007/08 BVPIs found variances outside normal parameters in five PIs, shown in Table 2. Table 2 BVPIs outside expected range 2007/08 PI Assessment Comment BV 184a Proportion of LA homes Fairly stated Reviewed in stage 3 which were non-decent BV 184b Percentage change in Fairly stated Links to 184a. the proportion of non decent homes BV 199a Local Street and Real performance improvement Environmental Cleanliness Litter and Detritus BV 199b Local Street and Real performance improvement Environmental Cleanliness graffiti BV 79b(i) Percentage of recoverable overpayments recovered (HB) Real performance improvement 36 BV184a and b were assessed in stage 3 and the remaining three were all fairly stated. Guildford Borough Council 10

11 Data Quality spot checks Data Quality spot checks 37 We assessed three BVPIs in stage 3: BVPI 78a HB/CT benefit time to process new claims; BVPI 78b HB/CT benefit time to process changes of circumstance; and follow up of last years review on BVPI 184a Proportion of non decent homes. BVPI 78a HB/CT benefit time to process new claims' 38 BVPI 78a was fairly stated. 39 The Council has good arrangements to generate BVPI 78a 'time to process new claims'. We found only one error out of 31 cases tested and the arrangements to calculate the indicator reflected good practice: data is captured as quickly as possible on the benefits system; claims are subject to a quality assurance processes; with both external audit and departmental check lists providing the assurance; data is recorded and used in compliance to DWP guidance; and performance is checked. BVPI 78b HB/CT benefit time to process change of circumstance. 40 BVPI 78b was not fairly stated. 41 The Council did not follow the correct definition for collecting information for claims where there was a change of circumstance. The Council updated information based on information received via telephone, whereas the 2007/08 the definition required notifications to be physically received. We found three cases in the 31 tested where the Council was counting telephone calls. The definition changed for 2008/09 and telephone calls can now be counted in the new indicator. However, consistency with the 2007/08 definition and with other councils leads us to qualify the indicator in 2007/ Benefit staff are not supported by clear and up to date guidance (including departmental procedures) on the PI, which may have contributed to misinterpretation of the definition. The Council has a system that relies heavily on the knowledge and competence of the staff in post. Formal and documented procedures would ensure a consistent approach to processing claims and increase the accuracy of data used in performance management. 43 The Council's method of calculating the indicator does not follow best practice. The Council has used a spreadsheet calculation based on a random sample of 500 cases. The spreadsheet approach is used both to correct inputting errors to the benefits system and to correct anomalies within the benefits system. We found a number of errors in the source data, but these were corrected in the spreadsheet. 11 Guildford Borough Council

12 Data Quality spot checks 44 Consistent, accurate and timely data can only be achieved using processes that capture data as close to the point of activity. The Council should focus on eliminating inputting errors and system anomalies which will help improve the accuracy of data and generate a more accurate indicator for monitoring performance using the source data from the system. Recommendation R3 Ensure benefits department staff are supported by clear and up to date guidance, including departmental procedures to ensure that data is more reliable, consistent and complete. R4 Capture data on processing changes of circumstance as close to the point of activity, and adhere to the principle of 'getting it right first time'. R5 Focus on proper compliance with the definition of the new indicator in 2008/09. BVPI 184a Proportion of non decent homes 45 The Council has taken positive steps to improve the record - keeping and monitoring of stock data used to calculate BVPI 184a. The Council has actioned the recommendations made last year and has introduced an internal check for the figure of homes made decent in 2007/08. There is greater scrutiny of the figures and this was evidenced in the detailed analysis of the Rand survey and adjustment made by the Council to ensure the non decent baseline figure was accurate. 46 We found inconsistent figures to support the calculation of this PI. Much of the evidence came from the adjustment of the total non decent home baseline, and although the figures have been agreed as materially accurate we recommend that a clearer audit trail be established. Recommendation R6 Establish a clear audit trail for BVPI 184a Proportion of non decent homes to give greater assurance to senior staff that the figures are reliable and accurate. Guildford Borough Council 12

13 Appendix 1 Action plan Appendix 1 Action plan Page no. Recommendation 7 R1 Update departmental guidance and procedures to support the reliability, accuracy and completeness of data. 8 R2 Develop training to support staff in understanding the six dimensions of data quality (accuracy, validity, reliability, timeliness, relevance and completeness), and to improve their understanding of departmental procedures and guidance. 12 R3 Ensure benefits department staff are supported by clear and up to date guidance, including departmental procedures to ensure that data is more reliable, consistent and complete. Priority 1 = Low 2 = Med 3 = High Responsibility Agreed Comments Date Head of Corporate Development Head of Corporate Development SW as Head of Service and GW as Benefits Manager Agreed Agreed Agreed Develop a template for individual services to use to produce service specific guidelines and roll out across the authority to for the production of NIs and Key Local Indicators. Further to Rec. 1- develop a programme of meetings with Heads of Service, their management teams and officers responsible for PIs to set out roles and responsibilities with regards to Data Quality. It is accepted that the errors highlighted during the audit were primarily caused by an agency temp who has since departed. It is planned to improve guidance and develop procedures once the additional increase in workload and additional claims has subsided. Targeted for summer /3/09 30/06/09 31/08/09 13 Guildford Borough Council

14 Appendix 1 Action plan Page no. Recommendation 12 R4 Capture data on processing changes of circumstance as close to the point of activity, and adhere to the principle of 'getting it right first time'. 12 R5 Focus on proper compliance with the definition of the new indicator in 2008/ R6 Establish a clear audit trail for BVPI 184a Proportion of non decent homes to give greater assurance to senior staff that the figures are reliable and accurate. Priority 1 = Low 2 = Med 3 = High Responsibility Agreed Comments Date SW as Head of Service and GW as Benefits Manager SW as Head of Service and GW as Benefits Manager Head of Neighbourhood and Housing Management Services Agreed Part of the procedures implementation referred to in R3 above will apply to accuracy of input. The other actions to be implemented will be: 1. Redefine the work of Haines Watts (independent verification auditors) to address accuracy checking of appropriate work (with immediate effect), and 2. Work with software provider to improve the accuracy of releases so that there is less reliance on manual checking (ongoing). The new indicator permits changes via the telephone and therefore this should not present a problem from now on. We are also responding to the transformation agenda by considering a telephone only application service, which will hopefully speed up the claims process and also contain costs /01/09 2. Monitor Qtly 31/03/09 Agreed Full audit trail to be maintained. 31/03/09 Guildford Borough Council 14

15 The Audit Commission The Audit Commission is an independent watchdog, driving economy, efficiency and effectiveness in local public services to deliver better outcomes for everyone. Our work across local government, health, housing, community safety and fire and rescue services means that we have a unique perspective. We promote value for money for taxpayers, auditing the 200 billion spent by 11,000 local public bodies. As a force for improvement, we work in partnership to assess local public services and make practical recommendations for promoting a better quality of life for local people. Copies of this report If you require further copies of this report, or a copy in large print, in Braille, on tape, or in a language other than English, please call Audit Commission 2009 For further information on the work of the Commission please contact: Audit Commission, 1st Floor, Millbank Tower, Millbank, London SW1P 4HQ Tel: Fax: Textphone (minicom):