Audit Summary Report March 2007 Data Quality Audit 2006/2007
External audit is an essential element in the process of accountability for public money and makes an important contribution to the stewardship of public resources and the corporate governance of public services. Audit in the public sector is underpinned by three fundamental principles. Auditors are appointed independently from the bodies being audited. The scope of auditors' work is extended to cover not only the audit of financial statements but also value for money and the conduct of public business. Auditors may report aspects of their work widely to the public and other key stakeholders. The duties and powers of auditors appointed by the Audit Commission are set out in the Audit Commission Act 1998, the Local Government Act 1999 and the Commission's statutory Code of Audit Practice. Under the Code of Audit Practice, appointed auditors are also required to comply with the current professional standards issued by the independent Auditing Practices Board. Appointed auditors act quite separately from the Commission and in meeting their statutory responsibilities are required to exercise their professional judgement independently of both the Commission and the audited body. Status of our reports The Statement of Responsibilities of Auditors and Audited Bodies issued by the Audit Commission explains the respective responsibilities of auditors and of the audited body. Reports prepared by appointed auditors are addressed to non-executive Members or officers. They are prepared for the sole use of the audited body. Auditors accept no responsibility to: any Member or officer in their individual capacity; or any third party. Copies of this report If you require further copies of this report, or a copy in large print, in Braille, on tape, or in a language other than English, please call 0845 056 0566. Audit Commission 2007 For further information on the work of the Commission please contact: Audit Commission, 1st Floor, Millbank Tower, Millbank, London SW1P 4HQ Tel: 020 7828 1212 Fax: 020 7976 6187 Textphone (minicom): 020 7630 0421 www.audit-commission.gov.uk
Data Quality Contents 3 Contents Summary report 4 Background 4 Scope and objectives 5 Main conclusions 5 Management arrangements 6 Completeness check 8 Review of performance indicators 8 Audit process 10 Appendix 1 Action plan 11
4 Data Quality Summary report Summary report Background 1 Public services need reliable, accurate and timely information with which to manage services, inform users and account for performance. Service providers make many, often complex, decisions about their priorities and the use of resources. Service users and members of the public more widely need accessible information to make informed decisions. Regulators and government departments need good quality information to make judgements about performance and governance. 2 Increasing reliance is being placed on performance information in assessment regimes including Comprehensive Performance Assessment and the need for reliable data has become more important. Also, from 2005/06 auditors are giving opinions on whether councils' have proper arrangements for securing economy, efficiency and effectiveness in their use of resources and one of the twelve criteria reviewed is data quality. 3 Good quality data is the essential ingredient for reliable decision making. The data used to report on performance must be fit for purpose, and be accurate and timely. Public bodies can improve the quality of their data by identifying the performance information that is important to them and their stakeholders, and securing the quality of the data to support these information needs. This is more likely if the performance information is routinely used for the day to day planning and management of services, and the people who collect the data understand its importance.
Data Quality Summary report 5 Scope and objectives 4 The Audit Commission has developed a three-stage approach to the review of data quality comprising. Management arrangements A review to determine whether proper corporate management arrangements for data quality are in place, and whether these are being applied in practice. The findings contribute to the auditor's conclusion under the Code of Audit Practice on the council's arrangements to secure value for money (the VFM Conclusion) Completeness check An arithmetic check and analytical review of specified Best Value Performance Indicators (BVPIs). Data quality spot checks In-depth review of a sample of performance indicators (PIs) selected following our analytical review and risk assessment (from a list of specified BVPIs and non-bvpis) to determine whether arrangements to secure data quality are delivering accurate, timely and accessible information. Main conclusions 5 The Council's management arrangements for data quality are satisfactory. Data quality systems and processes are robust but they are not yet being applied consistently. The Council has effective arrangements for the governance, monitoring and review of data quality, although these are not formalised in an overarching data quality strategy. 6 Our review and spot checks of eight specified PIs revealed that all had been fairly stated. Two required amendment before final submission.
6 Data Quality Summary report Management arrangements 7 The Council has effective arrangements for the governance, monitoring and review of data quality. These are not formalised in an overarching data quality strategy, but they are being applied in most cases. Responsibility for ensuring that data quality is good is at a senior level in each directorate and co-ordinated through the Corporate Performance Group chaired by the Deputy Leader and attended by the Director for Policy and Corporate Services. Areas of concern in the collection of data are reported to Members and action has been taken. For example improvements have been made in the reporting of sickness absence through reviewing and replacing the HR system for collecting information. 8 The Council has communicated individual, team and directorate responsibility for ensuring the quality of data collected and used to report performance. Responsibilities for data quality are evident in some job descriptions but this is not consistent across the Council. Individual targets in relation to data quality have not been set across all directorates. There is no corporate training provided for either staff or managers in relation to data quality. There is a framework in place for monitoring the quality of data at a local level but there is no formal programme of data quality review. 9 The Council s data quality policy framework is underdeveloped. The Council does not have a separate policy and procedures relating specifically to data quality. The Council has an embedded performance management framework 'Cutting the Mustard' which provides a framework for collection and reporting of data. However, this framework does not set out objectives for staff in relation to improving the quality of data that is collected. 10 The Council has data quality systems and processes in place to ensure the accuracy of the data used to produce the corporate performance information but these vary between departments. Internal audit reviews have not focused on the quality of the data and therefore are unable to provide assurance to the Council. Systems for ensuring accuracy are being applied across directorates but require some manual adjustments to produce the information required. 11 The Council has a robust system of internal control and validation of performance information. Checking for accuracy takes places at a number of levels including team managers, departmental managers and directors before being submitted to the policy team. Part of the remit of the Corporate Performance Group is to ensure consistency and sharing of good practice across directorates in relation to data quality but our evidence shows that this is not yet fully effective. Processes are in place to ensure that data returns are supported by clear and complete audit trails.
Data Quality Summary report 7 12 Appropriate controls and business continuity plans are in place to ensure that the Council's information systems are secure but these vary across departments. Reviews of controls are not taking place regularly and are not reported to senior management. The Council has secure performance information systems backed up by an Information Security Policy which includes data quality. Robust access control systems are in place for the Council's business critical performance information systems. Procedures exist for all systems and are available electronically. The Council has some data sharing protocols in place for sharing information externally and internally but these are not in place across all departments. Compliance with legal and confidentiality standards is managed within the directorates and varies in its robustness. 13 The Council is using performance information to improve services. Senior management routinely use performance information to plan and allocate resources. Members use high level information to assess delivery of services in relation to agreed plans. Performance information is regularly used to identify deviations from planned performance, and there is timely action on performance shortfalls and follow-up to ensure action has been taken. As a result improvements have been achieved in sickness levels, council tax collection and void levels. Data is also used to assess the quality of services for example in the contact centre service users are invited to comment on the quality of the Council's response through a call back system. Recommendations R1 Develop a Council wide data quality strategy. R2 Ensure responsibilities for data quality are consistently included in job descriptions. R3 Ensure targets and standards for data quality are set and staff are assessed against these. R4 Provide specific data quality training for appropriate staff and managers. R5 Ensure the effectiveness of controls to validate data across all departments is reviewed annually with reports going to senior management. R6 Ensure all instances of data sharing both internally and externally are identified and that appropriate protocols are in place. R7 Review management of legal and confidentiality standards by directorate.
8 Data Quality Summary report Completeness check 14 The calculations and completeness for the following eight selected BVPIs and ten non-bvpis were checked and found to be complete and within parameters set centrally by the Audit Commission, subject to validation in Stage 3 spot checks. BVPIs: Planning speed - BV109, Speed in fixing street lights - BV215, Percentage of pedestrian crossings with facilities for disabled people - BV165, Recycling and Composting performance - BV82a and b, Non-decent homes - BV184a, Bed and breakfast and Hostel temporary accommodation - BV183a and b. Non-BVPIs: Assessment of users aged 16 and over of the library service, library stock turn, library stock level, cost per library visit, service users who have moved on in a planned way from temporary living arrangements, average re-let times, planned to responsive housing repairs, private sector homes vacant for greater than six months, private sector unfit properties made fit, and repeat homelessness. 15 The Council's management arrangements for data quality were assessed as medium risk. This resulted in the selection of 8 of the 18 specified PIs for further testing - 3 non-bvpis in libraries, 1 non-bvpi in housing, 2 environmental BVPIs, 1 planning BVPI and 1 accommodation BVPI. 16 The remaining 10 indicators out of the 18 specified were judged to be lower risk and we carried out no further. The Council was informed of our selection before we carried out our spot checks. Review of performance indicators 17 The following PIs were reviewed using a series of standard Audit Commission spot checks and audit tests. Culture Stock turn - book issues per 1,000 population/books per 1,000 population (Institute of Public Finance and Accountancy PI). Stock level per 1,000 population (Institute of Public Finance and Accountancy PI). Cost per library visit (Institute of Public Finance and Accountancy PI). Environment Planning speed (BVPI109). Recycling performance (BVPI 82a). Composting performance (BVPI 82b).
Data Quality Summary report 9 Housing Average time in temporary accommodation (B&B) (BVPI183a). Planned to responsive repairs (Housing Investment Programme Business Plan Statistical Appendix). 18 All three culture PIs reviewed were fairly stated and no points were noted. 19 The planning speed BVPI was fairly stated. The recycling and composting BVPIs were fairly stated after revision during the review. 20 The recycling and composting BVPIs were fairly stated but the following points were noted. The Council's original submission was amended as unsupported estimates had been used for the exclusion of trade waste from the total waste figure. These were amended to ensure only reliable estimates were used. Records were not complete for batteries and cans recycled. This was not material to the indicator total so no reservation was required. The audit trail from the indicator figures to supporting records was difficult to follow and caused delays to the audit. Reports from the Council's waste management system (SWIS) took a long time to produce and the variety of reports available is limited which delayed our testing. Recommendations R8 Maintain an electronic record (spreadsheet) of all supporting documents used when calculating the recycling and composting indicators and ensure complete records are maintained. R9 Review the waste management computer system (SWIS) to determine whether improvements can be made, allowing reports to be run quicker and with a greater choice of data reported. 21 The percentage of planned to responsive repairs PI was found to be fairly stated. 22 The average time in bed and breakfast accommodation BVPI was fairly stated. We found one error out of a sample of 15 during testing when an incorrect date had been entered during adjustments to the source data. This error was investigated and considered an isolated case. Correcting the error would not have changed the indicator result. Recommendation R10 Reduce the need to make manual adjustments to OHMS accommodation data by modifying system reports parameters.
10 Data Quality Summary report Audit process 23 It is noted that 'self assessment' forms (currently done by the Council for BVPI data only) are a useful tool in our analysis of BVPI data. Due to the inclusion of 10 non BVPI indicators in the 18 indicators specified for audit we recommend 'self assessment' forms be completed for these 10 indicators for the 2006/07 Data Quality Audit. Recommendation R11 Produce 'self assessments' for the 10 specified non-bvpi indicators for the 2006/07 Data Quality Audit.
Data Quality Appendix 1 Action plan 11 Appendix 1 Action plan Page no. Recommendation 7 R1 Develop a Council wide data quality strategy. 7 R2 Ensure responsibilities for data quality are consistently included in job descriptions. 7 R3 Ensure targets and standards for data quality are set and staff are assessed against these. 7 R4 Provide specific data quality training for appropriate staff and managers. 7 R5 Ensure the effectiveness of controls to validate data across all departments is reviewed annually with reports going to senior management. 7 R6 Ensure all instances of data sharing both internally and externally are identified and that appropriate protocols are in place. Priority 1 = Low 2 = Med 3 = High Responsibility Agreed Comments Date 3 Carole Parnell Yes The Policy Unit has this work in hand and is on target to meet the deadline. 2 Carole Parnell Yes In line with the implementation of single status the Council is moving towards a system of generic, competency based job descriptions. This recommendation will be considered in light of that approach. 2 Carole Parnell Yes Starting in April the Council is introducing a new scheme for managing employee performance (replacing the old Job Review Scheme) and a new corporate competency framework. The expectation is that staff s individual performance targets will address such issues, where appropriate. 2 Carole Parnell Yes Recommendation addressed. The Policy Unit arranged the first such training session on 7 March 2007. Others are planned throughout the year. 2 Carole Parnell Yes This will be included in the new Data Quality Strategy (see recommendation R1 above). June 2007 June 2007 April 2007 End June 2007 2 Carole Parnell Yes As above. End June 2007
12 Data Quality Appendix 1 Action plan Page no. Recommendation 7 R7 Review management of legal and confidentiality standards by directorate. 9 R8 Maintain an electronic record (spreadsheet) of all supporting documents used when calculating the recycling and composting indicators and ensure complete records are maintained. 9 R9 Review the waste management computer system (SWIS) to determine whether improvements can be made, allowing reports to be run quicker and with a greater choice of data reported. 9 R10 Reduce the need to make manual adjustments to OHMS accommodation data by modifying system reports parameters. 10 R11 Produce 'self assessments' for the 10 specified non-bvpi indicators for the 2006/07 Data Quality Audit. Priority 1 = Low 2 = Med 3 = High Responsibility Agreed Comments Date 2 Carole Parnell Yes A whole Council approach needs to be adopted and this will be addressed through CMT. 3 Carole Parnell Yes Recommendation addressed. 2 Carole Parnell Yes New weighbridge software has been introduced at the Council's nursery, and ICT and Waste Managers are currently reviewing the benefits of introducing similar software at Factory Lane and replacing the current SWIS system. 2 Carole Parnell Yes Recommendation addressed. The Audit Commission has agreed an approach with Housing which includes rewriting 'Infomaker' report to specify relevant cases for this BVPI. 3 Carole Parnell Yes The Policy Unit will ensure that the Council s current good practice of producing self-assessment for BVPIs will be extended as part of the 2006/07 (and subsequent) data quality audit. End March 2008 End March 2008 June 2007