Data Quality Audit Commission report

Transcription

1 Wards Affected: Item No. Audit Committee 12 th December 2008 Report of the Deputy Chief Executive/Corporate Director of Resources Data Quality Audit Commission report 1.0 Purpose of this report This report summarises the key findings and recommendations from the Audit Commission s audit of 2007/08 data quality (DQ) arrangements. The Audit Commission s full report is attached along with the Council s DQ improvement action plan which responds to each of the recommendations in the report. 2.0 Recommendations Members of the Audit Committee are asked to: Consider and comment on the attached Audit Commission s DQ audit, Consider and approve the DQ action plan (Appendix A) which responds to the recommendations in the report, Note (as identified by the Audit Commission) the progress made in embedding the corporate approach to DQ, the positive impact of departmental leads and a well structured approach ensuring data is more robustly captured and reported. Note the intention to provide a progress update on implementing these improvement actions in April/May Reason for consideration 3.1 The Audit Committee has a key role in evaluating the results of audits and ensuring that robust arrangements are in place to implement and track improvement actions. 4.0 Background 4.1 Effective and robust DQ arrangements are critical in ensuring decisions are based on reliable, accurate and timely information. Without this there is a risk of flawed decision and policy making, wasted resources and poor services and there is also a danger that good performance may not be recognised, rewarded and promoted.

2 4.2 Data quality figures strongly both within the existing Comprehensive Performance Assessment (CPA) framework and the future CAA as part of the Organisational Assessment. 4.3 The DQ agenda has moved on significantly in recent years. The following table outlines the changes: From Only BV PIs Nottingham City Council Centrally imposed End of year exercise Detective, catching mistakes after they ve been made To All data financial, performance Nottingham City Council and partners Corporately/departmentally owned Rolling programme Directive/ preventative, identifying weaknesses and addressing them, right first time, 5.0 Overview of Audit Commission findings 5.1 The report highlights the Council s progress in developing and embedding a corporate approach to data quality. The Audit Commission s comment that the Councils overall management arrangements for ensuring data quality are consistently above minimum requirements (para 6) represents an improvement on last year and is attributable to a sustained commitment to data quality both at an operation and strategic level. 5.2 The report highlights a number of areas where we have made improvement across all aspects of DQ, but also identifies a number of areas where further progress is needed. These are detailed in each of the sections of the report and summarised in Appendix A together with the Council s response to each of the six recommendations made (Appendix B). 5.3 This improvement action plan has been prepared and agreed by the Council s Strategic Performance Group. Implementation of the improvement actions will bring about further improvement to our DQ arrangements and position the Council (and its partners) well in terms of DQ and the new CAA performance framework. 5.4 An update report detailing progress on implementing improvement actions will be brought back to the Audit Committee in April/May 2009.

3 6.0 Background papers other than published works or those disclosing exempt or confidential information Nottingham City Council Data Quality Policy/Guidance document 7.0 Published documents referred to in compiling this report None Carol Mills-Evans Deputy Chief Executive/Corporate Director Resources Contact Officer: Julie Rankin Director Performance & Scrutiny Simon Burton Corporate Performance & Quality Officer List of attachments: Audit Commission 2008/09 DQ report Appendix A DQ Action Plan

4 Data Quality Nottingham City Council Audit 2008/09 Date

5 Contents Introduction 3 Detailed findings 5 Appendix 1 Action Plan 13 Status of our reports The Statement of Responsibilities of Auditors and Audited Bodies issued by the Audit Commission explains the respective responsibilities of auditors and of the audited body. Reports prepared by appointed auditors are addressed to non-executive directors/members or officers. They are prepared for the sole use of the audited body. Auditors accept no responsibility to: any director/member or officer in their individual capacity; or any third party.

6 Introduction Introduction 1 The purpose of this report is to summarise the findings from our work on data quality for 2007/08. 2 Auditors work on data quality and performance information supports the Commission s reliance on performance indicators (PI) in its service assessments for comprehensive performance assessment (CPA). 3 Our work on data quality is complemented by the Audit Commission s paper, 'Improving information to support decision making: standards for better quality data. This paper sets out standards, for adoption on a voluntary basis, to support improvement in data quality. The expected impact of the Audit Commission's work on data quality is that it will drive improvement in the quality of local government performance information, leading to greater confidence in the supporting data on which performance assessments are based. Scope of our work 4 We have followed the Audit Commission's three-stage approach to the review of data quality as set out in Table 1. Table 1 Data quality approach Stage 1 Stage 2 Stage 3 Management arrangements A review using key lines of enquiry (KLOE) to determine whether proper corporate management arrangements for data quality are in place, and whether these are being applied in practice. The findings contribute to the auditor's conclusion under the Code of Audit Practice on the Council's arrangements to secure value for money (the VFM conclusion). Analytical review An analytical review of 2007/08 BVPI and non-bvpi data and selection of a sample for testing based on risk assessment. Data quality spot checks In-depth review of a sample of 2007/08 PIs all of which come from a list of specified BVPIs and non-bvpis used in CPA, to determine whether arrangements to secure data quality are delivering accurate, timely and accessible information in practice. For 2007/08 PI spot checks, the Audit Commission specified that it is compulsory to review two housing benefits PIs at all single tier and district councils as a minimum. Audit Commission data quality audit guidance Nottingham City Council

7 Introduction 5 As this is the third year of applying this approach to data quality, we tailored our work to focus on the key changes and actions taken to address previously identified weaknesses and recommendations. Summary conclusions Stage 1 Management arrangements 6 The Council's overall management arrangements for ensuring data quality are consistently above minimum requirements. 7 There has been good progress in developing and embedding a corporate approach to data quality. Departmental leads are now having a positive impact on the quality of data, and there is a well structured approach that ensures data is more robustly captured and reported. Stage 2 Analytical review 8 Our analytical review work at Stage 2 identified that most of the PI values we reviewed fell within expected ranges. Where this was not the case an explanation has been sought and given confirming changes in performance relate to real improvement or deterioration. In a few cases we have established that changes in values relate to methodological changes. A selection of five PIs was identified through the analytical review process for stage 3 spot checks. Stage 3 Data quality spot checks 9 Our review and spot checks of PIs found all to be fairly stated, although we have also provided more detailed feedback on specific PIs to assist the Council strengthening management arrangements. 10 An action plan has been agreed with the Council (see Appendix 1) to address the issues arising from this review. Nottingham City Council 4

8 Detailed findings Detailed findings Management arrangements (Stage 1) 11 Overall, the Council s corporate arrangements for data quality are consistently above minimum requirements. Governance and leadership 12 The Council's approach to assuring and improving data quality is clearly set out in a corporate strategy, comprising DQ objectives, a set of policies and an action plan. This data quality strategy is kept up to date, is well laid out, and is available to all staff on dedicated pages of the Council's intranet site. The strategy is supported by a set of practical tools, including a risk assessment matrix and a DQ e-learning module. This outlines the Council's intentions for, and clear commitment to data quality. 13 Appropriate governance arrangements are in place to ensure policies are followed and improvement actions progressed. Data quality sits within the Deputy Leader's performance management portfolio. A senior officer has primary responsibility for overseeing data quality, and two cross cutting working groups focus on performance arrangements at a strategic and operational level. However, at the strategic level, the strategy and performance group (SPG) has not monitored delivery of the action during 2007/08. In addition, while this group has adopted the strategy, departmental management teams have not. This means implementation of the action plan during 2007/08 has been a line managerial responsibility and not a departmental or corporate responsibility. 14 Accountability for PIs is clear in most cases, although accountability is less clear where PIs rely upon cross departmental input. A senior officer (the responsible officer) is accountable for collating evidence, for monitoring and reporting each PI accurately on a regular basis. Responsibility for the data quality for groups of PIs within each department rests with departmental performance officers (PMIDG) who liaise well with DMTs, teams and PI officers across their department. Some departmental DQ arrangements are better established than in others and so each performance officer faces a different set of challenges. All have made progress during 2007/08 in assessing risk, in planning a programme of verification work, in promoting data quality and in providing some degree of added assurance. 15 Most departmental management teams are providing clear governance on data quality, for example by agreeing programmes of quality assurance activity to be undertaken within the department, and ensuring the outcomes of DQ reviews are reported back to the DMT. This is particularly effective where DMTs dedicate time routinely to in-depth performance reporting. 16 Data quality is not yet full integrated within key plans and strategies. Data quality is not yet seen as a distinct aspect of performance management, although some departments are beginning to provide a routine commentary on data quality alongside 5 Nottingham City Council

9 Detailed findings performance information. Individual and team plans do not routinely set out clear data quality objectives. 17 Performance officers have focused on developing departmental and interdepartmental arrangements, but there has been no systematic development of arrangements to improve the data quality that involves local external partners. Recommendation R1 Raise the profile of data quality improvement by ensuring departmental management teams adopt the corporate strategy and monitor of delivery their own DQ improvement agenda. Policies 18 A comprehensive set of policies has been agreed and published on the Council's intranet, and this is promoted to those responsible for data quality. Policies cover developing and implementing a risk based programme of verification work, a risk assessment matrix that takes account of the control environment and systems, and extensive guidance to support the end-of-year audit sheet compilation process. 19 Officers from the performance improvement and policy (PIP) team support departmental performance officers through the PMIDG which meets monthly, through one to ones, and proactively when deadlines are approaching in order to help implement policies. PMIDG members also help develop policies in order to make them effective and to avoid duplication of effort. This ensures there is effective collaboration between the departmental and corporate DQ champions. 20 Policies are mostly clear. An exception is the policy relating to the annual audit sheet collection process. This requires a significant concerted effort from data owners, responsible officers, departmental performance officers and corporate PIP officers. However, the policy is not clear on the rationale for requesting all the information in the pro-forma, what purpose this information serves, who is to take account of it, and to what depth. There is a risk that the process is overly onerous for the level of assurance that it provides. 21 Departmental performance officers have a variable knowledge of the extent to which data quality is exposed to risks arising from the management arrangements of partner organisations upon which they rely. There has been limited formal assessment of these risks, and policies are at an early stage of development. This relates to a key activity in the data quality action plan for Recommendation R2 Review the use of end of year audit sheets, clarifying what information is needed and why, and the purpose to which it is put. Nottingham City Council 6

10 Detailed findings Systems and processes 22 Arrangements for recording and reporting data are becoming integrated into the Council s wider business management processes, in order to support officers and councillors in their day to day work. For example, the corporate 'PerformancePlus' performance management system ensures all responsible officers, and performance managers are able to feed nominated PIs into a formal corporate reporting process, and timeliness is improving as a result. This system provides regular reports to corporate and increasingly, service directors and DMTs. This continues to develop as new functions of PerformancePlus are rolled out across the Council and external partners. There are also clear plans for it to provide a platform for reporting on the quality of data as well as data values. 23 High level risk assessments of PIs have been successfully completed in all departments, but limited progress has been made in conducting in-depth risk assessments. With very few exceptions, performance managers have undertaken a high level risk analysis of PIs, based upon the level of scrutiny a PI has received in the past and the impact of it being inaccurately stated. This prioritisation process informs the programmes of verification work. However performance managers are not consistently clear on the extent to which 'checks and balances' are already built into and applied to local data collection by the responsible officer and data owner, for example, what level of manual manipulation and data cleaning is currently necessary and what verification work is undertaken. An understanding of such issues is necessary to assess the strength of the control environment and to address vulnerable areas, in line with the data quality strategy. 24 All performance officers, within PIP and the other departments have a good understanding about the scope and outcome of the work that Internal Audit does. In all cases departmental data quality work takes account of recent internal audit work (2005/06 and 2006/07 audits) and DMTs have maintained their focus on any recommendations arising. This increases the impact of internal audit work. 25 In some departments however, there is a risk of over-reliance on the work of Internal Audit, in lieu of assurance work beginning within the department. While all departments are developing programmes of in-year verification work, and the majority of departments have a good track record in implementing such reviews, two departments were not able to implement any in-year work during 2007/ A high level review of controls for all 250 specified PIs (those that define progress in relation to the corporate plan, the direction of travel and CPA assessments) is undertaken at least annually. This is achieved through the annual audit sheet return process. Responsible officers and departmental performance officers sign off summaries of PI values, method statements, and explanations of variances, and working papers for these, on a risk basis, are sample desk top reviewed. There is a good level of compliance for this corporate task. While some issues are dealt with as they arise at various stages of the process, there is no summary feedback on this process to the PMIDG officers, and so learning opportunities are being lost. 27 Business continuity plans are reviewed as part of annual service planning and when substantive testing is undertaken on particular PIs. The Council has a corporate business continuity plan in place. This is underpinned by departmental business 7 Nottingham City Council

11 Detailed findings continuity plans. These plans include data loss. Performance Plus is supported by Corporate IT's own business continuity and resilience plans. Data is regularly backed up and stored off site and can be restored by a third party. 28 In relation to management arrangements for specific PIs, bespoke spreadsheets are used frequently for data manipulation, but there is a general lack of use of the basic functions that are able to be built into spreadsheets in order to safeguard, manipulate and report data efficiently and effectively. Recommendation R3 Build on the sound initial, high-level risk departmental assessments of PIs by focusing on risks within the control environmental relating to each PI. People and skills 29 Roles and responsibilities of strategic management and operational staff in relation to data quality are clearly defined in the data quality strategy. Where a data quality role comprises a significant part of an officer's responsibilities, there are data quality targets, and standards and this forms part of the personal development and review process with line managers up to director level. 30 The Council can demonstrate that it has an effective internal network of data quality champions that is successfully driving data quality improvement throughout the Council. Although much work remains to be done, the network of performance officers, including PIP officers, is able to identify a foundation of effective management arrangements influenced by corporate strategies and departmental approaches. 31 Through the PMIDG group the Council has reviewed the need for training to develop skills relating data quality. In the latter part of 2007/08 a formal programme of training has been developed, and launched with good uptake and feedback in March In 2007 an e-learning DQ module was launched on the intranet, and in one instance all performance officers in one department did the training initially in order to be able 'sell' it to their colleagues. Data quality has also been included in generic manager and finance for non-financial manager training courses, as well as performance plus training. In this way, training is not limited to officers with specific responsibilities for data input or data quality, and therefore more officers are becoming aware of corporate expectations and are able to contribute to improving arrangements. Recommendation R4 Continue to develop the focus on data quality skills and expertise within the network of departmental performance managers. Nottingham City Council 8

12 Detailed findings Data use and reporting 32 Performance information is regularly used to identify trends and deviations from planned performance. Departments routinely report PI performance on a monthly and quarterly basis, and some departments hold regular meetings dedicated to performance matters. Reports include targets versus in-year performance to date, a commentary to explain variances, and increasingly, remedial action and its intended impact, although year-end projections are not routinely reported. 33 The use of performance data to facilitate corporate performance management is being reviewed. The Audit Commission corporate assessment in 2007 identified that corporate performance reports were too long, didn t focus consistently on remedial action and performance information was not always timely. This is an area that the SPG has focused on during 2007/08 and progress is being made, for example, in providing a clearer quarterly corporate report format to councillors. 34 The Council's data validation procedures comprising internal audit reviews, reviews by data owners and responsible officers, departmental review and PIP annual review procedures are effective with few exceptions. The outcomes of these reviews identify few failings in systems and few inaccurately reported PIs. For example Internal Audit PI work in 2007/08 identified 16 PIs as compliant, 4 as compliant except for minor deficiencies and 2 as non compliant. Departmentally-based verification work has highlighted the need to: establish clear definitions, especially for newly agreed local PIs, with sufficient detail to enable data collection and audit; and move from process mapping (generally well achieved) to control mapping. 35 Our spot checks identify that the PIs we have audited are fairly stated, and where there have been significant changes in outturn year on year the Council has investigated these and the reasons given. In general management arrangements are sufficiently robust to ensure the risk of PI mis-statement is low. 36 However in the case of the HIP HSSA PI for empty private sector homes (6 months plus), we identify that management arrangements are weak due to: unclear accountability arising from cross departmental ownership, and changes in managerial responsibility; and a lack of collaboration and timeliness of input As a result management arrangements currently provide limited assurance, and therefore there is risk of mis-statement. Recommendation R5 Ensure management arrangements relating to the HIP HSSA PI (empty homes H18) and other PIs where there is a high dependency on cross departmental contributions include clear lines of accountability and adequate preparatory time to collaborate effectively. 9 Nottingham City Council

13 Detailed findings Analytical review (Stage 2) 37 An analytical review of the following BVPIs and non-bvpis was carried out. The findings are shown in the two tables below. Table 2 Analytical review findings Checks on a set of specified PIs 2007/08 Performance indicator BVPI 165 (pedestrian crossings BVPI 82a (dry recycling) BVPI 199b (graffiti) BVPI 78b (benefits processing changes) Cost per library visit HIP HSSA - private sector homes vacant for more than six months HIP HSSA - repeat homelessness Assessment Improved PI, (in 4th quartile last year). Change in performance exceeds variance and standard deviation (SD) thresholds. Improved PI (in 4th quartile last year). Change in performance exceeds variance thresholds. Deteriorating PI (in 4th quartile last year). Change in performance exceeded variance and SD thresholds, and close to max plausible value. Deteriorating PI, (in 3rd quartile last year). Change in performance exceeds variance thresholds. Deteriorating PI. Change in performance exceeds variance thresholds. Improving PI. Change in performance exceeds variance thresholds. Improving PI. Change in performance exceeds variance thresholds. Comment Performance attributed to real improvement and small number effect. Performance attributed to real improvement. Performance attributed to real performance deterioration, since increased incidence has outstripped any service improvement. Performance attributed to real performance deterioration. Performance attributed to real performance deterioration (higher costs and lower usage). Performance attributed to changes in methodology applied by Council. Performance attributed to real improvement. Audit Commission data audit quality guidance (12 specified PIs) Nottingham City Council 10

14 Detailed findings Table 3 Analytical review of all other BVPIs Checks on non-specified PIs 2007/08 Performance indicator BVPI 216b Contaminated land (Information) BV187 Condition of footways BV223 and BV224a condition of principal and other roads BV218a abandoned vehicles BV79b(iii) HB debt recovery Assessment Deteriorating PI. Change in performance exceeds variance thresholds. Improving PI. Change in performance exceeds variance and SD thresholds. Improving PI. Change in performance exceeds variance and SD thresholds. Improving PI. Change in performance exceeds variance thresholds. Deteriorating PI. Change in performance exceeds variance and SD thresholds. Comment Performance attributed to real performance and change in methodology applied. Performance attributed to real deterioration in performance on year before last. A year on year comparison cannot be made (due to alternating year methodology). Performance attributed to performance (25% of improvement) and to method (75% of improvement). Performance attributed to real improvement, but some previous understatement and current lower incidence. Performance partially attributed to real improvement and partially explained by the change in the level of current year debt. Audit Commission data quality audit guidance 38 All other PIs we reviewed were found to be complete, and within plausible and permissible values and where variance thresholds are exceeded for this to be due to changes in real performance. 39 Some departments, for example Adult Services Health and Housing (ASHH), are already beginning to report on the quality of data as well as the data value itself. This analytical review demonstrates that a commentary against specific PIs helps those using PI reports for management overview and strategic decision making to interpret more fairly changes in levels of performance. Recommendation R6 Continue to develop performance management arrangements, for example, departmental quarter performance reports and the PerformancePlus system, to include commentary on data quality as well as data values. 11 Nottingham City Council

15 Detailed findings Data quality spot checks (Stage 3) 40 A number of PIs were reviewed using a series of detailed spot checks and audit tests. Our findings are shown below. Table 4 Spot check findings Performance indicator Assessment Comment Housing Benefits BVPI 78a Housing Benefits BVPI 78b Culture Cost per library visit Environment (BV199) cleansing Housing/Environmental health - HIP HSSA - private sector homes vacant for more than six months Fairly stated (27.9 days) Fairly stated (12.1 days) Fairly stated ( 4.26) Fairly stated (7.4%) Fairly stated (3.30%) Tested on a sample basis. Some minor errors discovered but no systemic weaknesses found. Tested on a sample basis. Some minor errors discovered but no systemic weaknesses found. Minor error in method due to wrong basis for extrapolation, but management arrangements are sound. Small change in outturn agreed due to errors in rounding and calculation over-simplification. Management arrangements are good. Change in outturn agreed on the basis of more robust rationale and evidence. However management arrangements for this PI are weak, with limited collaboration across departments to improve accuracy and usefulness of this PI. Audit Commission data audit quality guidance 41 We have fed back to departmental representatives (PMIDG members) the findings and more detailed recommendations for strengthening management arrangements for the PIs which have been subject to spot checks. Nottingham City Council 12

16 Detailed findings Appendix 1 Action Plan Page no. Recommendation 6 R1 Raise the profile of data quality improvement by ensuring departmental management teams adopt the corporate strategy and monitor of delivery their own DQ improvement agenda. 6 R2 Review the use of end of year audit sheets, clarifying what information is needed and why, and the purpose to which it is put. 8 R3 Build on the sound initial, high-level risk departmental assessments of PIs by focusing on risks within the control environmental relating to each PI 8 R4 Continue to develop the focus on data quality skills and expertise within the network of departmental performance managers. 9 R5 Ensure management arrangements relating to the HIP HSSA PI (empty homes H18) and other PIs where there is a high dependency on cross departmental contributions include clear lines of accountability and adequate preparatory time to collaborate effectively. 11 R6 Continue to develop performance management arrangements, for example departmental quarter performance reports and the PerformancePlus system, to include commentary on data quality as well as data values. Priority 1 = Low 2 = Med 3 = High Responsibility Agreed Comments Date 13 Nottingham City Council

17 Detailed findings Nottingham City Council 14

18 Appendix 1 Action Plan Page no. Recommendation 6 R1 Raise the profile of data quality improvement by ensuring departmental management teams adopt the corporate strategy and monitor of delivery their own DQ improvement agenda. 6 R2 Review the use of end of year audit sheets, clarifying what information is needed and why, and the purpose to which it is put. Priority 1 = Low 2 = Med 3 = High Responsibility 2 SB SPG mem SB SB 1 PIPT/ Perf Reps Agreed Take DQ statement, objectives and self assessment approach to SPG for adoption/approval. SPG members to brief respective DMTs. DQ Imp. Action Plan and DQ approach included in Q3 CPR to: o Corp. Delivery Board o Exec. Board Update members handbook Council Plan and inc. DQ statement Use PerformancePlus (P+) P+ to collect data for year end close down. Use P+ to record risk assessment of PIs Use P+ to maintain a record of DQ work undertaken and findings and Date 7 Nov 08 Dec 08 Feb 09 Mar 09 Jan 09 April 09 SB SB PIPT Undertake testing of workflow implementation. Commence pilot of workflow on LAA indicators for selected PI and directorates for Q3. Roll out workflow to further PIs and directorates. Nov 08 Dec 08 Jan 09 Feb Mar 09 SB/Perf Reps Develop work plan with respective directorates comprising reactive audit work and proactive self assessment work. Dec 08 Jan 09

19 Page no. Recommendation 8 R3 Build on the sound initial, high-level risk departmental assessments of PIs by focusing on risks within the control environmental relating to each PI 8 R4 Continue to develop the focus on data quality skills and expertise within the network of departmental performance managers. 9 R5 Ensure management arrangements relating to the HIP HSSA PI (empty homes H18) and other PIs where there is a high dependency on cross departmental contributions include clear lines of accountability and adequate preparatory time to collaborate effectively. 11 R6 Continue to develop performance management arrangements, for example departmental quarter performance reports and the PerformancePlus system, to include commentary on data quality as well as data values. Priority 1 = Low 2 = Med 3 = High Responsibility 2 SB SB/SC/LM Perf Reps PIPT/Perf Reps 2 SB/Perf Reps SB Agreed Develop DQ self assessment template. Pilot DQ self assessment templates in Culture and Community and Children s Services. Directorates to undertake initial risk assessment to support DQ self assessment work including reliance on the systems of partners. Commence wider roll out and support for DQ Self Assessment. Continue to identify relevant training opportunities for inclusion of DQ. Support PMIDG reps and Services roll out of DQ Self Assessment Identify dedicated DQ training needs in light of SB/Perf Reps new NIs CAA etc. 3 Subject to agreement with Environment & Regeneration. 1 PIPT Performance reports contain commentary/management actions for indicators below target/high risk. These are challenged by directorate performance managers, corporate performance staff, DMTs & Corp Perf Board. Date Complete Oct Mid Nov Late Nov-Dec Ongoing Ongoing Jan 09 Ongoing PIPT Track actions addressing high risk/under performing PIs through P+. Ongoing

20