Audit Committee 25 th February Data Quality Review Report

Transcription

1 Local Members Interest Nil 1 Item No. 4 on Agenda Audit Committee 25 th February 2008 Data Quality Review Report Recommendation 1. To note the recommendations of the data quality review report and to review the progress made over the last twelve months in relation to management arrangements for data quality across the County Council. Report of the Deputy Corporate Director (Policy and Performance) Background 2. The Audit Commission introduced requirements for a data quality audit in June The purpose of the data quality audit is to assess an organisation s arrangements for the management of data to provide a level of assurance as to the quality of that data. 3. This assurance is necessary as data is used ever increasingly by organisations to inform policy, decision making, resource allocation and service provision. Introduction 4. Staffordshire County Council received a second data quality review report from PricewaterhouseCoopers in October 2007, following our submission of a self assessment, on our data quality arrangements against a set of Key Lines of Enquiry. 5. This data quality review report recognised the progress that has been made in relation to corporate arrangements for data quality over the last twelve month period and also outlined a number of recommendations to improve our arrangements further. 6. A number of key improvement activities have been undertaken to improve our data quality arrangements including the following: The development of a corporate policy for data quality; The design, implementation and roll out of a corporate performance management information system which has improved consistency and clarity in performance reporting and has had data quality at the heart of its design; Identified responsible officers, data input officers and target approval officers for all of our corporate indicators; A pilot risk assessment of data quality in the Development Services Directorate; Identified data quality champions in Directorates; Data quality checklists for outturn performance information;

2 2 A monthly/quarterly scorecard for key indicators which has allowed for more robust scrutiny and challenge on performance data and information; and Embedding data quality in our key planning/policy documents. 7. Further activities planned for this year include: Raising awareness to new managers through the Managers Induction programe on the importance of data quality and our responsibility in relation to data quality. A review of the pilot of the risk assessment process in DSD to identify whether it is advantageous for such an approach to be rolled out across the Authority. Introducing Member/portfolio holder accountability for corporate performance indicator. A review of the corporate data quality policy to incorporate the Audit Commission s voluntary standards on data quality. 8. Delivering on these recommendations should bring about a heightened awareness of data quality responsibilities, lead to the embedding of data quality throughout our systems, processes and culture and ultimately contribute to improving the quality of the data which we use on a daily basis. Equalities Implications 9. There are no direct implications arising from this report. Legal Implications 10. There are no direct implications arising from this report. Resource and Value for Money Implications 11. The Data Quality judgement impacts upon the Value for Money Judgement of the Authority through the Key Lines of Enquiry set out in the Use of Resources Assessment. Risk Implications 12. The risk of not responding to the data quality recommendations will impact upon the Corporate Risk Register. Report Authors Author s name: Katie Cairns Ext. No List of Background Papers: Data Quality Review Report Pricewaterhouse Coopers

3 Government and Public Sector Staffordshire County Council Data Quality Reviews October 2007

4 Members of the Audit Committee Staffordshire County Council St Chad's Place Stafford ST16 2LR 26 October 2007 Ladies and Gentlemen, Data Quality Reviews We are pleased to present the summary results of our assessment of the Authority s data quality arrangements, which has been completed in accordance with the methodology and guidance issued by the Audit Commission. We met with the Policy Officer and directorate representatives on 3 October to discuss the results. Please contact Sara Bagnall / Stephen Lappage if there are matters that you would like to discuss further. Yours sincerely is a limited liability partnership registered in England with registered number OC The registered office of is 1 Embankment Place, London WC2N 6RH. is authorised and regulated by the Financial Services Authority for designated investment business.

5 Contents Section Page Introduction and Summary... 4 Summary of Key Findings and Areas for Improvement... 6 Appendix A: Stage 1 Detailed Findings Appendix B: Stage 3 Data Quality Spot Checks Appendix C: Follow-up of 2005/06 Recommendations Code of Audit Practice and Statement of Responsibilities of Auditors and of Audited Bodies In March 2005 the Audit Commission issued a revised version of the Statement of responsibilities of auditors and of audited bodies. It is available from the Chief Executive of each audited body. The purpose of the statement is to assist auditors and audited bodies by explaining where the responsibilities of auditors begin and end, and what is to be expected of the audited body in certain areas. Our reports and management letters are prepared in the context of this Statement. Reports and letters prepared by appointed auditors and addressed to members or officers are prepared for the sole use of the audited body, and no responsibility is taken by auditors to any Member or officer in their individual capacity, or to any third party. 3

6 Introduction and Summary Introduction The Audit Commission has developed a three-stage approach to the review of data quality comprising: Stage 1: Management arrangements A review to determine whether proper corporate management arrangements for data quality are in place, and whether these are being applied in practice. The findings contribute to the auditor's conclusion under the Code of Audit Practice on the Council's arrangements to secure value for money (the VFM conclusion). Stage 2: Analytical review An analytical review of 2006/07 BVPI and non-bvpi data, and selection of a sample for testing based on risk assessment. Stage 3: Data quality spot checks In-depth review of a sample of 2006/07 Performance Indicators (PIs) all of which come from a list of specified BVPIs and non-bvpis used in CPA, to determine whether arrangements to secure data quality are delivering accurate, timely and accessible information in practice. All three stages of the review have been carried out at this Council. Summary We have completed our assessment of the Council s data quality arrangements in accordance with the methodology and guidance prescribed by the Audit Commission in Local government data quality Refresh 2007: Stage 1: review of corporate arrangements, Stage 2: analytical review, and Stage 3: Data Quality Spot Checks and this report sets out the results of our assessment. The assessment of the management arrangements in place for data quality is used to: Direct the detailed work that we undertake on data quality spot checks ; and Inform our Use of Resources Conclusion in respect of performance information (as reported in our 2006/07 audit report). The work that we have undertaken is also reported to the Audit Commission to inform their CPA assessment. 4

7 Findings Stage 1 Management arrangements The Council's overall management arrangements for ensuring data quality are demonstrating adequate performance. The detailed results are shown in Appendix A to this report. The Council have demonstrated to us a number of areas of development since our 2005/06 review but due to the timing of the implementation of these, a number have not impacted upon the scoring for the 2006/07 year. We will undertake a more detailed review of the developments noted which did not impact upon the 2006/07 year when we undertake the 2007/08 review. Developments noted include introduction of a corporate data quality policy, data quality champions within directorates and at member level, the introduction of the PMIS system and a pilot risk assessment. These demonstrate that the Council is seeking to improve data quality management. Stage 2 Analytical review An initial analytical review was carried out by the Audit Commission PI team. This review identified that the PI values for indicators applicable to the authority fell within expected ranges. Therefore no PIs were flagged up on the EDC website for further investigation by ourselves. Stage 3 Data quality spot checks Our review and spot checks of the following PIs: BV165 % of pedestrian crossings with facilities for disabled people PLSS7 Assessment of users 16 and over of their library service found them to be fairly stated. The detailed findings can be seen in Appendix B. Previous Year s Recommendations As part of this review we followed up the recommendations raised in the previous year. In 2005/06 our review covered the following PIs: BV 82a Recycling performance BV 82b Composting performance BV 102 Bus Patronage BV 109 Planning Speed BV 165 % of pedestrian crossings with facilities for disabled people BV 215a and BV 215b: Speed in fixing street lights C13 Cost per library visits C12a Stock turn-book issues population/books per 1,000 population (IPF) C12b Stock level books available for issue per 1,000 population (IPF) Appendix C shows the details of follow up work undertaken by the Council to implement the recommendations. During this review we have not validated the explanations provided to us.. 5

8 Summary of Key Findings and Areas for Improvement Management arrangements (Stage 1) Overall, the Council s corporate arrangements for data quality are demonstrating adequate performance. Governance and leadership Has the body put in place arrangements at a senior level to secure the quality of data used to manage and report on performance? Overview Responsibility for data quality has been assigned within the Council and this now includes an individual at top management level who has overall strategic responsibility for data quality. A number of developments have been implemented which were not fully embedded in 2006/07 such as appointment of a member data quality champion. The member data quality champion has received training however this has yet to be delivered to the wider member community. Areas for Improvement and Recommendations To further improve the Council should: Demonstrate that the data quality member has effective input into the process Evidence that the members have received training on the importance of data quality and are aware of the arrangements the Council has put in place to mitigate the risks associated with poor quality data Integrate data quality fully into the Council s planning, monitoring and reporting processes Demonstrate that the corporate commitment to data quality is actively promoted, making clear to relevant staff their responsibility for data quality (eg accuracy, completeness, timeliness). Demonstrate that accountability for data quality throughout the Council is clearly and formally defined for relevant staff and is considered as part of the corporate performance appraisal process for those staff. 6

9 Policies and procedures Has the organisation defined its expectations and requirements in relation to data quality? Overview A corporate data quality policy has been prepared and this was approved by CMT in February Directorate level polices are in place and operational level guidance notes exist. The Council has demonstrated that relevant staff are aware of, and can access, policies, procedures and guidance. Areas for Improvement and Recommendations To further improve the Council should: Embed the corporate data quality policy into the organisation Formalise requirements of the Council regarding data quality within partnership arrangements Incorporate data quality aspects into partnership arrangements Demonstrate the effectiveness and impact of the data quality champions Demonstrate that policies and procedures are reviewed at least annually and updated when needed 7

10 Systems and processes Are there effective systems and processes in place to secure the quality of data? Overview The Council has identified that there are some weaknesses within the CISS system and a replacement system has been identified however implementation has slipped. Security arrangements are in place over the Council s key performance management systems however the Council should ensure that evidence is available regarding any work on continuity arrangements. Where relationships exist with other bodies the procedures in place cannot provide assurance over the quality of the data in all cases. The Council should assess the arrangements to determine the risk by impact and likelihood of issues arising from data quality in these relationships. Areas for Improvement and Recommendations To further improve the Council should: Recognise the impact of the delays in implementation of the CISS system replacement, PISCES on data quality Maximise the benefits of the introduction of the corporate Performance Management Information System (PMIS) system Strive to achieve right first time data Undertake control mapping and testing of performance information systems to prevent and detect data manipulation and error. Demonstrating a proactive approach to strengthening performance information system controls rather than merely reacting to issues when detected Strengthen the procedures with partnerships to ensure that awareness exists of which data is provided by third parties, the quality of that data, how quality is assured by the third party or can be gained by the Council and the risks if that quality cannot be assured 8

11 People and skills Does the organisation have the resources in place to secure data quality? Overview Roles and responsibilities regarding data quality have been clearly defined within the directorates and corporate responsibilities are becoming more clearly defined with some further embedding required. Formal training is available but evidence of take up is limited. Incorporating data quality into mandatory training for relevant staff, such as Data Protection, is anticipated to yield better results as this is taken up. Training applied at operational level continues to be effective. Areas for Improvement and Recommendations To further improve the Council should: Demonstrate that it s internal network of data quality champions are effective and have successfully driven data quality improvement throughout the Council Assess relevant staff against data quality targets and standards set Ensure that relevant staff have access to guidelines when inputting data e.g. classification conventions, on-line help or quick reference guides to hand Develop a formal programme of training (including updates when necessary) on data quality issues tailored to the varying needs of all relevant staff, including corporate arrangements to ensure that this training is periodically evaluated and adapted to changing needs 9

12 Data use Are there effective arrangements for the use of data for performance management and service improvement? Overview Developments in year to the self assessment process have further strengthened the audit trail supporting the Performance Indicators. Reporting of data in the previous year was used for the day to day management of the Council s operations and continued during the 2006/07 year. Where issues are identified regarding service delivery, improvement plans are put in place. All data is subject to senior approval prior to external reporting. Areas for Improvement and Recommendations To further improve the Council should: Introduction of a formal documented process for checking externally reported data/performance indicators, both departmentally and corporately to assure the quality of the data Evidencing the outcome of effective quality assurance of the audit trail confirming accuracy of the data Evidencing that all reported data is rigorously verified both departmentally and corporately, but the extent of this is informed by an analysis of the level of the risk of the data being misstated, likelihood and impact of data errors and the accuracy required in the reported performance. 10

13 Appendix A: Stage 1 Detailed Findings KLOE Criteria 2005/06 Assessment 2006/07 Assessment Governance and leadership Adequate performance Adequate performance Policies and procedures Adequate performance Adequate performance Systems and processes Adequate performance Adequate performance People and skills Adequate performance Adequate performance Data use Performing well Performing well Overall Assessment Adequate performance Adequate performance 11

14 Appendix B: Stage 3 Data Quality Spot Checks PI tested BV165 % of pedestrian crossings with facilities for disabled people PLSS7 Assessment of users 16 and over of their library service Overall conclusion Fairly stated at 98.7% Fairly stated at 94.4% Control issues identified Following control issue was noted: The total number of crossings as per the working papers (375) does not agree to the total number of crossing as per the database (377). The council has used the number of crossing as per the working papers to calculate the final outturn. This error does not materially affect the final outturn, therefore no amendments are proposed. Recommendations: The pedestrian crossing figure should be taken directly from the database. No control issues noted. 12

15 Appendix C: Follow-up of 2005/06 Recommendations PI tested Overall conclusion Control issues identified Follow-up (Please note that no testing and validation has been performed by PwC to confirm the actions taken by the Council) BV 82a Recycling performance BV 82b Composting performance Fairly stated No issues noted Not applicable Fairly stated No issues noted Not applicable BV 102 Bus Patronage Reserved The outturn was reserved for the following reasons: We were only able to verify the data received from three main operators in the county as included in the calculation, at 21,524,975. While this may be in-line with DfT suggestions, it is still not possible to verify weather the PI has been stated fairly as the remaining figure (around 25% before applying the growth factor) can-not be validated. The following control issues were noted; We were unable to validate the number of passenger journeys allocated to seven medium-sized bus operators; A growth figure of 33% has been applied to previous year's The following actions have been taken by the council to implement the recommendations: Pro-formas with clear guidance were sent to all bus operators for completing 2006/07 bus journey estimates. Questions were included on the pro-forma to clarify how the operators recorded their passenger journeys and the basis of assessing the number of journeys not recorded by the drivers (included concessionary fare passengers); A number of non-returns are currently being followed up (less than 300,000 passengers). However, the data used in compiling this 13

16 figure for FIRST by the operator. However, we have not been able to verify if the 33% growth factor applied is a true reflection of the patronage for the operator; Page 5 of the BVPI return clearly requires the compiler of the return to "give FULL details of how the indicator has been compiled". The statement has been made to state that three major operators account for over 90% of the total figure, this is not the case, as 21,524,975 / 29,011,514 = 74.2%; The spreadsheet used to collate data is not password protected. Recommendations: Wherever possible bus journeys included within the outturn should be based on the returns from the bus operators; Any growth factors used should be based on concrete data; Full details of how the indicator is compiled should be given on the BVPI return; The spreadsheet should be password protected to maintain the security and integrity of data. indicator is based solely on actual returns received up to and including 20th June 2007 and no estimates of nil returns have been used; The data collected was entered into a password protected excel spreadsheet together with each operator s response to the questions included in the pro-forma. If an operator responded that passenger journeys were recorded by Electronic Ticket Machine Data or Other, they were required to state what their assessment was of the number of journeys not recorded by drivers in Staffordshire. If, as in accordance with the guidance, an operator provided no reasonable estimate then an uplift factor of 4% has been applied to the raw estimate of all passengers provided by the operator. No uplift has been applied to minor operators which has been agreed with DfT as being below 200,000 patronage; The 4% uplift has been entered into the password protected spreadsheet and added to the raw estimates of passengers to provide an overall outturn for 2006/07. The 4% uplift accounts for less than 892,000 of total patronage; The data used and the performance indicator calculation has been validated by 2 different staff members, the formulas are cross checked and the data has been agreed back to its source. BV 109 Planning Speed Fairly Stated The following control issues were noted; There is no restricted access to the database used to produce The following actions have been taken by the council to implement the recommendations: 14

17 the outturn for the BVPI, therefore data can be easily amended by various users without any audit trail to record the details of any amendments; and The BVPI is prepared and submitted by the same officer. However it is good practice that the BVPI is reviewed and signed off separately prior to the final submission by the council. Password protection has been introduced to the underlying database; and The outturn for this indicator is now subject to formal quarterly and annual internal review by senior management and elected members. Recommendations: The database should be password protected to ensure only authorised individuals can access the system; and The performance indicator submission form should be independently signed off and reviewed to ensure its accuracy. BV 165 % of pedestrian crossings with facilities for disabled people Restated from 98.31% to 97.2% The outturn was revised for the following reason: Three of the twelve pedestrian crossings tested did not comply with the dropped kerb requirement of <6mm or 9mm for pre 2002 crossings. Therefore the performance indicator was amended to exclude these crossing from the outturn. The impact of the above was to adjust the final outturn by 1.13%. The following control issue was noted: The following action has been taken by the council to implement the recommendation: In 2006/07, the inspections were completed more rigorously by the council and the crossing which did not meet the dropped curbs criteria were subsequently corrected by the maintenance team. Our testing identified three pedestrian crossings which were included within the outturn. However these crossings did not fully meet the criteria for, Crossings with facilities for disabled people. Recommendation: Internal checks should be introduced to ensure only those crossings which are fully compliant are included within the performance indicator outturn. 15

18 BV 215a and BV 215b: Speed in fixing street lights Reserved The performance indicator was reserved for the following reasons: We were unable to trace 34 out of the 40 faults tested to the CRYSTAL report which was used to calculate the outturn for the performance indicator; and One of the key tests is to confirm the consistency of start and end date for Distributed Network Operation (DNO) faults between 215a and 215b. However the Mayrise system is not flexible enough to identify those faults which were subsequently completed by DNO, hence we were unable to perform this key test. The following control issues were noted; Asea Brown Boveri Limited (ABB), the contractor responsible for the maintenance of the Council s street lighting network was unable to locate the documentation to support the end dates for 2 out of the forty faults tested; The following actions have been taken by the council to implement the recommendations: All data is recorded on the IT based Asset Management System (AMS) and the data is reviewed by Internal Audit twice a year; and The Council has provisionally agreed to migrate the AMS data into the newly acquired departmental Integrated Highway Management System (IHMS). The IHMS will provide the functionality to enable the production of BVPI data in accordance with the Communities and Local Government BVPI guidance. Fault number has been included in the performance indicator calculation twice; Due to the system limitation, the Council was unable to perform independent validation checks on the CRYSTAL report provided by ABB; The system is unable to identify the link between the DNO faults recorded on BVPI 215a and 215b; and The Council place reliance on the data from ABB for internal monitoring purposes, however we were unable to trace a large number of faults from the Mayrise system to the Crystal report. Therefore we were unable to obtain any comfort over the completeness and accuracy of the CRYSTAL report. 16

19 Recommendations: All dates should be supported by appropriate documentation; Internal checks should be introduced to ensure all faults are included once only; and The Council should discuss the issues around the Crystal report and system limitation with ABB. Every effort should be made to resolve these issues before next year s audit to avoid any future reservations. C13 Cost per library visits Fairly stated The following control issues were noted: The following issue was noted at one library (45 libraries in the County). The method adopted to calculate the number of library visits at Leek is not robust. The current methodology calculates the number of library visits at Leek by the number of total transactions recorded at the electronic people counter less the number of transactions recorded at the one stop shop. The problem is that one person may have many transactions, therefore the council would record 'x' non-library visits but the people counter would only count that person as 1. Therefore the visits to Leek Library may be misstated however this does not have a significant impact on the performance indicator; and The audit trail for non library visits is not sufficient enough to gain assurance that they have been accurately recorded. Recommendations: The current method at Leek library should be reviewed to ensure additional procedures are introduced to ensure all visits are accurately counted; an d The following actions have been taken by the council to implement the recommendations: In order to exclude non library visits from the total visits figure recorded by the people counters, Staffordshire Library and Information Service asks partner organisations, generally District Councils, to conduct a one week survey of their customers at the relevant locations to identify how many visitors do not use library facilities. In response to the 2006 audit; Standardised procedures have been introduced In 2006/07. Partners sharing premises with libraries, including Leek, conducted a standardised non-library visits survey; and All documentation, including the survey sheets, is now signed by the responsible officers within the partner organisation before returning them to Staffordshire Library and Information Service. 17

20 C12a Stock turnbook issues population/books per 1,000 population (IPF) C12b Stock level books available for issue per 1,000 population (IPF) Reserved Reserved All non library visits should be supported by documentation to demonstrate a clear audit trail. The performance indicators were reserved for the following reasons: We were unable to fully complete Test 4 in that we could not Check that the book has been correctly included or excluded in the CIPFA return. This is because the Council, when conducting the stock census or producing TALIS reports of stock on loan, reserved and in transit, do not identify individual books included in the totals; and A reconciliation between the manual stock count and TALIS identified a difference of 12.4% which is above Audit Commission s tolerable limit (10%). Following control issues were noted: The stock count is not used to update the database; The Council s CIPFA submission is based on the manual stock count. A reconciliation is not carried out between the stock figures produced by the database and the stock figures as per the stock count; The following actions have been taken by the council to implement the recommendations: During 2006/07 a working group was established to fully investigate the stock count issues. Between October and December 2006 all items on the database which had not been issued since the conversion of the database from the Council s previous Library Management System (LMS) to the current system were checked and the database cleansed. In February 2007a manual stock count was undertaken in every service point and totals compared to a stock count from the LMS. An extensive programme of stock checking is now taking place in every library for items that the LMS has not seen for 18 months. The LMS will then be updated. The aim is to complete this work by February 2008; and During the February 2007 stock count, spot checks were implemented as recommended. One book was found to have not been added to the shelves on timely basis, which could mean that there is a delay in the availability of books to the public; and The Council has not stated their numerator and denominator per 1,000 population. Recommendations: The council should have a policy as to how the stock take will be used to update the database and how this may affect the CIPFA return; 18

21 Stock take counts should be compared to stock figures produced by the stock database; and The stock take counts should be supplemented by a number of spot checks, to be determined by the Council. This will involve checking titles per a manual stock count to titles on the database. 19

22 In the event that, pursuant to a request which [insert name] has received under the Freedom of Information Act 2000, it is required to disclose any information contained in this report, it will notify PwC promptly and consult with PwC prior to disclosing such report. [insert name] agrees to pay due regard to any representations which PwC may make in connection with such disclosure and [insert name] shall apply any relevant exemptions which may exist under the Act to such report. If, following consultation with PwC, [insert name] discloses this report or any part thereof, it shall ensure that any disclaimer which PwC has included or may subsequently wish to include in the information is reproduced in full in any copies disclosed All rights reserved. PricewaterhouseCoopers refers to the United Kingdom firm of (a limited liability partnership) and other member firms of PricewaterhouseCoopers International Limited, each of which is a separate and independent legal entity.