PERFORMANCE DATA QUALITY POLICY

Transcription

1 PERFORMANCE DATA QUALITY POLICY 2007 / 08 Improvement Service May 10 th 2007 Data Quality Policy V

2 INTRODUCTION / BACKGROUND Good quality performance data is accurate, valid, reliable, timely, relevant, and complete. Further definitions of these six key characteristics of good quality performance data have been extracted from the Audit Commissions Voluntary Code of Practice to support improvement in Data quality in the Public Sector (November 2005) and are contained in Appendix A. Hull City Council are committed to ensuring that good performance data quality is at the very heart of our performance management arrangements effectively turning our performance data into reliable information for our key decision makers to support good evidence based decision-making. Good quality performance data is needed to effectively manage and continuously improve service delivery and key outcomes for local people. Service users and members of the public need accessible performance information to make informed decisions about the overall quality of services we provide and their continuous improvement. Performance information is increasingly being used by external bodies to assess our performance, often as an alternative to inspection and this continuing trend is causing external bodies to place a bigger emphasis on performance data quality. In particular the external audit approach of checking calculations and system reports is evolving into a more challenging scrutiny of our data quality management arrangements and systems controls at both corporate and service level. Hull City Council has a Scrutiny function and their role is to monitor the strategic performance of the Council. An important part of their work includes monitoring and challenging service area performance against the Council's key strategic performance indicators and providing appropriate comments/recommendations to Cabinet and senior management, having considered monitoring reports. This policy helps to underpin the strategic work by the Scrutiny function and outlines the key steps necessary to maintain the highest possible performance data quality standards throughout the Council in order to ensure that key decisions are based on reliable performance information. The council is committed to providing regulators and government departments with timely robust and reliable information to enable them to make judgements about our overall performance and governance arrangements. The need for reliable data has therefore become more critical to the authority. The Audit Commission (AC) Data quality audit (Dec 2006) looked at the arrangements for governance and leadership, data policies in place, systems and processes, people and skills and data use. The AC used a set of Data Quality Key Lines of Enquiry (KLOEs) to assess and score our current data quality management arrangements. The AC concluded that the Councils management arrangements for data quality met minimum requirements and our data quality systems and processes are robust but they are not yet being consistently applied across the authority. They further stated that although the Data Quality Policy V

3 Council has arrangements for the governance, monitoring and review of data quality, these are not yet formalised in an overarching data quality strategy. Currently the data quality score directly contributes to the annual use of resources score via the following Value for money criteria The body has put in place arrangements to monitor the quality of its published performance information and to report the results to members This policy is accompanied by an action plan (in Appendix B) approved by the AC with clear responsibilities and timescales to support the necessary improvement to our Corporate data quality management arrangements. The guidance notes have been constructed to address the key recommendations from the AC Data quality review findings and also follow the themes of governance and leadership, data policies in place, systems and processes, people and skills and data use. A more detailed implementation plan will be developed prior to policy approval to assist policy implementation across the authority. This work will be coordinated by the Corporate Performance Team (CPT), Key corporate contacts from information governance, workforce development, risk management and those service staff involved in using third party performance data will be collectively involved in developing the implementation plan. The following Services have been consulted during the development of this policy: Corporate Risk, Workforce Development, Strategic Procurement, Information Governance Team, Internal Audit Team, Oracle programme Manager / Project Manager, Corporate Virtual Performance Team (VPT) Improvement Service, Corporate Employee Group (CEMPOL), Corporate Policy Manager, Local Strategic Partnership (LSP) Performance Officer, Area Programme Manager, Partnership Manager and the Corporate Finance section. This draft policy was also submitted to the Corporate Employee Group (CEMPOL) on May 8 th 2007 for their consideration and feedback. Data Quality Policy V

4 1. SCOPE AND PURPOSE 1.1. This policy outlines the City Councils approach to Performance Data Quality Assurance and applies to all employees of the Council This policy will be communicated to all service areas and is accompanied by guidance notes 1.3. This policy applies to all Performance information held and shared by the Council irrespective of ownership 1.4. This policy also applies to all contractors, partnerships and agencies operating on behalf of the Council or on Council premises. For the purpose of this Policy the term employee covers all of these groups The Policy guidance notes tell you who is responsible for Performance Data Quality across the Authority. A matrix of responsibilities and the approved action plan are contained in appendix B 1.6. This policy should be read in conjunction with the Corporate Performance Management framework, Risk Management policy and Strategy, Information Security Policy, Freedom of Information policy, Section 151 reporting procedures and two external documents; HCC AC Data quality report Jan 2007 and the Improvement in data quality framework document Improving information to support decision making : standards for better data quality in the Public sector March The Council has a statutory responsibility to publish performance information and is committed to ensuring that all performance data is accurate, valid, reliable, timely, relevant, and complete. The Council is committed to proactively responding to impending changes to how the AC will deal with invalid or poor quality performance data and the increased emphasis on the use of published key performance indicators as the primary means by which local authorities overall performance will be categorised within the Comprehensive Performance Assessment (CPA) Framework. The policy will also address the key findings and recommendations from the AC Data Quality Audit report (Jan 07) For the purposes of this policy the term Performance data can be defined as all financial and non- financial information (both quantitative and qualitative) reported and used in key decision making by the Cabinet Committee on Performance and Finance (CCPF), Scrutiny and Overview Committee, Corporate Strategy Team (CST) / Transformation board, Cabinet and Council members and includes all performance information relating to the progress against the Local Area Agreement (LAA) reported via Partnership Data Quality Policy V

5 scorecards, statutory Best Value Performance Indicators (BVPIs) including customer satisfaction surveys, CPA Key Performance Indicators (KPIs) Local KPIs, CPA related processes such as Use of resources and Value for Money activities, the Council Customer promises, all Service Performance reported via Service balanced Scorecards and Area delivery scorecards. This list is not exhaustive and subject to change 2. POLICY OBJECTIVES 2.1. To ensure the arrangements for the governance, monitoring and review of performance data within the Council are formalised and an organisational culture that values the quality and reliability of performance data is fostered To ensure the Councils overall CPA rating / annual CPA Service Assessment Scores are not adversely affected by invalid or poor quality key performance data which can be treated as missing or qualified by the AC in future annual Data Quality audits 2.3. To provide assurance to residents, key stakeholders and external bodies that the Councils published performance information is valid and accurate 2.4. To provide Members, Council employees, contractors, the LSP, other key partnerships and agencies with a formal policy framework to ensure performance Data Quality assurance is consistently applied across the authority 2.5. To acknowledge the increased emphasis on the use of published Key Performance indicators as the primary means by which local authorities overall performance will be categorised within the current CPA framework and the new Performance framework for Localities post CPA To ensure that the key performance information used to monitor, manage and report Council performance, aid benchmarking, set targets and allocate human and financial resources at all levels is accurate, valid, reliable, timely, relevant and complete. 3. MONITORING AND REVIEW 3.1. The policy and guidelines will be reviewed and updated annually by the Improvement Service to take into account any changes to the CPA framework and AC statutory working practices to ensure they remain current and relevant Data Quality Policy V

6 3.2. The effectiveness of the policy will be assessed by comparing subsequent internal audit report findings and ratings and external data quality audit report findings and continuous improvement against the relevant Data Quality KLOEs and / or national data quality standards The policy and guidelines will be published on the Council website and feedback from the VPT and all Council employees will be invited and fed into the review process to aid continuous improvement. 4. NON COMPLIANCE WITH THE POLICY 4.1. Line Managers are responsible for ensuring that all employees are aware of this policy but in particular those employees who have a role in collecting and reporting financial and performance data. Employees must read and understand the Policy and Guidelines and refer any questions to the CPT. Employees not complying with this policy will be dealt with under the Councils disciplinary procedure. In the event of an agency worker or casual worker, his / her work with the Council may be terminated. The contract may also be terminated if the employee is an employee of a contractor. Data Quality Policy V

7 GUIDANCE NOTES 5. GOVERNANCE AND LEADERSHIP 5.1. The overall responsibility for performance data quality within Hull City Council rests with the CST and Head of Improvement Service. CST assigns this role to Heads of Service who are empowered to ensure that appropriate human and financial resources are committed to performance data quality with clear responsibilities and accountability for performance data quality being assigned within their own Service. The matrix of responsibilities in Appendices B provides full details The Corporate Performance Office and the VPT will be responsible for communicating both the Corporate and service level commitment to performance data quality across the Council fully utilising Hull Talk, the council intranet, bespoke workshops, Service and team meetings. Formal training sessions will be provided by the CPT to staff involved in collecting and reporting performance data. A key message that all staff have a responsibility for data quality will also be communicated simultaneously using the same communication channels Specific responsibilities and accountability for data quality will need to be considered when writing all new job descriptions. Staff with specific performance data quality responsibilities will have formal data quality objectives that are formally incorporated, set and measured using the Personal Performance and Development (PPD) process. The opportunity should be taken to take into account specific performance data quality responsibilities when reviewing and updating all job descriptions Data quality will be embedded in the Corporate risk management arrangements initially through the Section 151 reporting arrangements. The Corporate risk register will be refreshed in April 2007 and include the identification and management of those key risks associated with unreliable and inaccurate performance information 5.5. The organisation s commitments to data quality (for example, the importance of, and arrangements for, securing the quality of key performance data) will be outlined by CPT in key strategic documents such as the Corporate plan, Best Value Performance Plan (BVPP) and the Corporate Performance management framework All Performance Data will be subjected to both departmental and service level checks and reviewed by management and signed off at each level for each reporting period before being reported corporately. Data Quality Policy V

8 5.7. All performance related reports submitted to the main council committees / CCPF / Scrutiny and overview and CST will be subjected to timely and objective review and validation by the CPT prior to their submission Performance data provided by the Council is also used, in conjunction with performance data provided by partner organisations, to report on the LAA scorecards and other performance reporting currently undertaken through the (LSP)1. To this end data from both the Council and partnership organisations will need to meet certain quality standards in order to meet the reporting requirements of the partnership Reporting on the LAA will require the Council, through the LSP to use a significant amount of data that is provided by outside partner organisations. Partner organisations will include other public sector bodies and, potentially, organisations from both the private and Voluntary and Community sectors. It is likely that such diverse bodies will have varying levels of data quality assurance processes in place. The implementation of the data quality policy will need to ensure that there is a minimum level of data quality assurance across partner organisation to enable the effective and correct reporting against the LAA 6. MAINTENANCE OF INFORMATION SYSTEMS MANUAL AND I.T Responsibility for maintaining a robust control environment for both system and paper based performance information systems lies within Services, but there are some corporate resources for supporting improvement in this area. A central record of manual based and IT based Business critical performance information systems and their named data quality lead officers will be held by the CPT. Services may be using a mixture of manual and IT based systems to produce and report performance information. For the purpose of these guidance notes the following systems are deemed to be Business critical Oracle, Northgate, Care first, IQuanta, Keystone, Customer Relationship Manager (CRM) and Information Business System. This list is not exhaustive and subject to change Key service performance contacts will continue to use the corporate KPI file system for all statutory BVPIs and CPA KPIs and for those local KPIs that are also Council Customer promises. These files will continue to be internally audited by the service contacts and the CPT using the internal audit checklist (Appendix c) to ensure the necessary audit trails are in place for external audits by the AC. 1 Technically, the only reporting that should be undertaken by the LSP from April 2007 is on the Local Area Agreement. Data Quality Policy V

9 Internal Audit will also conduct file audits using a risk based and proportionate approach Each system should have a named officer responsible for performance data quality issues. The responsible officer would be required to ensure that: Users are appropriately trained and supported in the use of the system/s There is security of access / amendment Periodic tests of the integrity of the data are undertaken Performance information is processed accurately with no need for subsequent intervention, manipulation or correction System upgrades are made where necessary (including to accommodate amendments to PI definitions) The system meets the CPT, Head of Service and Senior Managers information needs Feedback from users is acted upon The system can produce adequate audit trails for the purposes of Internal audit and the AC data quality review process Actions recommended by system reviews (e.g. by the external auditors are implemented) A process chart and a set of complementary written procedures (user guides) exist for the purpose of data collection, completeness checks, recording variance analysis and reporting of performance information A business continuity plan for the system/ s is in place to provide protection for records and performance data which are vital to the continuous effective functioning of the Service / organisation A named substitute officer can deputise in the absence of the data quality lead officer to maintain the day to day running of the system / s. Information which is used for external reporting is subject to rigorous verification, especially where errors may lead to loss of income 7. EVALUATION AND IMPROVEMENT OF SYSTEM/S 7.1. There will be systems where work has to be undertaken to rectify gaps in the control environment. To identify these systems there needs to be a coordinated evaluation of every information system used in the Council to produce performance information. The ICT Service will be responsible for maintaining a database of all systems, including: The identity of the data quality lead (and deputy) A summary of data quality and verification actions undertaken; and Risk assessment undertaken 7.2. There are a number of conditions that might lead to a system being considered high risk and every system needs to be considered Data Quality Policy V

10 against these factors. The risk assessment will be updated annually by the CPT / Updates Officer in consultation with the data quality lead officer. High risk conditions will include: A high volume of performance data / transactions Technically complex PI definition / guidance Problems identified in previous years by internal audit / district audit Inexperienced staff involved data processing / KPI file production System being used to introduce a new KPI; and Known gaps in the control environment 7.3. Where high risk systems have been identified for attention, the following steps will need to be taken: Analysis of the control environment Identification of gaps; Design of additional controls and procedures to address gaps Preparation of an action plan; and Monitoring the implementation of the action plan 7.4. The purpose of undertaking a risk assessment is to target limited resources at the areas that require most attention. A coordinated programme of data quality improvement will be compiled by the CPT in consultation with internal audit and named data quality lead officers. Progress will be reported regularly to Corporate Implementation Group (CIG) / CST using an exception basis via the existing reporting arrangements. 8. THIRD PARTY DATA 8.1. Particular attention needs to be paid to data provided by external sources. A number of KPIs are calculated using information provided by contractors and the Councils intention is to work alongside contractors to ensure that such data is accurate All contracts should be tendered on the Council s standard terms and conditions and include a system of performance monitoring based on the specification and evaluation. When entering into contracts with service providers it is essential that, there is a requirement to provide timely and accurate performance information, and that we are clear with the contractor about their responsibilities for data quality validation and how we will be double checking the information they provide. Advice on designing contract monitoring systems can be sought from Corporate Procurement and should be done at the outset of the tendering process It might not always be possible to alter existing contracts so that contractors are fully committed to providing an agreed quantity of Data Quality Policy V

11 performance data. In this case, the data must be treated as high risk and thought must be given to establishing a system of checks and measures to ensure that we are confident about the accuracy and validity of this data. When carrying out checks on such information it is important that this is documented and signed off by the relevant Data quality lead Officer and their Head of Service Some important performance information for example, road safety and crime statistics is provided directly to the council by external agencies. The initial priority of this policy is address shortcomings in performance information provided directly by us and to us, but where concerns exist about the integrity of the externally provided performance information, the Councils intention is to work with other agencies constructively wherever possible to provide assurance and rectify any problems identified Responsibility for data verification lies within Service areas, but internal audit and the CPT can offer advice and guidance about third party verification procedures. In some cases it might become apparent that existing procedures are not sufficient to maintain a robust control environment and will need to be modified accordingly. 9. PEOPLE AND SKILLS 9.1. All Council employees will have their awareness about Performance data quality and its importance raised through Hull Talk articles supplemented by regular updates on the intranet Formal training on the Corporate Data quality policy and associated guidance notes will be provided to all nominated data quality lead officers by the CPT and subsequently rolled out Council wide by Corporate Training and development Intranet based training on performance data quality will be developed by the CPT and made available to all council staff The CPT will provide the necessary update training to ensure the latest changes by the AC in data quality procedures, guidance and systems are disseminated and acted upon in a timely manner by data quality lead officers All Data quality Training will be periodically evaluated and adapted to respond to changing needs 9.6. Any weaknesses and associated recommendations identified through internal or external reviews of performance data quality are to be addressed by follow up audits by the CPT and / or additional training and support for the relevant data quality lead officers. Data Quality Policy V

12 9.7. Prior to the implementation of the policy, the CPT will undertake an assessment of existing data quality skills that are in place for the councils Business critical systems and identify potential gaps and resource issues. Data Quality Policy V

13 Data Quality: Appendix A Appendix A Dimensions of data quality Dimension Accuracy Validity Reliability Timeliness Relevance Data should be sufficiently accurate to present a fair picture of performance and enable decision-making at all appropriate levels. The need for accuracy must be balanced with the costs and effort of collection. A prerequisite is that definitions for data should be specific and unambiguous. The data must be at an appropriate level of detail to influence related management decisions, and must be within a reasonable margin of error (the amount of variation normally expected from a given data collection process). Data should clearly and appropriately represent the intended result. Where proxy data is used, organisations must consider how well this data measures the intended result. Data should reflect stable and consistent data collection processes and analysis methods across collection points and over time, whether using manual or computer based systems or a combination. Managers and stakeholders should be confident that progress toward performance targets reflects real changes rather than variations in data collection methods. Data must be available for the intended use within a reasonable time period. Data must be available frequently enough to influence the appropriate level of management decisions and should be current enough when they become available: for example, it may be appropriate to accept a small degree of inaccuracy where timeliness is important. The data reported should comprise the specific items of interest only. Sometimes definitions for data need to be modified to reflect changing circumstances in services and practices, to ensure that only relevant data of value to users are collected, analysed and used. Completeness All the relevant data have been recorded. Monitoring missing or invalid fields in a database can provide an indication of data quality and can also point to problems in the recording of certain data items. Data Quality Policy V

14 Data Quality: Appendix B APPENDIX B - MATRIX OF PERFORMANCE DATA QUALITY RESPONSIBILITIES Who What they must do When CST / Head of Overall responsibility for the reliability of performance information presented at main council ONGOING Improvement committees / CST. Corporate Performance Team / Updates Officer Provide high quality data to regulators and government departments to enable them to make sound judgements about our overall performance and governance arrangements. Review and update Annual DQ policy /guidance notes Communicating corporate commitment to DQ policy / guidance across the authority. Provide formal data quality training sessions / update sessions to data quality lead officers. Outline the organisations commitment to good quality performance data in key strategic documents/ Review all performance information prior to their submission to CCPF / Overview and Scrutiny / CST Working with the data quality lead officers maintain a database of performance information systems and co ordinate an evaluation of those business critical systems using a risk based approach. Evaluate all Data quality training and adapt to meet changing organisational demands. ANNUAL ONGOING Data Quality Policy V

15 Data Quality: Appendix B Who What they must do When Undertake an assessment of existing data quality skills in place for the Councils business critical system and report to the Head of Continuous improvement on potential gaps and issues Continue to audit BVPI / CPA and Local KPI arrangements using a risk based approach Heads of Service Working with the Internal Audit function - Co ordinate all Internal audits and DQ reviews and prepare services / KPI owners accordingly Ensure appropriate human, financial and technical resources are allocated to ensuring good quality performance data within their Service areas. Assign designated responsibility and accountability for data quality within their Service area /s. Identify where performance information is currently provided by third parties and assess the risks to data validity and reliability. Build in clear data quality responsibilities and requirements for the third party as part of existing / future contracts /and specifications. ANNUALLY ONGOING Line Managers Ensure all staff are aware of the policy and importance of good performance data quality. ONGOING Take into account specific data quality responsibilities when reviewing, updating or creating job descriptions. Ensure data quality objectives are formally incorporated and measured as part of the PPD process. Data Quality Policy V

16 Data Quality: Appendix B Who What they must do When IT / Manual Full compliance with relevant sections of the policy. ONGOING based System Data quality Lead officers All staff with Full compliance with relevant sections of the policy. ONGOING responsibility for inputting / extracting performance information from IT and manual based systems including KPI owners Internal audit Embed Data quality into Section 151 reporting arrangements. ONGOING Identify and manage those risks associated with unreliable and inaccurate performance data as part of the Corporate risk register process Provide guidance an advice on third party data validation to Services where appropriate Continue to audit BVPI / CPA KPI arrangements using a risk based approach Incorporate DQ issues in routine audit work Support Performance Officers with all DQ issues. Support improvements to all IT / Manual based reporting systems Data Quality Policy V

17 Data Quality: Appendix B Who What they must do When Working with the Corporate performance Management Officers - Co ordinate all Internal audits ANNUALLY and DQ reviews and prepare services / KPI owners accordingly LSP Performance Officer Details to be agreed with Sylvia Yates and David Blowers Data Quality Policy V

18 Data Quality: Appendix C Appendix C - INTERNAL AUDIT CHECKLIST FOR KPI FILES 1 Process Map a Is the whole process shown from start to finish and is it clear where the process starts and finishes? b Is there evidence in the file to support each stage within the process map? c Is there a date on the process map and has it been updated / reviewed for the current year? 2 Indicator Definition a Does the file contain a clear approved definition for the performance indicator and does it cross-reference to the original source is the definition dated? b Have all aspects of the definition been complied with when calculating the indicator? c Is there a clear definition for sub-parts of each indicator (e.g. visits to museums is it clear what a visit is)? 3 Calculation a Have detailed workings out been provided in the file to clearly and transparently show how the calculation was performed? b Has the calculation been performed as per the indicator definition? c Has the indicator been rounded to the correct number of decimal places as per the ODPM guidelines? d Where a deprivation adjustment has been used, have both the pre-adjustment and post-adjustment calculation and figures been provided? 4 Validation of calculation a Have all sections of the file got information in them (for an indicator with several parts a,b,c,d etc., have all relevant sections been completed for each part)? b Is there evidence of independent validation / independent audit of the calculation? c Has a second check been carried out? d Have checks on input been carried out (either manual of computer checks)? e Has data sampling been used to validate data (e.g. sampling of staff data on Oracle)? f Are there any errors in the calculations (e.g. omissions, miscalculations etc.)? 5 Cross-referencing a Are all documents / reports etc. clearly referenced to their source (a good cross-reference might be report generated form Oracle using Discoverer on 31 st March 2006) b Is the calculation clearly cross-referenced to the definition / process map? 6 Approval / sign-off a Has the file been signed and dated by the Manager and the performance contact? Data Quality Policy V

19 Data Quality: Appendix C 7 Target-setting a Does the file explain how and why the target has been set? b Is the target realistic (i.e. we are not trying to get from lower quartile to upper quartile in one year)? c Has the corporate protocol been followed? d Are the quartile figures / threshold figures supplied in the file? 8 Use of Estimates a Is there evidence to show this been agreed with the Audit Commission? b Have the Audit Commission guidelines for sampling been followed? 9 Statistics a Are copies of relevant statistics available in the file (e.g. population statistics, office for national stats. estimates etc.)? b Has the source of the statistics and the date been provided in the file? 10 Follow-up Actions a Have any previous issues (from Internal Audit / District Audit / central QA) been actioned and has this been evidenced / recorded in the file? b Is a record of previous issues identified by District Audit/Internal Audit held on file? 11 Variance Analysis a Has a variance analysis been provided in the file and has the calculation been carried out correctly? b Has the variance analysis been carried out as per central guidance (e.g. has a detailed explanation been given for variance greater than 15% (10% for financial indicators))? c Have we compared the latest out-turn with the target and the previous year s outturn? 12 Documentation a Has all key paperwork been collected (e.g. where all sickness notes need to be collected for BV12) is there evidence that all paperwork is available)? b Is all key paperwork that needs to be collected / provided in the file referred to in the indicator process map? c Is all documentation (e.g. risk logs) up-to-date? 13 Certificates (are calibration certs or certs. for subcontract work relevant?) a Where applicable, are relevant certificates in file (e.g. calibration certificates for measuring equipment etc.)? 14 Audit Trail a Do all documents link together / cross-reference? b Is it clear which spread-sheet has been used and which cells information was taken from within the spread-sheet? KPI ref KPI Owner signature Internal auditor Date Data Quality Policy V