Data Quality Assurance: Quality Gates Framework for Statistical Risk Management

Transcription

1 Data Quality Assurance: Quality Gates Framework for Statistical Risk Management Narrisa Gilbert Australian Bureau of Statistics, 45 Benjamin Way, Belconnen, ACT, Australia 2615 Abstract Statistical collections are often exposed to the risk that the components of the process fail to meet the quality standard expected, such that the quality of the statistical outputs are affected. We refer to this kind of risk as "statistical risk". Statistical risk arises for various reasons, some of which may include inadequate inputs, processes not being well defined, changes to existing processes, or human error. This paper introduces an approach to managing the quality of statistical processes called quality gates. Quality gates are a risk mitigation tool designed to improve the early detection of errors or flaws in statistical processes. They improve a user's ability to manage statistical risk by providing explicit evidence relating to the statistical process, improving knowledge management and information sharing. This paper will provide an explanation of the concepts underpinning quality gates so that they can be applied for statistical risk management. A discussion of some of the challenges that the Australian Bureau of Statistics has experienced in implementing quality gates will also be provided. Key Words: quality gates, statistical risk, statistical quality management, quality management, quality assurance 1. Introduction As the world of data continues to expand at an increasingly rapid pace, so too does the need of a National Statistics Office to provide relevant, timely, accurate, coherent, easy to understand and accessible data. National Statistics Offices are faced with the challenges of responding to increasing requests for data, reduced resources, the need for improved efficiency and higher staff turnover more than ever before. Those challenges coupled with increased scrutiny of data post global financial crisis and increasing organisational fiscal restraints, mean that new improved methods need to be created to produce statistics at a lower cost whilst maintaining or improving quality. All of these stress factors increase the chance of things going wrong in the statistical process which can impact on an organisation's credibility. The Australian Bureau of Statistics is Australia's official statistical organisation. To be trusted to assist with informed decision making in the community, the Australian Bureau of Statistics uses quality assurance as a fundamental principle around statistical processes. Yet often the approach to quality assurance has been left to the discretion of each statistical area of the organisation to undertake. This has led to variations in how the statistical outputs are quality assured across collections. Due to the increasing scrutiny and expectations by external clients and a need to provide leadership to other providers of statistics in Australia, the Australian Bureau of Statistics has been

2 promoting the use of a standard and systematic framework for quality assuring data. This framework helps to maintain the credibility and relevance of statistical outputs. The method primarily focusses on reducing known statistical risk, where statistical risk refers to the risk that the components of the statistical process fail to meet the quality standards expected, such that the quality of the statistical outputs are affected. The standard procedure that the Australian Bureau of Statistics promotes for use by its statistical subject matter areas is an initiative called quality gates. This initiative is similar to process control which is used in the manufacturing industry. The Australian Bureau of Statistics has adapted this concept and applied it to a statistical process cycle. As a result, quality gates are defined by six components: Placement; Quality Measures; Roles, Tolerance; Actions and Evaluation. A summary of the components of quality gates is below. More detailed information on the components, application and an example of quality gates can be found in the information paper "Quality Management of Statistical Risk Using Quality Gates, Dec 2010, (cat no )". 2. Six Components of Quality Gates 2.1 Placement The key driver behind implementing quality gates into a statistical process cycle is to enable errors to be detected earlier in the statistical process. This allows sufficient time to correct for the issues that the problem presents. To do this quality gates need to be 'placed' at key strategic points throughout the statistical process in order to be most effective at helping to identify errors sooner rather than later. To assist with placement of quality gates it is useful to consider: what can go wrong?; when can it occur?; and what impact can it have? Examining the risk profile of the collection is a good place to start when considering the placement of quality gates throughout a statistical process. Alternatively, placement could be considered as the occurrence of natural beginnings and endings throughout the statistical process. For example, before data is collected a sample needs to be selected. Hence, there is a natural quality gate occurring between these two events to ensure that the statistical process up to that point (before collection), is fit for purpose not only at that point but for the entire statistical process outcomes. Drawing a simple business process map of the statistical process cycle is recommended as a useful way to help identify the placement of quality gates. It is important however to not create too many quality gates as this will detract from their effectiveness. 2.2 Quality Measures The effectiveness of quality gates is highly dependent on the use of appropriate quality measures, which help to identify if there are issues with the statistical process or data. Quality measures can be a summary of a group of similar quality assurance tasks (e.g. Validation). Or they may be distinct detailed items considered to be important for determining the quality of the statistical process (e.g. movements in data items of interest). Quality measures occur throughout the process leading up to quality gates. Issues identified by the quality measure are dealt with at the time that they are discovered. Quality gates provide a consolidation of the quality measures that have occurred up to that point to allow a big picture assessment of the quality of the process and data to be made.

3 2.3 Roles With any process it is important to ensure that roles are clearly defined. Quality gates include roles relating to sign-off, compilation, and stakeholders (providing information, receiving information). Quality gates may involve people from different areas of the organisation, hence it is important to make sure that the role and responsibility of each party to the quality gate is well defined. 2.4 Tolerance Having an expectation about an acceptable level of quality at a given point in time is an important component of a quality gate. Putting in place a pre-determined tolerance assists with making a decision as to whether the statistical process and data are fit for purpose. A tolerance can be qualitative (yes or no) or quantitative (97%). 2.5 Action If the expectations (tolerance) set for a quality gate at a given point in time are not met, having actions clearly stated as to what needs to be done to fix the issue is an important way to ensure effective quality management of the statistical process. The idea of a traffic light is often used to describe this component where green is 'go'; amber is to 'proceed with caution whilst investigating'; and red is 'stop and remedy the situation before continuing with the process'. 2.6 Evaluation Systematically reviewing quality gates in a statistical process cycle to identify improvements is an important aspect of quality gates. It ensures that the specifications of the quality gate are fit for purpose because there is an implicit assessment of each quality gate against the seven dimensions of the Australian Bureau of Statistics Data Quality Framework. For example during the evaluation stage the assessment may reveal that the tolerance was too strict and needed to be relaxed for a particular quality measure; additional quality measures are required for a particular quality gate; and some quality measures need to move into different quality gates, etcetera. 2. History of Development and Implementation The quality gates initiative was developed by the Australian Bureau of Statistics in the second half of The motivation for the development of quality gates was to minimise the risk of the reoccurrence of a significant statistical quality incident. It impacted on the interpretation of the status of the economic and the social conditions in Australia and on the organisation's reputation. This led to an independent review of the processes involved in the quality assurance of the data that was released. The review identified ways in which improvements could be made to that process and applied more generally to all statistical processes across the Australian Bureau of Statistics. There were a number of recommendations for improvements to quality assurance and processes from that independent review of the statistical process in question. These included that "an organisation's strategy throughout development and operation should be to minimise the chance of an error occurring and maximise the chance of its detection" and "Such an (quality risk) analysis should identify areas of greatest vulnerability or greatest impact from an error, and checks decided and implemented to quality assure processes from development, through to implementation and then ongoing operation..."(cornish 2005). The introduction of quality gates to the Australian Bureau of Statistics began with an internal methodological paper that was endorsed by senior managers to be promoted across the statistical subject matter areas of the organisation. The paper also included templates to guide areas in their implementation of quality gates into their statistical processes.

4 The methodology division supplied assistance to those statistical subject matter areas that were keen to adopt and utilise the quality gates method. However, early adoption and implementation of quality gates by statistical subject matter areas was slow in the first few years after their launch to the organisation. As a result of the limited uptake of quality gates the methodology division developed a set of three half day training courses to help promote data quality assurance which were heavily promoted, especially to new staff in the organisation. Of the three, one focussed on the concept of quality gates and was implemented in the middle of The quality gates training course provided specific information to help people better understand the concept of quality gates and facilitated the implementation of them within statistical subject matter areas. A few statistical subject matter areas who were successful early implementers of quality gates in were enlisted to help build the profile and promote the benefit of quality gates in their work processes to other statistical subject matter areas. This was achieved through presentations to other similar statistical subject matter areas (either economic or social statistics) in forums designed for information sharing across statistical subject matter areas. These presentations allowed individual statistical subject matter area experiences in the implementation of quality gates to be shared in a way that allowed robust discussions to take place in order to assist other statistical subject matter areas with their understanding and perception of quality gates. In December 2010, an information paper was published on the Australian Bureau of Statistics website called "Quality Management of Statistical Processes Using Quality Gates, Dec 2010, (cat no )". This paper was designed to provide step by step guidance on how to implement quality gates in a statistical process. It was primarily aimed at organisations involved in the collection, process, analysis or dissemination of statistics. However, many statistical subject matter areas of the Australian Bureau of Statistics found the paper to be useful in assisting them with their work. This information paper helped to consolidate the various tasks that had been undertaken to promote the use of quality gates across the organisation. After the reissue of the September quarter 2009 National Accounts estimates, quality gate workshops for economic areas were instigated at the Australian Bureau of Statistics. This error raised concerns from key users of economic statistics as to the quality and coherence of the economic statistical data produce by the Australian Bureau of Statistics. As a result of these concerns the economic areas of the Australian Bureau of Statistics have placed more importance on implementing quality gates to reduce the risk of errors and consequential lack of coherence occurring in future releases of economic data. A co-ordination area within the economic group became the Project Owner for co-ordinating, managing and monitoring the introduction and development of quality gates in each economic area, this gained momentum in The project owner area launched a campaign to promote quality gates and make economic statistical subject matter areas aware that there would be a requirement for all of them to participate. Campaigning included a seminar, articles in an internal electronic economic area specific newsletter ("The Hub"), promotion of the concept of quality gates and the reporting requirements by economic statistical subject matter areas, as well as a contact for more information. Posters were also displayed across the largest Australian Bureau of Statistics office (where the majority of economic areas reside), to lift the profile of quality gates. These posters used comical situations to promote the use of quality gates. They were displayed near the lifts and stairwells throughout the office to capture all employees' attention as they went about their daily work. An example of a poster used to promote awareness and buy-in from economic statistical subject matter areas (and others) across the Australian Bureau of Statistics is below:

5 The project owner area assisted the co-ordination between the methodology division and the economic statistical subject matter areas to find suitable times to conduct workshops. These workshops focussed on the statistical business process of a particular statistical subject matter area and discussed the potential placement of quality gates and some of the components that would fit within the quality gates. The onus was still on the statistical subject matter areas to continue to develop and finalise quality gates outside of the workshops in order to encourage ownership of the development of quality gates for their statistical processes. The project owner area specified up front in the workshops the reporting requirements required from the economic statistical subject matter areas. The deadlines of six and twelve months for quality gate development and completion were set from the date of the workshop for each economic statistical subject matter area. The project owner area was responsible for reporting back to senior management on the progress of each area in their implementation of quality gates post workshop. Leniency and flexibility around the deliverable of quality gates to the project owner area was provided to some statistical subject matter areas which were already extremely busy implementing new systems, methodologies or processes. This flexibility has helped in the implementation of quality gates across economic statistical subject matter areas, as the workshops and expectations from the workshops took into account their individual needs. 3. Lessons Learnt and Challenges Quality gates are a useful quality assurance tool that can provide a lot of benefit in the long term. However, there are barriers to implementation despite the expectation that the benefits will outweigh the costs. Some of the challenges that the Australian Bureau of Statistics has faced in the implementation of quality gates include, but are not limited to: time pressure; misunderstanding of concepts; system issues; and competing priorities. This section discusses some of these challenges and how they have been addressed to try and overcome them.

6 3.1 Time Pressures Over many years the limited time that a statistical subject matter area has to process information for release has been a barrier to the uptake of quality gates. This includes the notion of areas already trying to adapt their processes to become more efficient as part of an ongoing organisational need to find efficiencies in all aspects of work. This combination of stresses in itself is a large risk to the organisation and its production of statistics that are fit for purpose. Whilst statistical subject matter areas acknowledge the benefit of quality gates and their ability to help manage the quality of the processes during this environmental shift in the organisation, few were able to commit to implementing quality gates into their practices. Senior management outside the methodology division of the Australian Bureau of Statistics started to actively engage in the promotion and use of quality gates across collections. This eventuated from the few high profile errors mentioned earlier, making it into the public domain and near misses occurring more than is liked in the pre-release of data. Time was set aside for quality gate workshops (a minimum of two hours), for the statistical subject matter areas to discuss the statistical risks and quality assurance of collections. This included discussion around how quality gates and quality measures could be implemented to help monitor the process more effectively. Time was also set aside for members of the statistical subject matter area to further develop the quality gates outside of the initial workshop. 3.2 Common Misunderstandings Quality gates, at a broad level, are an easy to understand concept. However, when applied in practice, issues in peoples understanding of the concepts may become apparent. This was the case with the implementation of quality gates. Experience across the Australian Bureau of Statistics in the implementation of quality gates highlighted gaps in the explanation of some of the components underlying quality gates. This included confusion with the components of Placement, Quality Measures and Roles Placement vs. Quality Measures Extra effort explaining the difference between Placement and Quality Measures for a quality gate occurred. Statistical subject matter areas had difficulty in distinguishing between the two and as a result had a quality gate placed wherever a major quality measure needed to be examined. This led to an excessive number of quality gates being created. To help clarify this issue, further effort was put into explaining the difference between the two components and basic examples provided in training and workshops to emphasise this difference. Work was done to highlight that a quality gate consists of multiple quality measures and that quality measures occur throughout the process (and problems fixed if issues identified) before a quality gate is reached. The placement component refers to the point at which an overall assessment of the quality measures (collectively) is made Roles A quality gate can have many stakeholders involved in creation and sign-off. Not all of these stakeholders are from the same statistical subject matter area in the Australian Bureau of Statistics. This caused confusion for some areas in the administration of quality gates, whereby duplicate quality gates were created for different areas to populate rather than consolidating the reporting to one gate. The methodology division provided advice to help statistical subject matter areas better understand the physical monitoring and use of quality gates. Advice on contacting a stakeholder in another statistical subject matter area to obtain specific information pertaining to quality measures within a quality gate was empahsised to owner areas.

7 3.2.4 Use of Existing Quality Assurance Procedures Another issue with misunderstanding the concept of quality gates has been that some statistical subject matter areas have regarded quality gates as an entirely new way to undertake quality assurance for their processes. Quality gates are a systematic way of managing statistical risk. The majority of the content of a quality gate is often already in use by the statistical subject matter area as part of their existing quality assurance processes. Misunderstanding has occurred where areas have thought they needed to develop all new quality measures to populate quality gates. This has led to the protest about not having enough time to implement quality gates as they are perceived to be entirely new work. This is not the case for quite a large proportion of the content. Often what has been found during workshops is that there is a distinct lack of documentation around processes and the workshop has highlighted this as one of the main deficiencies in current practices. 3.3 System Issues There have been several issues raised over the past few years with the implementation of quality gates in regards to system issues. This has varied from an assumption that quality gates needed to have a specific system built in order to use them; to existing systems not being flexible enough to assist in the implementation of quality gates as much as statistical subject matter areas would like. Statistical subject matter areas initially assumed quality gates were an information technology fix to quality assurance. This is not the case. Whilst quality gates can be automated to a degree, for instance quality measures built into systems so that the results are automatically generated for monitoring purposes, one of the myths that the methodology division has tried to debunk is that quality gates require a new system to be built. This myth debunking helped overcome some initial hesitancy in the uptake of quality gates. Another challenge that presents itself in the current climate of system change at the Australian Bureau of Statistics is that a few statistical subject matter areas are waiting to undergo system redevelopment. As a result they are understandably not enthusiastic about putting a lot of effort into developing quality gates into their current processes when their systems may change. However, one thing that quality gates highlight which is immaterial of system issues, is the need for better documentation of all quality assurance procedures for a statistical collection. 3.4 Competing Priorities The implementation of quality gates at the Australian Bureau of Statistics varies across statistical collections due to competing priorities of everyday business requirements. Some statistical subject matter areas have implemented quality gates, other areas have not due to business as usual demands and general collection updates (e.g. new classification implementation). These constraints have meant that for the economic statistics implementation of quality gates, reporting of progress and setting of target deadlines for implementation is tailored to each statistical subject matter area need. 4. Conclusion The success of the implementation of quality gates across the organisation will be measured in several ways. One is to compare performance indicators relating to the number of quality incidents that are identified before and after the dissemination of statistical data. This will be used to monitor the effectiveness of implemented quality gates, both within and across statistical subject matter areas, to address issues highlighted.

8 Success will also be measured by the progress with which statistical subject matter areas develop and implement quality gates into their existing quality assurance processes. In the economic statistics division a dedicated project owner will be reporting back to senior management on the progress of each economic statistical subject matter area in implementing quality gates into their work. Future work in the implementation of quality gates at the Australian Bureau of Statistics will see them being embedded into a new information management transformation system that is currently being developed. This work will require that all statistical subject matter areas have well defined quality measures and quality gates in order to monitor their statistical processes effectively under the new system. Due to this organisational change it is imperative that statistical subject matter areas across the Australian Bureau of Statistics identify and implement quality gates and more importantly identify the corresponding quality measures required for their statistical processes before they are required to move to the new system. This will enable statistical subject matter areas to take advantage of the re-engineering opportunity that this new information management transformation system will afford. It also has the effect of improving knowledge management of statistical processes by promoting an understanding of the data and the processes in order to specify the quality measures required. Quality gates require effort to implement successfully into an organisation because of the challenges involved. However, given all the challenges that the Australian Bureau of Statistics has noted with their implementation, it is not impossible to overcome these with persistence. This persistence is aided by the collective desire of an organisation to succeed and improve on the current situation. Acknowledgements The quality gates framework described in this paper is based on an internal ABS paper prepared by Paul Schubert (2007). Many staff of the ABS, both past and present, contributed to the implementation of quality gates. Special thanks to Jen Dunn, Mike Booth, Bill Allen, Paul Schubert, Emma Farrell and the subject matter areas of the ABS for their work in this area. References ABS (Australian Bureau of Statistics) 2009a, Quality Gates in the Integrated Collection Branch, April 2009, Internal ABS paper, ABS, Canberra. ABS 2009b, Managing Statistical Risk, July 2009, Internal ABS presentation, ABS, Canberra. ABS 2010, Quality Management of Statistical Processes Using Quality Gates, Dec 2010, cat no , ABS, Canberra. Website address: ABS 2012, Quality Gates Implementation Strategy, February 2012, Internal ABS presentation, ABS, Canberra. Conrish, J 2005, Review of retail Trade Error, July 2005, Internal ABS paper, ABS, Canberra. Schubert, P 2007, Quality gates SISC paper, March 2007, Internal ABS paper, ABS, Canberra. UNECE (United Nations Economic Commission for Europe) Secretariat 2009, Generic Statistical Business Process Model, Version 4, April 2009, ENICE, Geneva.