Towards a Quality Framework for Composite Indicators. Enrico Giovannini (OECD Chief Statistician)

Transcription

1 Towards a Quality Framework for Composite Indicators Enrico Giovannini (OECD Chief Statistician)

2 1. Quality approaches and dimensions A lot of work has been done in recent years to apply the concept of quality to statistical data. IMF, Eurostat, Statistics Canada and other NSOs have identified various sets of quality dimensions. Quality is usually defined as fitness for use in terms of user needs.

3 Two main approaches: IMF and Eurostat IMF: This framework views quality through a prism that covers governance of statistical systems, core statistical processes and observable features of the outputs. The Data Quality Assurance Framework (DQAF) addresses a broad range of questions that are captured through the prerequisites of quality and five dimensions.

4 Prerequisites: How is the quality of statistics affected by the legal and institutional environment and resources, and is there quality awareness in managing activities?

5 Dimensions: Assurance of integrity: What are the features that support firm adherence to objectivity in the production of statistics, so as to maintain users confidence? Methodological soundness: How do the current practices relate to the internationally agreed methodological practices for specific datasets?

6 Accuracy and reliability: Are the source data, statistical techniques, etc. adequate to portray the reality to be captured? Serviceability: How are users needs met in terms of timeliness of the statistical products, their frequency, consistency, and their revision cycle? Accessibility: Are effective data and metadata easily available to data users and is there assistance to users?

7 Eurostat: This framework focuses on the statistical outputs as viewed from the users and works its way back to the underlying processes only where the outputs do not yield a direct measurement. It is based on seven dimensions: Relevance. Are the data what the user expects? Accuracy. Is the figure reliable? Comparability. Are the data in all necessary respects comparable across countries? Coherence. Are the data coherent with other data?

8 Timeliness and punctuality. Does the user get the data in time and according to pre-established dates? Accessibility and clarity. Is the figure accessible and understandable? The idea of Eurostat quality definition is to ensure that certain standards are met in aspects of statistical production that are subkect to quantifiable measures, such as standardised measures (e.g. measurement errors).

9 Conclusions: There are several areas of commonalities and the two approaches were modified to further harmonize them. IMF focusing on process-oriented indicators and providing qualitative measurements. Eurostat focusing on output-oriented indicators and providing, to the extent possible, quantitative measures. They are not immediately applicable to composite indicators.

10 OECD approach The OECD has developed its own approach to improve statistics it disseminates. For an international organisation the quality of statistics disseminated depends on two dimensions: the quality of national statistics it receives; the quality of its internal processes for collection, processing, analysis and dissemination of data and metadata

11 Elements of the OECD Quality Framework Four pillars: a definition of quality and of its dimensions; definition of internal quality guidelines covering all phases of the statistical production process; a procedure for evaluating the quality of ongoing statistical processes and output on a regular basis; a procedure for assuring the quality of new statistical collections.

12 Seven dimensions: Relevance: The relevance of data products is a qualitative assessment of the value contributed by these data. Accuracy: The accuracy of data products is the degree to which the data correctly estimate or describe the quantities or characteristics that they are designed to measure. Credibility: The credibility of data products refers to confidence that users place in those products based simply on their image of the data producer, i.e., the brand image.

13 Timeliness: The timeliness of data products reflects the length of time between their availability and the phenomenon they describe. Accessibility: The accessibility of data products reflects how readily the data can be located and accessed. Interpretability: The interpretability of data products reflects the ease with which the user may understand and properly use and analyse the data. Coherence: The coherence of data products reflects the degree to which they are logically connected and mutually consistent.

14 2. Relevant dimensions for composite indicators OECD IMF/Eurostat Relevance Relevance/Serviceability Accuracy Accuracy/reliability/meth. soundness Credibility Integrity Timeliness Timeliness Accessibility Accessibility Interpretability Clarity Coherence Consistency/Comparability

15 The quality of composite indicators depends on the following aspects: the quality of basic data used to construct the indicators; the quality of procedures used to compute the indicators; the quality of approaches used to disseminate the indicators. We will make reference to indicators for international comparisons.

16 For basic data the most important dimensions are: Accuracy Timeliness Coherence over time and across countries (comparability) The credibility of original sources is also very important (official statistics).

17 For procedures to compute indicators the most important dimensions are: Relevance Accuracy (method. soundness) Coherence Reliability (robustness) Interpretability

18 For approaches to disseminate indicators the most important dimensions are: Accessibility Interpretability Credibility

19 An overall view Basic data Computation Dissemination Timeliness Accuracy Accuracy (method.) Coherence Coherence Reliability (robust.) Interpretability Interpretability Accessibility Credibility Relevance

20 Main risks for the overall quality of composite indicators Inaccurate, incoherent, non-credible and late sources Unbalanced or biased choice of individual indicators Inconsistent approaches used in various steps (standardisation, aggregation, etc.) Lack of robustness analysis Limited availability of metadata on processes adopted for computing composite indicators Incorrect presentation of results

21 Proposed structure of the quality framework for composite indicators Methodological guidelines for each phase of the process Definition of quality dimensions (relevance, timeliness, accuracy, coherence, reliability, interpretability, accessibility, credibility) Quality checklist

22 3. The way forward (1) Agree on the proposed quality framework Prepare methodological recommendations for each phase of the process: identification of the purpose; design of the indicator; choice and evaluation of available data; choice of procedures and computation of the indicator; test for robustness; interpretation of results; dissemination of final data and metadata.

23 The way forward (2) Develop a quality checklist to assess the overall quality. The results of the checklist should be included in metadata disseminated with data