Validation of Household Consumption Survey

Transcription

1 International Comparison Program Validation of Household Consumption Survey Draft version Operational Guide

2 Contents 1. INTRODUCTION VALIDATION OBJECTIVES, MEANS AND LIMITS INTRA-COUNTRY VALIDATION INITIAL DATA VALIDATION STATISTICAL TESTS FINALIZATION OF DATA INTER-COUNTRY VALIDATION INITIAL DATA VALIDATION VALIDATION OF METADATA VALIDATION AT BH LEVEL VALIDATION AT AGGREGATED LEVELS TEMPORAL ANALYSIS FINALIZATION OF DATA GLOBAL VALIDATION VALIDATION OF GCL ITEMS VALIDATION OF REGIONAL PPPS VALIDATION OF GLOBAL PPPS ANNEX: SUMMARY OF VALIDATION STAGES, PHASES AND STEPS List of abbreviations BH DT GO GCL NC QT RC Basic Heading Dikhanov Table Global Office Global Core List National Coordinator Quaranta Table Regional Coordinator 2

3 Validation of Household Consumption Survey 1 1. Introduction National Coordinators (NCs) of the countries participating in ICP regional comparisons provide the Regional Coordinators (RCs) with a set of national quarterly and annual purchasers prices for a selection of items chosen from a common list of precisely-defined items. These regional lists include both purely regional and global items belonging to the Global Core List (GCL). The first set of prices refers to quarterly national average prices and the final set of prices refers to annual national average prices of the year of the comparison and they both cover the whole range of final goods and services included in GDP. These prices are used to calculate purchasing power parities (PPPs) for basic headings; the basic heading PPPs are further used to derive intra-regional measures of price and volume relatives for the countries participating in the comparison. The regional PPPs are then linked to form a global set of PPPs and measures of price and volume relatives. The measures are subsequently published by the RCs and the World Bank thereby reaching a variety of users including policy makers, economic analysts, researchers, politicians and journalists as well as the general public. It is therefore essential that the prices on which the PPPs are based are rigorously checked and corrected for error in other words validated - before the final PPPs are calculated. Only after this step measures can contribute accurately to informed debate. The validation of price data is thus an important phase in producing PPPs. Being a multilateral comparison, the ICP exercise does not only need correct, but also comparable price data. To ensure both correctness and comparability, the validation process requires close cooperation and collaboration between NCs, RCs and the Global Office. The purpose of this chapter is to provide a step-by-step breakdown of the complete validation process for the household consumption survey. The ICP validation process comprises of three stages: (1) intra-country validation at national level, (2) inter-country validation at regional level, and (3) global validation at global level. Chart 1 below illustrates the stages. The Intra-country validation is directed at a country s individual price observations and the resulting average prices. The objective is to verify that price collectors within a country have priced comparable products and have priced them correctly. It is carried out mainly by the country s NC. There may or may not be input from the RC, depending on the region. Inter-country validation takes place after intracountry validation and is carried out by the RCs and NCs, but the overall coordination is done by the RCs. Inter-country validation is directed at the average prices reported by participating countries or in some cases at the individual price observations and the price ratios that the prices generate between the countries. The objective is to verify that price collectors in different countries have correctly priced comparable products. The final validation stage is the global validation where the Global Office, together with the NCs and RCs, validate prices for the Regional and Global Core List items, regional PPPs and subsequent global PPPs. The objective is to ensure that prices collected for the items across the regions are comparable and that the linked global PPPs are plausible in wider terms. 1 This chapter is prepared by Marko Rissanen with input from Nada Hamadeh, Biokou Mathieu Djayeola, and Mizuki Yamanaka, based on the material prepared by David Roberts. 3

4 Chart 1: Validation Stages Intra-country For regional and GCL items 2. Regional validation NCs, RCs Inter-country across regions For regional and CGL items and regional PPPs 1. National validation NCs Inter-country For regional and GCL items 3. Global validation NCs, RCs, GO Prices for consumer products can be collected on monthly or quarterly basis, with the final aim to produce annual average prices. The validation at country level should be carried out following the price collection schedule either on monthly or quarterly basis. The validation at the regional level should be carried out according to the agreed validation schedule, preferably on a quarterly basis. The global level validation is to be carried out on a quarterly basis. Various phases of validation require careful planning of the overall validation timetable. In addition, timeliness is a crucial aspect requiring serious consideration. First, because of the need to release data on a timely basis and second, because the longer the delay between price collection and verification the more difficult it becomes to correct erroneous prices. It may also happen that countries within a region are not able to follow a common timeline for the validation. In this case RCs may need to create validation groups for the regions countries. Moreover, the inter-country validation is an especially iterative process requiring a number of rounds of editing and verification. Possible errors are found by identifying prices that diverge significantly from other prices in the series. They are detected by having a measure of divergence that is greater than a given critical value or a value that falls outside a given range of acceptable values. The divergence measures are generally defined by the parameters of the series being edited, parameters such as the average and the standard deviation. Hence, if some of the possible errors identified in the initial edit are found to be actual errors and are corrected, the parameters of the price series will change and so will the divergence measures of each price remaining in the series. A second edit will find new possible errors that will need to be verified. Again, when actual errors are corrected, the parameters of the price series will change which may lead to more possible errors being detected if a third edit is done. Usually the number of new possible errors falls as validation progresses until the return on further rounds is considered marginal and not worth pursuing. Chart 2 below illustrates validation iterations. The following sections deal with each validation stage individually. Stages are divided into validation phases which are further divided into validation steps to be carried out. Before moving on to the validation process, it needs to be considered what are the objectives and related general means of validation as well as limits faced. 4

5 Chart 2: Validation iterations Original or edited price data and metadata sent Data edited based on the findings Validation tables or comparisons calculated Tables and comparisons analyzed by all parties Tables and comparisons delivered to all parties 2. Validation objectives, means and limits The first objective of the validation is to clean data from pricing errors. Two types of non-sampling errors can be identified: price error and product error. A price error occurs when price collectors price products that match the item specification but fail to record the price correctly. A price error can also occur throughout the process of reporting and transmitting a price which was initially recorded correctly. Associated with each price is a quantity. There is the specified quantity (the quantity to be priced) and there is the reference quantity (the quantity to which the price collected is to be adjusted). Price error can also arise because, even though the price is correctly recorded, the quantity priced is recorded wrongly (or is recorded correctly and error is introduced later during processing), so that the adjusted price for the reference quantity, which is the price that is validated, will be wrong as well. A product error occurs when price collectors price products that do not match the item specification and neglect to report having done so. This can be because they are not aware of the mismatch, such as when the item specification is too loose or because they price a substitute product as required by the pricing guidelines but do not mention this on the price reporting form. Price collectors are usually instructed to collect the price of a substitute product if they are unable to find the specified product. They are further instructed to flag the substitution and to note the differences between the substitute product and the specified product. Flagging brings the substitution to the attention of the NC who, together with the RC, can then decide what to do with the price collected. It may be possible to adjust the price for 5

6 quality differences between the product priced and the product specified. Alternatively, if other countries report prices for the same substitute product, price comparisons can be made for the substitute product as well as for the product originally specified. If neither one of these options is possible, the price will have to be discarded. Substitution does not in itself introduce error; it is the failure of price collectors to flag and document the substitution that leads to a product error. Means to detect product and price errors are to identify extreme observations or outliers. These are observations that diverge so much from the average as to be treated as prima facie implausible and therefore requiring further investigation and verification. It should be noted that the policy is not to reject outliers automatically but to investigate whether or not they are genuine extreme observations. An element of judgment is still required. The amount of divergence that triggers further verification depends largely on the variance of the price observations. It is also possible that the high price variation reflects the country s situation correctly. The Regional Coordinator will have to rely on the country s experience and local market knowledge in these circumstances. There is no exact rule about what makes a price an outlier. The Regional Coordinator is advised to discuss the thresholds for his region with market and industry experts. It is important to remember, however, that all possible country specific situations may not fit into the regional thresholds. The second objective of the validation is to ensure the comparability of prices. The purpose of the price comparison is compare comparable products i.e. like with like - Loose item specifications or difficult survey areas may introduce product errors because the products whose prices are compared may not be the comparable ones. For example, if a generic product specification - i.e. a specification that does not define the brand to be priced - is used; it may embrace a wider range of products than was intended when the specification was drawn up. In this case, errors may be introduced into the estimated PPPs because the products differ significantly in their characteristics. Errors due to loose specifications or difficult survey areas cannot be attributed to the data of any particular country as they are not errors per se at the country level, given that the data is coherent at sub-national level. The role of the RCs and the Global Office is to ensure that collected prices are comparable by analyzing both price data and metadata. It should also be noted that as the product errors at regional may not be errors at country level, it is not always clear for NCs why the given edits are proposed. Thus even if a country would actively participate in detecting price and product errors, it is the liability of coordinators of the program to guarantee the comparability of the final prices. The points made above relate mainly to non-sampling errors. Sampling error on the other hand occurs because the prices on which the PPPs are based are collected from a sample of outlets rather than from all outlets. An important distinction between the two types of error is that sampling errors would disappear should prices be collected by an enumeration of all outlets, but non-sampling errors would not as they would continue to occur regardless of the types of outlets or the number of observations. Sampling error should be controlled before price collection through sample design; non-sampling errors should be handled both before and during price collection through a good survey design and management and after price collection through validation. ICP validation is mainly directed at nonsampling error. The object is to minimize, if not eliminate, the incidence of non-sampling error among regional price data after collection. This is achieved by editing and verification. Regarding the sampling errors, the possible validation actions are limited. However, RCs should try to assess the plausibility and comparability of sub-national sample and shop sample across the region. 6

7 Finally it is worth noting that prevention is preferable to correction. The incidence of non-sampling error can be significantly reduced through good survey design and management. These require price collections to be carefully planned, efficiently carried out and properly supervised; item specifications to be sufficiently detailed to enable price collectors to identify products unambiguously in the outlets they visit; price collectors to be well trained, given clear instructions and provided with price reporting forms that are user friendly; fieldwork to be closely monitored to ensure that price collectors record the prices, quantities and other data required; and staff engaged in processing and validating the prices to be properly trained and supervised. Validation complements good survey practice. 3. Intra-country validation Intra-country validation Inter-country validation Global validation The first validation stage within the ICP validation process is the intra-country validation at country level. Within this stage it is verified that price collectors within the same country have priced products that match the item specifications and that the prices they have reported are correct. Also the relevancy of the survey frame, i.e. outlet and location sample, needs to be verified. The intra-country validation can be carried out entirely by the NC itself without the assistance or intervention of the regional coordinator, although the regional coordinator can provide a valuable second opinion about doubtful or marginal cases. Intra-country editing searches for discrepancies between the requested and reported prices and metadata and analyzes extreme values; first among the individual prices that a country has collected for each item, and then among the average prices for these items. The editing and subsequent verification is the responsibility of the country s NC and is carried out without reference to the price data of other countries. When the NCs complete the intra-country validation, they provide the RC with validated average prices for the items priced plus the importance indicator, the variation coefficient, the minimum- maximum price ratio and the number of price observations for each of the average prices reported. It is also encouraged that the NCs submit the price observations and related metadata in order to facilitate throughout inter-country validation. The Intra-country validation stage consists of 3 validation phases that are initial data validation (A), statistical tests (B) and finalization of price data (C). During the first phase the price data generated by price collectors are merged and the first checks are made. Second phase analyses the data more systematically using statistical means. Final phase prepares that data to be delivered to RC. Each phase and underlying steps is described in the following sections Initial data validation Inital data validation (A) Statistical tests (B) Finalization of data (C) Step A1 Add price and metadata to validation software 7

8 The first step in the validation process is to enter the individual prices and related metadata into the validation software. It is preferable that this be done by staff other than the price collectors. A strict division of labor between price collection and data entry ensures that the operations are completely objective. For the purposes of validation, data entry should be done as close to the price collection as possible while price collectors and outlet personnel still have a clear recollection of the circumstances prevailing when the prices were collected. Ideally, the individual prices would be entered and screened the day following their collection as this will allow field supervisors to catch and correct the mistakes of price collectors from the beginning. Step A2 Check prices and metadata for errors and discrepancies Next step is to check the entries to ensure that item codes are correct; that codes and the corresponding price observations have been entered in sequence; that there are only numeric entries for reference quantity, quantity observed and price observed and for those fields, such as location of outlet or type of outlet, using numeric codes; that the quantity observed is in the same unit as the reference quantity. It should also be checked that the prices seem to be initially plausible and according to the NC s knowledge of the market. Boxes 1 and 2 below give examples of quantity related problems. In addition following points should be considered: Alternative outlets. In most cases the price collectors have been provided with a list of items and their detailed specifications, and also a list of outlets from which to collect the prices of the products. The price collectors are instructed to visit the outlets listed. If for some reason they are obliged to choose another outlet, the reasons should be explained and detailed information should be provided about the outlet from which the prices were actually collected. The NCs should ensure that the alternative outlets are good substitutes. Alternative products. The price collectors make a written note of the prices on the paper forms provided together with any accompanying notes, comments or explanations. They are instructed to collect prices only for the items listed, if possible. If a product is not available exactly as specified on their lists, a price collector may substitute another product within the same outlet whose characteristics are very similar to, even though not identical with, the item specification. The NC should assure that alternatively priced products are comparable with the rest of the priced products and with the item specification. Box 1: First example on quantities It is often the case that the price reported refers to a different unit of quantity from that requested and specified in the item list. The price reported may refer to the wrong weight: For example, to 250 or 500 grams instead of the required quantity unit of a kilo: or, it may be the price per egg instead of per dozen eggs, or per pint or gallon instead of per liter. However, price collectors are obliged always to report the actual unit of quantity, whether or not it coincides with the quantity requested. Thus, the use of different quantity should not introduce an error as it will be recorded and known. Either the price collected can be adjusted to convert it into the price for the quantity requested, or if this is not feasible the price may have to be deleted. 8

9 Box 2: Second example on quantities Suppose that a price collector is only able to obtain a price for a kilogram package for an item specification of gram. The priced package size falls well outside the pre-determined range. In this case, an automatic proportional adjustment on the basis of the relative quantities may not be appropriate. In general, a bulk purchase cannot be treated as if it were not different from several smaller purchases adding up to the same quantity. In this case, for example, a package of a kilogram cannot be treated as if it is the same, from a consumer s point of view from, say, as four 250 gram packages. Thus, the price for a kilogram package may have to be rejected because it is too difficult to make an appropriate quality adjustment for such a large difference between the reported quantity and the reference quantity. Step A3 Check that prices are sub-nationally plausible A large country may be divided into different areas with price collections carried out separately. It is essential that the NCs ensure that the data received from different areas are correct and plausible, based on the knowledge of the regions. Separate editing and validation steps may need to be carried out for the different areas. The relationship of the various areas to the country as a whole resembles the one between the countries and the ICP region; even large differences in prices between the different parts of the country may be entirely appropriate. In any case, realistic national average prices have to be the end results. Step A4 Check that prices are temporally plausible Temporal comparison analyzes possible inconsistencies in price data which may be the result of price errors. Prices can be compared temporally either between ICP rounds or between price collection periods within an ICP round. As the intra-country validation deals with the price observations and respective average prices at item level, the most reasonable temporal validation is done between the price collection periods within an ICP round. Temporal validation between ICP rounds can be conducted at price level, but the following points have to be taken to account: The data at item level may not be available for previous round(s) If the item level data is available, it has be ensured that the item definitions are comparable between the rounds i.e. that the item parameters have not changed substantially and that the reference units and quantities are the same It should also be noted that comparison of prices directly between different ICP rounds may not be valuable due to real price level changes i.e. inflation or deflation - within a country. Thus without a CPIadjustment the consistency of the prices may be difficult to analyze. On the other hand direct comparison can be used to check whether PPP price level change is in line with the respective change observed CPI change. The temporal analysis between price collection months or quarters can be done without a CPIadjustment as prices are generally collected for identical item definitions. As the average prices within a given year should be relatively stable within a country, large inconsistencies may be a sign of errors in 9

10 the data. The possible reason for the variation should thus be investigated and a reasonable justification or correction should be provided. A challenging aspect of temporal analysis is the difficulty to directly identify the potential source of error. This is because potential causes can be either the previous or current dataset or, in the case of temporal analysis between the ICP rounds, problematic CPI indices. However the main advantage of the temporal comparison is to flag cases for further analysis. Step A5 Check the important items As expenditure shares below the Basic Heading level are not available, each priced item has to be determined either as important or less important following the estimated expenditure share for the items within a BH. Box 3 below gives guidance on how to determine important products. During the intra-country validation it needs to be verified that the items selected as important are following the respective guidance for determining an item as important. It should also be noted that if the selection is done prior the actual price collection, the market situation may have changed causing an improper selection. Regarding the total number of important items for each BH, it needs to be checked that: At least one Global Core List item and preferably at least one regional item are classified as important. The share of important items vis-à-vis less important items is realistic. In most cases 50 to 80 percent of the items per BH should be classified as important. Higher shares may be reality for example if a BH includes very few items. Lower shares may be matter-of-fact if the given market within a country is atypical in relation to the regional item list. As both the selection of important items and the number of them within the BH have an impact on the PPP calculations, it is important to validate the estimates made. Box 3: How to determine if an item is important or less important? 1. If an item is in the CPI - If an item is the same as, or very similar to, one that is included in a country s consumer or retail price index, the country should always classify the item as important. 2. Use expert judgment/common knowledge - The statistician can call upon his or her own knowledge of what are widely available and commonly bought brands of cigarettes, soap powder, biscuits, etc. 3. Ask expert - Most often the experts will be shop-keepers. The success of their business depends on knowing which products are best sellers and which are bought less often Statistical tests Inital data validation (A) Statistical tests (B) Finalization of data (C) After data has been added to the validation software and initially validated, 4 statistical tests are to be conducted. Detailed price information and the statistical tests are read using two tables. The first one is called Average Price Table, which shows average prices as well as results of Min/Max-Price-ratio test and Variation Coefficients for each item. The second table is called Price Observation Table, which gives 10

11 the individual price observation as well as results of deviation and T-Value tests. Table 1 below gives an example of an Average Price Table, whereas Table 2 presents an example of a Price Observation Table. Statistical tests are done for the prices that are converted into a reference-quantity i.e. each price observation within an item has been converted to same quantity or volume, such as 1 kg or 1 liter. Regarding the suggested limits for the statistical test, Box 4 below discusses on these, whereas Boxes 5 and 6 further below gives guidance on the editing of price data based on the results of statistical tests. Box 4: Suggested limits for statistical tests For each statistical test introduced below a proposal for acceptance limits is given. However it should be noted that these limits are suggested as default values and they may need to be modified depending on the situation. It should be stressed that each BH or aggregate may have its own limits or critical values. For example with the Coefficient-of-Variation test, limits for the prices of new cars have to be tighter than for services, whose prices tend to vary substantially more. Secondly it may be needed to apply different limits for different countries, if, for example, it is known that the sub-national price differences are high. Step B1 Check average-price-table for extreme values using Min/Max-Price-ratio test The first test is the Min/Max-Price-ratio test, which checks the ratio of minimum reference-quantity price for the product to the observed maximum reference-quantity price observed for the product. The formula for calculating Min/Max-Price-ratio is as follows: Min/Max-Price ratio= (Min Price / Max Price) Suggested limit for the Min/Max-Price-ratio test is 0.5 or below. An item with a min/max-price ratio below 0.5 fails the test and will be flagged in the average-price-table as having a high value that needs to be verified. Price data of all products having this flag needs to analyzed and corrected or approved. Step B2 Check average-price-table for extreme values using Variation Coefficient The second is the coefficient-of-variation (CV) test, which checks the standard deviation for the product expressed as a percentage to the average reference-quantity price for the product. The formula for calculating CV is as follows: Coefficient-of-Variation= ((Standard deviation / Average)* 100) Suggested limit for the CV test is 39 percent or below. An average price of the item with a CV greater than 39 percent fails the test and will be flagged in the average-price-table as having a high value that requires verification. Price data of all item having this flag needs to analyzed and corrected or approved. Step B3 Check price-observation-table for extreme values using deviation test The third test is the deviation test which measures deviation of the reference-quantity price for a price observation to the average reference-quantity price for the product. The formula for calculating deviation is as follows: Deviation= ((Price - Average Price) / Average price) *100 11

12 Suggested deviation limit is 50 % or below. A price observation with deviation higher than this fails the test and will be flagged in the price-observation-table as having an extreme value that needs to be checked. All prices having this flag needs to analyzed and corrected or approved. Step B4 Check price-observation-table for extreme values using T-Value test The final test is the T-Value test, which checks the ratio of the deviation of the reference-quantity price for a price observation from the average reference-quantity price for the product to the standard deviation of the product. The formula for calculating deviation is as follows: T-Value = ((Price - Average Price) / Std. Deviation) Suggested limit for the T-Value test is 2.0 or below. A price observation with a ratio greater than 2.0 fails the test and will be flagged in the price-observation-table as having a high value requiring investigation. All prices having this flag needs to analyzed and corrected or approved. Box 5: Editing of price data based on the results of statistical tests Editing for product errors and price errors involves identifying prices that have high values that is, prices whose value is determined to be either too high or too low vis-à-vis the average according to given criteria. The price may score a value for a given test that exceeds a pre-determined limits or its value may fall outside some pre-specified range of acceptable values. Both are standard ways of detecting errors in survey data and both are employed by the ICP. Prices with extreme values are not necessarily wrong. But the fact that their values are considered to be outside of pre-specified range suggests that they could be wrong, that they are possible errors and need to be investigated. It is not ICP practice to reject prices with extreme values outright but to establish first whether or not they are genuine observations. Once this is known, it can be decided how to deal with them. Prices with high values that are found to be wrong are errors and should be corrected or dropped, while prices with extreme values that are shown to be accurate observations are outliers and should be retained if they are part of the population defined by the rest of the price observations. In practice, it is not unusual for outliers that meet this criterion to be corrected that is discarded or replaced by an imputed value - in order to remove the noise they introduce into the data set. Box 6: High but justified Coefficient-of-Variation (CV) A high or even extreme CV can be due to reasons other than a straightforward product or price error. The price of the product may vary greatly between different types of outlet or the product may not have been priced consistently across outlets because either the product specification is too broad or it has been interpreted differently by different price collectors. Provided the price observations are correct and a comparable product has been priced across outlets, price variation arising from different outlet types is an economic fact of life. The average price should be kept and the reason for the variation explained to the RC. It is possible that the mix of outlet types selected for the survey does not reflect the distribution profile of the product in question. This should be investigated and, if necessary, the mix adjusted as appropriate by suppressing prices from those types of outlet that are overrepresented or by duplicating the prices from those types of outlets that are underrepresented. Items with average prices whose variation is caused by too broad specifications or inconsistent pricing across outlets should be deleted unless they are important and the basic heading does not have a sufficient number of items. In the latter case, they and their average prices should be retained provisionally. This information should be noted in relation to the item. Later it can be decided with the RC whether the products should be dropped, retained or split on the basis of what other NCs in the region have reported. 12

13 Table 1: Example of Average Price Table Product Code Product Name Importance Number of Price Quotations Average Price Min Price Max Price Min/Max Price Ratio Var. Coefficient Rice, basmati, first degree Rice, long grain Rice, Uncle Ben Rice, white Rice, white, 25 % broken Rice, Egyptian Rice, Indian

14 Table 2: Example of Price Observation Table Product Code Product Name Locatio n Code Location Name Location Type Code Outlet Code Outlet Name Outlet Type Code Obs Date (yyyy-mmdd) Obs. Qty Obs Price Conv. Price Deviation Rice, basmati, first degree 12 Bawshar Market Rice, basmati, first degree 11 Mutrah Market Rice, basmati, first degree 11 Mutrah Market Rice, basmati, first degree 11 Mutrah Market Rice, basmati, first degree 11 Mutrah Market Rice, basmati, first degree 11 Mutrah Market T Value Rice, long grain 12 Bawshar 1 56 Market Rice, Uncle Ben 12 Bawshar Market Rice, Uncle Ben 12 Bawshar Market Rice, Uncle Ben 12 Bawshar Market Rice, Uncle Ben 12 Bawshar Market Rice, Uncle Ben 11 Mutrah Market Rice, Uncle Ben 11 Mutrah 1 14 Market Rice, Uncle Ben 12 Bawshar 1 56 Market Rice, Uncle Ben 0 OMAN 1 TS Test Rice, white, 25 % broken 11 Mutrah 1 1 Market Rice, white, 25 % broken 11 Mutrah 1 1 Market Rice, white, 25 % broken 11 Mutrah 1 1 Market Rice, white, 25 % broken 12 Bawshar Market

15 3.3. Finalization of data Inital data validation (A) Statistical tests (B) Finalization of data (C) Step C1 Confirm price data and metadata to be intra-country validated Before submitting the data and metadata it needs to be confirmed that steps from A1 to B4 have been successfully conducted and that the data has been intra-country validated. Step C2 Submit price data to Regional Coordinator After the data is confirmed to be intra-country validated it can be send to the RC. The data and metadata has to be in a correct pre-specified format. 4. Inter-country validation Intra-country validation Inter-country validation Global validation The inter-country validation stage starts as soon as a sufficient number of countries within the region have submitted intra-country validated price data. Inter-country validation is a collective process involving the regional coordinator and a group of countries and it is designed to establish that price collectors in different countries have priced products that are comparable between countries in other words, they have all interpreted the item specifications in the same way - and that the prices they have reported are correct. Inter-country editing looks for extreme values among the average prices as well as discrepancies with the reported metadata that the region s NCs have reported to the RC for the same items within a basic heading. In order to do this, the average prices, which are expressed in national currencies, have to be converted to a common currency. After being converted, the average prices of each country are checked against the average prices of the other countries in the region. This cannot be done effectively without the lead and active participation of the RC and the NCs agreeing to share their average prices with each other. The main tools for the inter-country validation are Quaranta Table (QT), named after Vincenzo Quaranta who first proposed them for use in the European PPP program in 1990 and Dikhanov Table (DT) named after Yuri Dikhanov who first proposed them for use during the course of ICP These tables are explained in detail in the preceding chapter Validation Tables. Both validation tables are prepared by the RCs, but they should be circulated to all countries within region. NCs are expected to make their own assessments of their data relative to other countries independently of the regional coordinator. In other words NCs are expected to be proactive during the inter-country validation. The RC may refer doubtful prices back to the countries for clarification or further investigation. The countries may then revise or correct their data, or seek to justify the data as reflecting the market situation in their countries. Intercountry validation is an iterative process and can involve several iterations or rounds before being completed as explained in the introduction section of this chapter. After each round, as incorrect or incomparable prices are removed or corrected, the PPPs will become more reliable. Box 7 below discussed on the editing of price data based on the analysis of Quaranta and Dikhanov tables 15

16 Inter-country validation stage consists of 6 validation phases. The first phase, initial data validation, is a quick validation phase before any in-depth validation to find out extreme problems with the data that would make any deeper data analysis impossible by blurring the validation tables. The second phase is validation of metadata which seeks to identify possible large deviations with requested and reported data and any other comparability issues that may not be visible as outlier indices. Validation at BH level is the main phase of intra-country validation and most of the edits are done within this phase. Validation at aggregated levels takes BH average prices to a wider context. Temporal analysis checks the consistency of PPPs or PLIs with regard to a previous ICP round or previous quarters within an ICP round. Finalization of data concludes the inter-country validation phase. Box 7: Editing of price data based on the analysis of Quaranta and Dikhanov tables Editing a basic heading with a Quaranta table or an aggregate with a Dikhanov table entails identifying average prices with low or high values or, more precisely, the XR-Ratios or CUP-Ratios with low or high values. Average prices underlying these ratios flagged as deviant values are only possible errors. Regardless of how strictly defined the criteria used to identify these errors is, they are not necessarily errors by definition and cannot be therefore automatically removed. They have to be referred back to the NCs reporting them for verification. NCs are required to investigate the average prices returned to them as possible errors and to confirm whether they correct or incorrect. When prices are found to be incorrect, NCs are expected to correct them, otherwise they are suppressed. But if they are found to be correct, they are outliers and the decision has to be made whether to keep them, to replace them with an imputed value or to drop them, not necessarily an easy decision. Some of the deviations, even larger ones, can be legitimate. Individual economies may have particular pricing policies, such as low fuel prices in some of the oil-producing countries. Such prices may be flagged as extreme values, but they would be not incorrect, and it would be wrong to remove them despite the noise they may introduce into the dataset Initial Data Validation Inital data validation (D) Validation of Metadata (E) Validation at BH Level (F) When the data received from the NCs is compared for the first time, it normally includes number of severe errors that make any in-depth analysis impossible since the indices in the validation tables are blurred by the potential extreme problems with data. Most common extreme errors are quantity related or simple typing mistakes which may not be visible at country level, but become clear when data for different countries are compared. Since focus is only on the extreme problems with the data, one initial validation round should be enough to conclude this validation phase. Step D1 Add price and metadata to the validation software Inter-country validation starts with the RC entering the average prices reported by the NCs into the validation software. First problems with the data are normally encountered during this step stemming from the incorrect data formats. Step D2 Calculate initial validation tables After the prices have been added it is possible to produce an initial Quaranta Table. It is recommended that the QT instead of DT is used within this validation phase. This is because at early phases of data validation BH PPPs and respective item PPP-Ratios and CPD residuals are not reliable as they are based 16

17 on invalidated data. DT focuses on residuals with color scheme whereas QT gives equal weight to the Exchange Ratios of Price (XR-Ratios) and the PPP Price Ratios (PPP-Ratios). Step D3 Check initial validation tables for extreme XR-Ratios XR-Ratios are standardized price ratios based on exchange rate converted average prices. The ratios are thus a proxy measure that allows the average prices for a product to be compared across countries. Each of these ratios refers to a particular item in a particular country. An extreme XR-ratio means that the national average price for the product in question is very much out of line when compared with the prices of the same product in other countries when they are all converted into the common numeraire currency using the exchange rate. The focus should be only on very high or low XR-Ratios, meaning values approximately below 40 or over Validation of Metadata Inital data validation (D) Validation of Metadata (E) Validation at BH Level (F) In addition to the validation based on the validation tables, the metadata underlying the average prices should be checked separately against large deviations between requested and priced products. For example, in case a country has systematically priced products with larger quantities that are requested for a BH, this may have resulted in bias that is not visible at BH level indices. However references to the metadata are also to be done within the following validation phases when potential outlier prices are investigated. Step E1 Check metadata for large deviations between requested and priced products Small deviations with metadata should be allowed, but it should be considered when a product is not anymore the one priced by the other countries. Common rules for allowed deviation should be accomplished while taking into account that these can differ across the BHs. Box 8 below gives an example of edits done based on the metadata validation. Box 8: Example validation dialogue for a Men's tennis socks item Q: The item specification asks for multipacks, but you have priced both single packs and multipacks. This results in a very high coefficient of variation. Please remove the incorrect package sizes. A: OK, incorrect package types deleted. Step E2 Check metadata to ensure that survey frames are comparable across the countries Survey frames established by the NCs are following the typical national purchasing patterns. However errors or very different purchasing patterns may result in incomparable data. For example, data from a country that prices products solely at capital city department stores may not be comparable to those from other countries that collect prices from balanced mix of outlets and locations. RCs should try to 17

18 assess the comparability of national survey frames to the largest extent possible while understanding that in practice the means for this may be limited Validation at BH level Inital data validation (D) Validation of Metadata (E) Validation at BH Level (F) Validation carried out at the BH level is the main phase of the inter-country validation. Step F1 Calculate validation tables After the data has been cleaned from extreme errors, the first real validation tables can be calculated. For this purpose either Quaranta or Dikhanov tables can be used 2. Step F2 Check plausibility of BH Price Level Indices (PLIs) Price Level indices (PLIs) provide a comparison of the countries price levels with respect to the regional average. If the PLI is higher than 100, the price level of the country concerned is relatively expensive compared to the regional average and vice versa. PLIs are not intended to rank countries strictly. In fact, they only provide an indication of the order of magnitude of the price level in one country in relation to others, particularly when countries are clustered around a very narrow range of outcomes. During the validation PLIs can be used in two ways to check: The consistency of relative price levels across basic headings for a single country The rank of countries for a single BH If PLIs across basic headings for a single country vary considerably, this may be a sign of problems with the data for the respective outlier BHs. However, at the same time the variation may be justified if, for example, individual economies have particular pricing policies, such as low fuel prices in some of the oilproducing countries. Nevertheless, such a BH should be flagged for further analysis. Regarding the PLIs for a single BH, the ranking of countries tends to follow a certain pattern. If the price level of a country is considerably more or less expensive than it usually is against the countries within the group, this may again be a sign of problems with data for this BH. Step F3 Check BH tables for high or extreme Average Coefficients of Variation Average Coefficient of Variation measures dispersion among all the PPP-Ratios for a basic heading. In doing so, it measures the homogeneity of the price structures of the countries covered by the basic heading and the reliability of the PPPs calculated for the basic heading. The higher the coefficient s value 2 This leaves open the question of whether inter-country validation should commence with the Quaranta Table or with the Dikhanov Table. Some analysts prefer to start with QT and consult the DT after there have been a number of rounds of verification when the PPPs are more reliable. Others prefer to begin with the DT using it to identify countries and items that need investigating and subsequently organize the investigation around the QT. 18

19 the less homogeneous are the price structures and the less reliable are the PPPs. A value over 39 percent is considered to be extreme. During verification of high (low) or extreme values, priority should be given to basic headings with a coefficient value greater than 39 percent, particularly if they have a large expenditure weight. Basic headings with large expenditure weights will have greater influence on the overall PPPs than basic headings with small expenditure weights. Only Quaranta Table show expenditure weights, given that they are available at the time of the validation 3. The value of the coefficient should fall as validation progresses thereby providing a means of assessing the overall effectiveness of the validation process. Step F4 Check BH tables for high or extreme Country variation coefficients Country variation coefficient measures dispersion among a country s PPP-Ratios for a basic heading. In other words, it measures the variation in a country s price levels among the items it priced for the basic heading and the reliability of its PPP for the basic heading. The higher the coefficient s value the less uniform are the country s price levels and the less reliable are its PPPs. A value over 39 percent is extreme. During verification, priority should be given to basic headings for which the value of the country variation coefficient is greater than 39 percent, particularly if the expenditure weight for the basic heading is large. The coefficient should decline in value as the validation progresses. This allows for an assessment of the effectiveness of the validation process. The country variation coefficient complements the item variation coefficient (below) by introducing a different perspective to the same set of data. Focusing on countries rather than items can help to detect countries which have suspicious data 4. Step F5 Check BH and item tables for important products The selection of important items needs to be verified across the countries within a region since the choice is based on estimated expenditure shares for the items within a BH. The official method for the 2011 ICP round Basic Heading PPP computation within a region is weighted CPD, which weighs items to reflect their classification as important or less important. Especially crucial at the inter-country stage is to assess the comparability of the approach followed by the countries regarding the determination of important items. During the validation the following checks have to be conducted: Items selected as important (at item level) Number of important items per BH Percentage share of important items per BH 3 The weights in the QT are only presented they are not used in the calculation of PPPs. 4 Country variation coefficients are especially useful when validation is conducted at aggregated levels. Only Dikhanov tables can be used for aggregates over BH. 19

20 Differences between the PPPs calculated with and without taking importance to account The first check to be done is to analyze items determined as important. The assumption behind an important item is that the products priced for this item are typical for the country s consumption pattern. In most of the cases the typical products tend to have similar relative price as the other typical items within a BH. Thus if an item with a very low or high relative average price is selected as an important item, it should be confirmed that this item is indeed an important one and no mistake is made. This is important since the selection of potential outlier average prices will have an impact on the BH PPP. The second check is to see the number of important items per BH; the minimum rule is that at least one Global Core List item is classified as important and preferably at least one regional item. Without this BH PPPs cannot be calculated using methods that take importance into account. The third check is to calculate the percentage share of important items per BH. In most cases the share of important items should fall to range of 50 to 80 percent. If share is lower or higher than this, the data should be checked and the importance indicators need to be either adjusted or approved. The share could be unusually low or high for certain reasons such as atypical market in relation to the regional product list. In order to achieve plausible results, it is important that the countries within a region follow similar approach with treatment and determination of important items. In order to assess the impact of using the concept of importance, it is recommended that differences between calculation methods that use importance in PPP calculations (EKS* and CPRD) is compared against those methods that do not use it (EKS, CPD). If large differences are found, the selection of important items should be verified and corrections or reasonable justification provided. Step F6 Check Item tables for high or extreme Item variation coefficients Item variation coefficient can be said to be the most important of the variation coefficients for validation purpose. It measures dispersion among the PPP-Ratios for an item. It is an indicator of comparability and accuracy that addresses the questions: have the NCs priced the same products or equivalent products and have they priced them correctly. The higher the coefficient s value, the less uniform are the item s price levels and the more suspicious are the product s comparability and the accuracy of its pricing across countries. Such products are candidates for splitting or deletion and the RC should ensure that they are thoroughly investigated by the NCs. During validation, priority should be given to products with a variation coefficient greater than 39 percent. Step F7 Check Item tables for high (low) or extreme XR-Ratios XR-Ratios are standardized price ratios based on exchange rate converted average prices. The ratios are thus a proxy measure that allows the average prices for a product to be compared across countries. Each of these ratios refers to a particular item in a particular country. A high (low) XR-ratio means that the national average price for the item in question is high (low) compared with the prices of the same item in other countries when they are all converted into the common numeraire currency using the XR. However, it must be remembered that the principal reason for calculating PPPs is the fact that when the prices of a given product are converted into a common currency unit using exchange rates, they are not in fact equal in all countries. The general level of prices tends to be systematically higher or lower in some countries than others. Thus, a high or low XR price for an individual item in one country may be 20

21 largely due to the fact that the general price level for that country is high or low when exchange rates are used. It may not signal any abnormality in that particular price. For this reason analysis of the XR-Ratios facilitates the identification of extreme values among price ratios for an item at the beginning of inter-country validation when PPPs and PPP-Ratios are likely to be unreliable as they based on the average prices that are being validated. Initially, XR-Ratios outside the range of 80 to 125 should be investigated during the first and second round of validation. In later rounds, when the PPP-Ratios become more reliable, high (low) or even extreme values among the XR- Ratios can be ignored. For this indicator, it is better to consult the Quaranta Table because the series is clearly displayed. Box 9 below discussed on the detected outliers. Box 9: Outliers The disturbance created by an outlier can impact not only on the PPP for the country reporting the outlier but also on the PPPs for other countries in the regional comparison. In such cases replacing the outlier with an imputed value or suppressing it are options to be considered. If, within the context of a basic heading, the outlying average price refers to a product that is particularly important for the reporting country deleting it may not be justified, though imputing a value may be. But if the average price refers to a product that is less important or not important removing it is probably warranted. Whatever action is taken, it has to be decided jointly by the country s NC and the RC on a case-by-case basis. Step F8 Check Item tables for high (low) or extreme PPP-Ratios or CPD Residuals PPP-Ratios are standardized price ratios based on PPP converted average prices. These ratios are the correct measure with which to compare the average prices for a product across countries and the average prices of a country across products and it is the main validation indicator at product level. It is thus the extreme values among these price ratios for an item that inter-country validation seeks to identify and verify. CPD residuals in the Dikhanov table are equal to the logarithms of the PPP-Ratios in the Quaranta table. High (low) PPP-Ratios for the same product in different countries imply that the relative price of the item tends to vary considerably from country to country. Although plausible, this may signal that one or more of the prices for the items are incorrect. Ratios outside the range of 80 to 125 should be investigated as well as residuals smaller than or higher than Boxes 10 and 11 below provide an example on edits based on high PPP-Ratios. Box 10: Example validation dialogue for a Whiskey product Q: This item is relatively very expensive as the XR-Ration is 207 and PPP-Ratio is 168 and you are the most expensive country within the region, which is not normally the case. Please check that the correct products have been priced and that the quantity is correctly reported. Please also note that within previous quarter the average price for this item was in-line with the other items within this BH. A: Sorry, we have priced 12 years old whiskey which is not according to the product specifications. The prices are deleted. Box 11: Example validation dialogue for Hairdressing services Q: PPP-Ratio is high (160) and the average price for this dry haircut item is higher than for the next wet haircut item - basically this should not be possible. Please check. Also, if all serviced are offered typically only with wash, you should only price next item. 21 A: Only haircuts with wash are offered in our country. Prices are moved to the next item

22 Step F9 Check Item tables for high or extreme Price observation variation coefficients Price observation variation coefficient measures variation in the price observations on which the average price reported for an item by a country is based. It is taken straight from the average-price-table and it is used to identify extreme values among average prices during intra-country validation when average prices with a variation coefficient over 39 percent were considered extreme. Should the variation coefficient remain over 39 percent after intra-country validation, the NC may need to re-edit the underlying price observations if there are extreme values among the item s PPP-Ratio or if the item variation coefficient is over 39 percent. Box 12 below gives an example of edits done based on the high price observation variation coefficients. Box 12: Example validation dialogue for Loose mushrooms product Q: The variation coefficient for loose mushrooms seems unusually high. Particularly, price observations 5 to 10 should be checked again. Are you sure these prices are for the right product and quantity and that they have been collected in representative outlets? In addition, Product 5 is marked as sold loose. Are you sure your price collector weighed the product and gave the price for 500g? Example Answers: The price collector may have priced a special offer by mistake. We have deleted the price. The price was collected from an unrepresentative small neighborhood shop. We have deleted the price. We have no explanation for the products. Our market specialists consider it as exceptional: it has been deleted as an outlier Validation at aggregated levels Aggregated levels (H) Temporal Analysis (I) Finalization of data (J) Dikhanov tables can be compiled for a group of basic headings constituting an aggregate. Validation at the aggregate level puts the editing and verification of average prices into a broader context. In other words, in order to ensure consistency between the average prices, an analysis within the basic heading as well as within a larger set of products is conducted. Editing at the aggregate level enables inconsistencies to be identified which would not be found by editing solely at the basic heading level. The validation steps carried out at BH level validation and introduced in previous section should also be conducted at aggregated levels. For example, if for the basic heading alcoholic beverages, a country priced all its beverages in quarts instead of liters as specified, its price ratios would be consistent within the basic heading, but they would not be consistent with the country s price ratios in other basic headings. Such errors are identified by editing across basic headings. In this respect, it useful to compile Dikhanov tables at different levels of aggregation. For example, basic headings covering food items could be first checked in a Dikhanov table covering food and non-alcoholic beverages and subsequently in a Dikhanov table covering all household final consumption expenditure. Step H1 Calculate the Dikhanov Tables This should be done for different levels of aggregation. 22

23 Step H2 Conduct steps from F2 to F9 See previous section. Box 13 below gives an example of validation at aggregated level. Box 13: Example of validation at aggregated level Suppose that the PPP for a fruit aggregate for country A are around 40 percent higher than the PPP for a vegetable aggregate while for the other countries the two sets of PPPs are of a similar order of magnitude. This may reflect reality or it may be that the fruits selected for the product list are not representative for county A, but it needs to be verified Temporal Analysis Aggregated levels (H) Temporal Analysis (I) Finalization of data (J) Temporal validation at regional level allows comparisons to be made using PLIs or PPPs, in addition to item level average price comparisons. Comparisons of PLIs or PPPs are more robust over time as they are not comparing directly the average prices for certain items whose availability is not assured or whose item specifications may have changed between the two comparison points. Nevertheless, it should be noted that the basket of items as well as participating countries may have an impact on the results, which is not related to the actual price level changes within the individual countries. A point to make regarding the temporal comparison is that the source of potential errors may be difficult to determine as it can derive either from the previous or current dataset or from the CPI indices. In any case, the idea of temporal comparison is to flag cases for further analysis. Step I1 Compile temporal validation tables for BH and aggregated levels As during the intra-country validation two types of temporal comparisons can be done: Comparison between the ICP rounds Comparison between months or quarters within the ICP round In most cases tables for these comparisons need to be created manually using available BH and aggregate PPPs or PLIs. For the comparison between ICP rounds, CPI-adjustment has to be applied or alternatively the magnitude of PPP price level change can be compared to that of CPI price level change. Step I2 Check tables for large temporal fluctuations in average prices, PLIs or PPPs If differences in item average prices prices, PLIs or PPPs are observed, these should be investigated. If errors are not found, a reasonable justification for the differences should be provided Finalization of data Aggregated levels (H) Temporal Analysis (I) Finalization of data (J) Inter-country validation is an iterative process. It can commence before all countries participating in the regional comparison have supplied their average prices. After each iteration or verification round, the 23

24 RC will change the region s price database in line with the findings reported by the NCs of countries covered in the round, add the prices of countries joining the validation process to the database and produce new Quaranta and Dikhanov tables. These tables will identify new extreme values as a result of the changes introduced by the RC and will need to be investigated by the NCs. Gradually, after a number of rounds of verification with prices of all countries participating in the comparison included in the database, a convergence will occur and the return on further rounds of verification will be deemed marginal by the NCs and the RC and therefore not worth pursuing. Consequently, the inter-country validation can be considered finalized. Step J1 Confirm prices and metadata to be inter-country validated In signing off from the validation process, NCs are accepting responsibility for their average prices. The process is concluded when the NCs formally approve the validated price data. Step J2 Submit prices and metadata to the GO After the data are confirmed to be intra-country validated it can be send to the Global Office. The data and metadata have to be in correct pre-specified format. 5. Global Validation Intra-country validation Inter-country validation Global validation After the data is validated at country and regional level, the final validation stage is the Global validation in which NCs, RCs and the Global Office verify the prices for the Global Core List products, regional PPPs and subsequent global PPPs. The objective is to ensure that prices collected for the GCL items across the regions are comparable and that the linked global PPPs are plausible in wider terms. Global validation stage includes 3 validation phases that are Validation of GCL Items, Validation of regional PPPs and Validation of Global PPPs Validation of GCL Items Validation of GCL Items Validation of Regional PPPs Validation of Global PPPs The validation of GCL products is to be done in several parts; one using the global matrix of all ICP countries and another for core prices validated by region. The purpose here is to identify items that may have passed the data validation steps within the region but become outliers when compared to prices from other regions. Some country prices will be outliers to be returned to the regional coordinators for review and possible deletion. For some basic headings, the entire countries may be outliers again subject to review by the regional coordinators and possible deletion. In these cases, the missing PPP should be imputed using similar basic headings. The importance classification should be reviewed and those with a classification not consistent with the price level should be flagged. This may raise questions whether the classification for the regional comparison was appropriate. 24

25 5.2. Validation of Regional PPPs Validation of GCL Items Validation of Regional PPPs Validation of Global PPPs Once the basic cleaning process has been completed, the within region basic heading PPPs from the regional comparison should be used to deflate the core prices into common regional currencies. The importance classification should be carried forward from the above validation. The validation steps carried out in phase 2 should be repeated. It is at this phase that countries and products will appear as outliers at the basic heading level. The Global Office, RCs and the NCs will have to agree that there will be cases where some will be deleted Validation of Global PPPs Validation of GCL Items Validation of Regional PPPs Validation of Global PPPs The next and final phase of Global Validation starts with the computation of global PPPs and continues with validation of these PPPs using the Dikhanov Table. It is at this phase that a region may need to reexamine within regional prices and basic heading PPPs for some countries. Once basic heading linking factors have been validated, there should be a global aggregation. The PLS for each level of aggregation should be reviewed where there are outliers, the weights and basic heading PPPs should be reviewed. At each level of aggregation, the variation between the direct and indirect PPPs needs to be reviewed, mainly to find outliers that can affect the final estimated PPPs. A final steps comes from the review of the PLS and direct/indirect PPPs. This shows that, with a few exceptions, the results are consistent across countries varying considerably in size and economic structure. 25

26 Annex: Summary of validation stages, phases and steps Intra-country validation Inter-country validation Global validation Inital data validation (A) Statistical tests (B) Finalization of data (C) Step A1 Step A2 Step A3 Step A4 Step A5 Add price and metadata to validation software Check prices and metadata for errors and discrepancies Check that prices are sub-nationally plausible Check that prices are temporally plausible Check the important items Inital data validation (A) Statistical tests (B) Finalization of data (C) Step B1 Step B2 Step B3 Step B4 Check average-price-table for extreme values using Min/Max-Price-ratio test Check average-price-table for extreme values using Variation Coefficient Check price-observation-table for extreme values using deviation test Check price-observation-table for extreme values using T-Value test Inital data validation (A) Statistical tests (B) Finalization of data (C) Step C1 Step C2 Confirm price data and metadata to be intra-country validated Submit price data to Regional Coordinator Intra-country validation Inter-country validation Global validation Inital data validation (D) Validation of Metadata (E) Validation at BH Level (F) Step D1 Add price and metadata to the validation software Step D2 Calculate initial validation tables Step D3 Check initial validation tables for extreme XR-Ratios Inital data validation (D) Validation of Metadata (E) Validation at BH Level (F) Step E1 Check metadata for large deviations between requested and priced products Step E2 Check metadata to assure that survey frames are comparable across the countries Inital data validation (D) Validation of Metadata (E) Validation at BH Level (F) Step F1 Calculate validation tables Step F2 Check plausibility of BH Price Level Indices (PLIs) Step F3 Check BH tables for high or extreme Average Coefficients of Variation Step F4 Check BH tables for high or extreme Country variation coefficients 26

27 Step F5 Step F6 Step F7 Step F8 Step F9 Check BH and item tables for important products Check Item tables for high or extreme Item variation coefficients Check Item tables for high (low) or extreme XR-Ratios Check Item tables for high (low) or extreme PPP-Ratios or CPD Residuals Check Item tables for high or extreme Price observation variation coefficients Aggregated levels (H) Temporal Analysis (I) Finalization of data (J) Step H1 Step H2 Calculate the Dikhanov Tables Conduct steps from F2 to F9 Aggregated levels (H) Temporal Analysis (I) Finalization of data (J) Step I1 Step I2 Compile temporal validation tables for BH and aggregated levels Check tables for large temporal fluctuations in average prices, PLIs or PPPs Aggregated levels (H) Temporal Analysis (I) Finalization of data (J) Step J1 Step J2 Confirm prices and metadata to be inter-country validated Submit prices and metadata to the GO Intra-country validation Inter-country validation Global validation Validation of GCL Items Validation of Regional PPPs Validation of Global PPPs Validation of GCL Items Validation of Regional PPPs Validation of Global PPPs Validation of GCL Items Validation of Regional PPPs Validation of Global PPPs 27