International Comparison Program IDEAS

Transcription

1 International Comparison Program Guidelines for archiving data in the ICP context ICP Data Electronic Archiving System IDEAS Draft version Operational Guide

2 Contents Introduction Little questions - Definition and concepts Data Set Formats Catalogs and lists of databases Metadata and thematic documentation Software solutions and data archiving systems Next steps

3 General guidelines for archiving data in the ICP context 1 Presenting the ICP Data Electronic Archiving System IDEAS Introduction The International Comparison Program (ICP) is a worldwide statistical partnership to collect comparative price data. It compiles detailed expenditure values of countries gross domestic products (GDP), estimates purchasing power parities (PPPs) of the world s economies. Using PPPs instead of market exchange rates to convert currencies makes it possible to compare the output of economies and the welfare of their inhabitants in real terms (that is, controlling for differences in price levels). ICP collects individual price quotations on household consumption items such as food, clothing, private health, pharmaceutical products, and private education. It also survey prices for Gross Fix Capital Formation that is Machinery and Equipment Goods and Construction as well as wages of Government employees, housing and rents. ICP is an important source for benchmark price information on major categories of goods and services consumed by households, governments, private and non-governmental organizations. On the expenditure side, the International Comparison Program is the only statistical initiative providing periodic information of Gross Domestic Product and its major components. During its 2011 round, the Program has put emphasis on documenting its methodology, concepts and definitions, as well as finding from data and relevant analyses. It intends to generate a wealth of information in forms of data, indicators, metadata, and flagship describing close to two hundred economies covering Africa, Asia-Pacific, the Community of Independent States (CIS), Countries of Eurostat and OECD, Latin America and the Caribbean Islands, and Western Asia. A quick assessment of data availability for 2005 round of International Comparison Program concluded that average prices and expenditures are available for all regions at the Global Office. The same applies to population and exchange rates and published ICP tables. At regional level though the situation varies. Some regions have no detailed data available; some have partially validated prices and National Accounts data while others indicated that the data is available with limited access. No systematic assessment was done at country level but the situation may not differ from the one of the regions. Some countries may have raw data available and accessible, some may not. Regions are invited to locate any potential source of 2005 data for proper safeguard and archiving. 1 This chapter is prepared by Michel Mouyelo-Katoula and Biokou Mathieu Djayeola

4 This guideline is provided by the ICP Global Office in response to request from ICP Regional Coordinators to set guidelines for proper archiving of survey data collected during the 2011 round of the International Comparison Program. It includes recommendations about: Definition and concepts Data set formats for archiving Listing of database and tables to allow cataloging, search, retrieval, and reuse Meta data and documentation. Preservation of specific software 1. Little questions - Definition and concepts For the purpose of this document the following terms and concepts definitions apply: What is IDEAS: The ICP Data Electronic Archiving System is an assembly of electronic components with interdependent functions, forming a self-contained unit. The System is composed of data, metadata and word processing documents on methodology, operational materials and logistical implementation of the ICP Software used in the field are part of the IDEAS as they are used to translate ICP methodologies into publishable indicators. What is ICP Data? Any individual fact, statistics, or item of information collected in the context of the 2011 round of the International Comparison Program. These data are price quotations, expenditures derived from GDP breakdown, exchange rates, population figures, etc. The concept of metadata is relevant to this guideline. ICP metadata will be any data that describes other ICP data. This provides information about a certain item's content. For example, a picture may include metadata that describes how large the picture is, the color depth, the image resolution, when the image was created, what item it relates to, etc. A text document's metadata may contain information about how long the document is, who the author is, when the document was written, and a short summary of the document. An important set of ICP metadata is the Global Core List, the ICP Classification and any information qualifying price quotations and expenditure values collected in the context of the International Comparison Program. An ICP file refers to a collection of related data or program records stored as a unit with a single name. Such documents are databases of price for a given country during a specified period of time, a methodology document, or a software program developed for ICP purposes. Is archiving just about backups? There are both conceptual and operational differences between backup and archiving. Data backups are designed for fast recovery of operational data. Data archiving stores data that is no longer in day-to-day use but must still be retained. Data backups are intended to provide a quick means of recovering data that is in current or recent use in cases ranging from data corruption or accidental deletion to full disaster recovery scenarios. Data archiving is intended as a repository for data that needs to be stored for periods that may extend to decades. Speed of restoration from a data archive usually is not as critical as - 4 -

5 from a data backup, but search-ability is of vital importance. ICP users will be performing regular backups of their data while they are in use but data archiving will be necessary at the end of the 2011 Round once the results are published. 2. Data Set Formats The guiding principles for collection of price data in the context of the International Comparison Program put emphasis on designing appropriate frameworks and data models in order to compute purchasing power parities. Expenditures data are availed in forms of breakdown of GDP in twelve (12) categories and one hundred and fifty (150) components known as basic headings. To manage its survey data, the Program developed tailored applications called ICP Kit for data collection, validation, diagnostics and processing. This multi platform software requires basic infrastructures such as Microsoft Windows, Microsoft.net Framework, Microsoft Sample Query Language (SQL), and Microsoft Office (mainly MS Excel) to run. Few regions have specific software developed under Java, php, or Oracle environments but all data exchange is though Microsoft Excel. Therefore these guidelines will focus on archiving ICP data using Excel databases. At each level of data management, survey data are structured in order to address specific needs. At country level individual quotations are processed in an SQL Lite environment before they are validated and summarized for validation at regional level. The main concept at this level is national annual average price. But to obtain this variable individual price quotations have been combined by location, time period (month, quarter, semester, etc.), and relevant categories of goods and services. At sub-national level data are disaggregated by location type, outlet type, and price type. All these data set are subject to archiving in the original, intermediate and final (validated) versions. It is worth noticing that ICP Price data goes through several back-and-forth rounds of validation in order to minimize if not eliminate human and conceptual errors. 3. Catalogs and lists of databases The ICP Data Electronic Archiving System (IDEAS), like other data retrieval systems, uses a combination standalone and online database tables to search for requested data sets. In many cases, ICP prices and expenditures metadata and data are stored in both catalogs or database tables and regular office documents management systems. To simplify adding new data sets into the system, it is important perform few precaution: 1. Provide a target list or catalog of observations 2. Create a comprehensive lexicon/glossary such that users can easily access the catalog 3. Provide necessary headers, keywords, and classification for metadata management - 5 -

6 Catalogs should contain information on fields and structures to help users identify and locate the desired observation(s). In terms of typological organization IDEAS should provide space coordinates (geo-references), observation date, time, and mnemonic identifiers fairly essential to navigate the system. Tailored parameters are critical for analyzing or interpreting archived data. It is proposed that IDEAS is developed in Microsoft compatible databases preferably SQL but depending on availability this system could run under Sybase, Oracle, Access Database Management systems or other specialized environments such as IHSN Tool-Kit, CSPro, SAS, or StatA. At national, regional and global levels, it is recommended to adopt one of these platforms for efficiency purposes. Fair combination of two or three systems are possible but the less the heterogeneity the better. Tables can be exchanged between most database systems by copying using Excel Comma Separated Variable (.csv) or to American Standard Code for Information Interchange (.txt) also known as ASCII table files. As a matter of precaution definition of database structures must have a sufficient number of significant figures for representing decimal floating point values. This will prevent data losses and mismanagement. When possible a web archiving platform is ideal for easy access and larger dissemination. It might seem challenging but web archiving is one of the easiest, affordable, and efficient technologies available in the industry. The key for successful web archiving system is security and integrity. A proper data access policy is required to monitor usage and data request in order to measure the impact of the web based data archiving systems. The glossary will provide general description of the individual fields within each table or list. To the extent possible this glossary should also provide information source of entries. For instance, users should easily recognize whether coordinates were supplied during data collection or geocoded for archiving purposes. 4. Metadata and thematic documentation For proper understanding of the information in ICP data set by IDEAS archive users, the system should provide detailed documentation in the following categories: 1. Program Description - General descriptions of its mission and implementation mechanisms, 2. Data Collection Survey framework, guidelines, manuals, questionnaires, logistics, etc. 3. Data Validation - 4. Data Processing - how was the data processed, aggregated, reduced or summarized, 5. Data Description - Documentation on data characteristics (e.g. price type, observed quantity, expenditure values, averaging methods, etc.), anomalies and method of correction (e.g. persisting outliers, flagged data for investigation, validated extreme values, etc.), units of measure, estimates, and imprecision (e.g. subsidies, weighted items, indigenous and non-standardized units, etc.), and database field descriptions, 6. Data Format - A general description of the contents of the archived IDEAS data sets including, for example, documentation on the glossary entries. It is important to emphasize that generally the content of comment field alone is insufficient to properly - 6 -

7 define keywords. A certain form of codification is necessary to facilitate data retrieval. Without additional documentation, many of the comments provided by data collectors will barely serve their purpose. For online versions of IDEAS all data must be in wed safe format (.xls,.xlsx,.txt,.xml,.htm,.html, etc) for easy access. To the extent possible document packages such as extractable files, compressed folders, and executable binders must be used with caution. It is demonstrated that these file types are easy targets to viruses, Trojans and malwares. All text processing documents originally generated in Microsoft Word or other editors (.txt,.doc,.docx,.rft,.wps, etc) must be converted to.pdf or html for web dissemination. Usually these are large documents such as user manuals or data analysis guides provided for downloading or pop-up opening. These features also apply to stand alone IDEAS in order to prevent data loss and prepare for future online dissemination. 5. Software solutions and data archiving systems The International Comparison Program has developed specific software solutions for data collection on prices and National Accounts. Its survey data can be archived using common micro data management systems such as the IHSN Toolkit, dedicated statistical software such as CS Pro, and web based archiving systems. Example of these solutions are provided below. ICP-Kit Modules: During the 2011 round of the International Comparison Program the Global Office has coordinated the design of the ICP Kit software with two complementary purposes: the ICP Regional Module has a setup and diagnostics and an index processing components whereas the ICP Country Modules are organized around the main ICP survey categories. These are Household consumption items, Compensation of employees, Machinery and equipment goods, Construction, Health, Education, Housing, and National Accounts. Each area has data entry and data validation functions. The Regional Module, the Household Consumption Module and the Compensation Module have been developed under Microsoft.net Framework while other modules are Excel based files with embedded macro and validation commands. IHSN Tool Kit: The Microdata Management Toolkit developed by the World Bank Data Group for the International Household Survey Network aims to promote the adoption of international standards and best practices for microdata documentation, dissemination and preservation. The Toolkit comprises three modules. The Metadata Editor is used to document data in accordance with international metadata standards (DDI and Dublin Core). The Explorer is a free reader for files generated by the Metadata Editor. It allows users to view the metadata and to export the data into various common formats (Stata, SPSS, etc). The Metadata Editor and Explorer are based on Nesstar technology and developed by the Norwegian Social Science Data Services (NSD). The CD-ROM Builder is used to generate user-friendly outputs (CD-ROM, website) for dissemination and archiving 2. 2 Source: The International Household Survey Network

8 CS Pro: CSPro (Census and Survey Processing System) is a public-domain software package for entering, editing, tabulating and mapping census and survey data. CSPro was designed and implemented through a joint effort among the developers of IMPS and ISSA: the United States Census Bureau, Macro International, and Serpro, S.A. Funding for the development is provided by the Office of Population of the United States Agency for International Development. CSPro is designed to eventually replace both IMPS and ISSA. CSPro combines and expands upon the capabilities of both ISSA and IMPS. It takes advantage of the power and flexibility of both of these programs, but adds the friendliness, ease of use, and intuitive nature of Windows. CSPro provides a more visual approach to the creation and manipulation of data and reduces the programming needs. This easy to use package facilitates defining data structures, developing applications, entering and checking data, generating reports, and producing tabulations. More advanced users, including computer programmers, can access the full CSPro language to perform complicated tasks. CSPro consists of modules for Data Entry, Batch Editing and Processing, and Cross Tabulation, as well as Designer modules for creating Data Dictionaries and Data Entry Forms, and for writing applications. It also includes a set of tools, including Text Viewer, Table Viewer, Map Viewer, Table Retrieval, Tabulate Frequencies, Sort Data, Export Data, Reformat Data, Compare Data, Convert Dictionary, and Convert Shape to Map 3. Best practices 1: The Archived news releases on US Consumer Price Index is a good example of web based data archiving systems. This basic web based archiving system presents official releases of US CPI by the US Bureau of Labor Statistics. Due to institutional Data Access policy, only selected indicators are included in the news releases. To include more detailed data and to preserve micro data on web based platforms, it is necessary to develop an access control system. Archiving is ensured through regular institutional security backup and web hosting solutions 4. Best practices 2: As part of its effort to accelerate dissemination of development data, the World Bank has launched in April 2010 the Open Data Initiative. This offers broader access to World Bank statistics to allow policymakers and advocacy groups to make better-informed decisions and measure improvements more accurately. The Open Data Initiative also provides valuable tools to support research by journalists, academia and others, broadening understanding of global issues. Archiving is ensured through regular institutional security backup and web hosting solutions 5. 3 Source: The United Nations Statistics Division Reference: 5 Reference:

9 6. Next steps For 2011 we expect a different picture. The development of the ICP Portal, meant to be the ICP data repository and micro data management system, offers a unique opportunity for versioning, referencing, archiving and retrieval of micro data, aggregated indicators and published results from ICP surveys. These features constitute the base for the ICP Data Electronic Archiving System (IDEAS) which will allow each country to store and retrieve data using a well designed system allowing for suitable level of confidentiality. The main principle for the proposed system is to strictly stick to the ICP 2011 Data Access Policy as approved by the ICP Executive Board. Access will be granted to users of the ICP Portal as per their respective access rights. How should all stakeholders prepare data for the ICP Data Electronic Archiving System? For the system to run effectively, the Global Office believes that the cooperation of regions and countries is vital. First regions will be invited to comply with the data access policy and submit all necessary data and metadata for prices, national accounts and supporting indicators to feed the system. These include but are not limited to: purchasing power parities, average prices, micro data for all surveys, GDP, basic heading expenditures, population, exchange rates, etc... But most importantly countries are encouraged to allow their region to share these data with the Global Office. Mainly three types of datasets are accepted by the system: 1. Validated price data will be transmitted to the regions by participating ICP countries to the regions. These are two-fold datasets: average prices and individual price quotations. 2. National Accounts data and metadata to be stored in the system at regional and global levels. 3. Other macroeconomic data to be archived include Gross Domestic Product, Population, Exchange Rates, etc. Archiving is also planned for all methodological and operational documents published during the 2011 round of the International Comparison Program. This includes: operational guidelines, forms, structural information, lexicons for language support, software user guides, data dictionaries, translated documents where available, etc. We expect the ICP Data Electronic Archiving System to be an exhaustive and accessible system running in full compliance with the approved data access policy