Big Data in Price Statistics -

Transcription

1 Ingolf Boettcher Vienna 22. October 2015 Big Data in Price Statistics - Scanner Data and Web-scraping Wir bewegen Informationen

2 What is scanner data (in price statistics)? -Sales information on article level (turn-over + quantity ) -generated by point of sales terminals in shops Folie

3 What is scanner data (in price statistics)? Not a sample of transactions. LIMO- NADE OTHER JUICE COLAS Non-alcoholic beverages = 100% Folie

4 What is scanner data (in price statistics)? a census of all transactions. LIMO- NADE OTHER COLAS JUICE Non-alcoholic beverages = 100% Folie

5 What is scanner data (in price statistics)? SHOP ID PERIOD PRODUCT ID /EAN PRODUCT Group PRODUCT Description _ Juices BrandXY Orange, mild, 500 ML, bottle. TURN OVER 220 EUR _ Juices Price(unit value) p i =2EUR = Turn over i Sales i = SALES (Quantity) Folie

6 What is scanner data (in price statistics)? A lot of data Estimation (rule of thumb): # of Households Austria Average # of Retail Transactions per month per household Total # of Retail Transactions per month About Folie

7 What is scanner data (in price statistics)? A lot of data Actual Test - Scanner Data for segment personal care provided by an Austrian drugstore chain (about 350 shops): # of sales information on article level per week per shop About Total # of sales information per month About Share of product segment personal care in CPI basket of goods 1,4% Folie

8 What is Web-scraped data? (in price statistics) Website content turned into a spreadsheet Folie

9 What is Web-scraped data? (in price statistics) SHOP URL PRODUCT ID PRODUCT Group PRODUCT Description Juices BrandXY Orange, mild, 500 ML, bottle Juices PRICE 2 EUR Folie

10 Why scanner and web-scraped data for price statistics? The Consumer Price Index ( Inflation) Representative measure of consumer prices (about 780 product groups, e.g. milk, Petrol ): Regional price collection in 20 cities + Central price collection by Statistics Austria staff = about prices Folie

11 Why scanner and web-scraped data for price statistics? Broader and more segmented sortiments Past / Today Today / Future Whole Milk Skimmed Milk Organic Whole Milk Non-Organic Whole Milk Skimmed Whole Milk SkimmedLactose-Free Skimmed Lactose-Free Fat-FreeLactose-Free Fat-Free Fat-Free Folie

12 Why scanner and web-scraped data for price statistics? More dynamic (promotional) pricing Past / Today Today / Future List Price Promotion Price List Price Promotion Price Member Price Individual Price for Members Time specific prices (e.g. flights, hotels) Location specific prices Hardware specific prices Folie

13 Advantages of using scanner and webscraped data for price statistics Web-scraped data Better coverage (time, markets, products) Scanner data Better coverage (time, space, markets, products) Transaction prices Information on current quantities Folie

14 Advantages of using scanner data Information on current quantities allow new types of price indices: P i,laspeyres = (Price i,t1 Quantity i,t0 ) (Price i,t0 Quantity i,t0 ) International standard for Consumer Price Indices Takes into account availability of data sources Tendency to overstate inflation Folie

15 Advantages of using scanner data Information on current quantities allow new types of price indices: P i,paasc he = (Price i,t1 Quantity i,t1 ) (Price i,t0 Quantity i,t1 ) Not available before scanner data Tendency to understate inflation P i,fisher = P i,laspeyres P i,paasc he Folie

16 Challenges scanner data SHOP ID Mapping of Articles to CPI Classification PERIOD PRODUC T ID TEXT MINING PRODUCT Group PRODUCT Description _ Juices BrandXY Orange, CPI EA Orange Juice, ML mild, ML, = bottle. CPI Division Group Class Sub-Class ALL FOOD& BEVERA. BEVERA GES JUICE ORANGE JUICE APPLE JUICE. TURN OVER 220 EUR 110 SALES (Quantity) 1234 (Retailer code for JUICE) AND (>=1000 AND <=2000) AND ( Orange OR BRANDX OR BRAND Y OR [FLAVOR_NAME] OR BRAND_B (.)? AND NOT ( APPLE OR Mango OR Grapefruit ( ) Folie

17 Challenges scanner data Common methodological framework for use of scanner data work in progress? Temporal Aggregation (days vs. week) Spatial Aggregation (shop vs. region)? Article Matching?? Article Mapping to COICOP Data Cleaning (outliers, etc.) Type of price index Folie

18 Challenges scanner data - Product re-launches: New ID (EAN/GTIN) for same product - Example and picture: Statistics Netherlands (Antonio Chessa) Folie

19 Web-scraping (in Price Statistics) Statistics Austria: click-and-point web-scraping software Folie

20 Challenges Web-scraping - Web-sites change frequently High workload for Web-scraper maintenance - Identification of articles with high turn-over - Identification of re-launches - Legality to crawl on websites Folie

21 Challenges Web-scraping web-sites change frequently Decision to use click-and-point web-scraping software (import.io) No IT-developer needed, therefore: Cheap Flexible No programming skill required Folie

22 Challenges Web-scraping Identification of articles with high turn-over??? Folie

23 Challenges Web-scraping Legality to crawl on websites Technical hurdles of websites may not be circumvented (robot.txt) The database may not be replicated as a whole elsewhere through web-scraping Web-scraping may not negatively affect a web site s performance Folie

24 Potential Scanner data and Web-scraping Austrian CPI basket of good Folie

25 Potential of Scanner data and Web-scraping Already use scanner data for HICP/CPI compilation Testing-phase using continuous realtime scanner data transmissions Testing-phase using historical scanner data Netherlands France Austria Germany Sweden Luxembourg Poland Italy Switzerland Belgium Great Britain Norway Portugal Spain Denmark (from Jan 2016) Slovenia Slovakia Hungary Testing-phase using purchased historical scanner data from market research institutes Folie

26 Ingolf Boettcher Vienna 22. October 2015 Big Data in Price Statistics - Scanner Data and Web-scraping Wir bewegen Informationen