Big data in official statistics Insights about world heritage from the analysis of Wikipedia use

Size: px

Start display at page:

Download "Big data in official statistics Insights about world heritage from the analysis of Wikipedia use"

Elmer Moore
9 years ago
Views:

1 Big data in official statistics Insights about world heritage from the analysis of Wikipedia use Fernando Reis, European Commission - Eurostat International Symposium on the Measurement of Digital Cultural Products Montreal, 9-11 May 2016

2 Defining big data in 1 minute Data deluge Exhaust data + sensors High detail, massive size (large p, large n) Data-driven analytical applications Statistical modelling, machine learning Visualisation Data-driven economy Official statistics does not have a nearly statistical monopoly anymore Eurostat

3 Scheveningen Memorandum on Big Data Examine the potential of Big Data sources for official statistics Official Statistics Big Data strategy as part of wider government strategy Address privacy and data protection Collaboration at European and global level Address need for skills Partnerships between different stakeholders (government, academics, private sector) Developments in Methodology, quality assessment and IT Adopt action plan and roadmap for the European Statistical System Eurostat

4 Big data strategy Start with concrete pilots 3 time-frames Short-term Medium-term Long-term Review the roadmap Eurostat

5 Big Data Action Plan and a glance Governance Policy Quality Skills Experience sharing Legislation IT Infrastructures Methods Ethics / Communication Big data sources Pilots Eurostat

6 Policy Quality Skills Experience sharing Methods Governance Legislation Ethics / Communication IT Infrastructures Big data sources Challenges cooperation, sharing of know-how development of a sound methodology ("from designbased to model-based approach") exploration & tentative implementation Looking for partners Pilots Action (example) Pilot projects, carried out by the Member States (ESSnet) (European Statistical System network) Exploring different big data sources (but also IT architecture, partnerships), developing generic guidelines and frameworks Establish Parternships with data providers and research and international organisations Cooperation with UN on Metodological Framework 6 Eurostat

7 Contracts Eurostat big data pilots Feasibility study on the use of mobile phone data for tourism statistics Internet as a data source for information society statistics Accreditation of big data sources Internal projects Wikipedia use Mobile phone for urban statistics Web evidence for nowcasting Eurostat

8 Wikipedia as a big data source Insights about world heritage from the analysis of Wikipedia use

9 World Heritage Sites Convention Concerning the Protection of the World Cultural and Natural Heritage List of World Heritage Sites maintained by UNESCO

10 Data sources: UNESCO 1. List of World Heritage Sites from UNESCO Public source Official information

11 Data sources: Wikipedia 2. Wikipedia Public source Digital traces left by people Widely used In 2013, 44% of individuals 16 to 74 years old living in EU consulted wikis to obtain knowledge (e.g. Wikipedia) This was 69% for individuals between 16 and 24 years old Community Survey on ICT Usage by Individuals

12 Data sources: Wikipedia 2.1 Content (text and links) Selection of articles related to World Heritage sites 2.2 Page views Wikistats: hourly number of page views for all articles of all wiki projects of the Wikimedia foundation

13 Wikipedia page views raw data Example: zu.z Ulimi 8 AE1,LN2O1Q1,AX1,FB2, [wiki code][article title][monthly total][hourly counts] Monthly files From Jan 2012 to Oct 2015 Total: 935 GB

14 Data processing Sandbox computer cluster 4 nodes, each: 2 x Intel Xeon E v3 10 cores 128GB RAM 4 x 4TB disk FDR Infiniband (56Gbit) 3 stages: Pre-processing Extraction Analytics

15 Pre-processing Scripts in Unix shell and Pig Filtering of raw data to needed project and language Change of format: en.z Banc_d'Arguin_National_Park [0, 0, 0, 5, 6, 16, 5, 20, 25, 21, 48, 29, 43, 40, 46, 0, 30, 55, 36, 39, 28, 28, 204, 218] Processing time: 8 Hours

16 Extraction Map-reduce jobs Scripts in Unix shell and python Filtering to list of articles supplied Time aggregation from hourly to daily, weekly and monthly Processing time: 2 hours

17 Data analysis R and RStudio Querying APIs (CatScan, Wikipedia Miner, Wikimedia) Web scrapping of Wikipedia for selection of articles (geocoordinates, categorisation, information boxes, article redirects, articles links) Statistics, maps, graphics

18 Number of page views of related Wikipedia articles per country of location of the WHS Reference: Jan.2012 Oct languages

19 Average number of page views according to the date of inscription Reference: Jan.2012 Oct languages

20 Top 20 World Heritage Sites in number of page views of related Wikipedia articles Reference: Jan.2012 Oct languages

21 Distribution of page views of articles related to World Heritage Sites by language of Wikipedia en de fr ru it pt nl tr es pl Reference: Jan.2012 Oct languages

22 Top 5 WHS in number of page views of related Wikipedia articles by language English Spanish German French Reference: Jan.2012 Oct languages

23 45M Page views of Wikipedia articles related to World Heritage Sites 40M 35M 30M 25M 20M 15M 10M 5M 0M Reference: Jan.2012 Oct languages Mar 2012 Jul 2012 Nov 2012 Mar 2013 Jul 2013 Nov 2013 Mar 2014 Jul 2014 Nov 2014 Mar 2015 Jul 2015

24 Page views of Wikipedia articles related to World Heritage Sites (English Wikipedia) Vatican City What happened in March 2013?!

25 Distribution of WHS by number of page views (log)

26 Distribution of WHS by number of page views (NOT log) The percentage of page views going to the top 20 WHS is 32%

27 Thank you for your attention Fernando Reis Eurostat Task Force on Big Data Eurostat

big data in the European Statistical System

Conference by STATEC and EUROSTAT Savoir pour agir: la statistique publique au service des citoyens big data in the European Statistical System Michail SKALIOTIS EUROSTAT, Head of Task Force 'Big Data'