Big CBS. Experiences at Statistics Netherlands. Dr. Piet J.H. Daas Methodologist, Big Data research coördinator. Statistics Netherlands

Transcription

1 Big CBS Experiences at Statistics Netherlands Dr. Piet J.H. Daas Methodologist, Big Data research coördinator Statistics Netherlands April 20, Enschede

2 Overview Big Data Research theme at Statistics Netherlands Examples of our work Lessons learned Important topics in

3 Data, data everywhere! X

4 Two types of data CBS Primairy data Secondary data 4 Our own surveys Data from others - Administrative sources - Big Data

5 Big Data research at Stat. Netherlands Exploratory, data driven Cases: Road sensor data, Mobile phone data, Social media There is no standard Big Data methodology (yet) Combination of IT, methodology and content (Data Science) Important issues for official statistics Getting access Selectivity (representativity) Data editing at large scale Data reduction (dimension reduction) 5

6 Results of case studies

7 Example 1: Roads sensors Road sensor (traffic loop) data Each minute (24/7) the number of passing vehicles is counted in around highway loops in the Netherlands Total and in different length classes 7 Nice data source for transport and traffic statistics (and more) A lot of data, around 230 million records a day Our current dataset has 56 billion records! Locations

8 Dutch highways 8

9 Dutch highways with road sensors 9

10 Road sensor data All vehicles All vehicles Large vehicles (> 12m) All vehicles 10

11 Minute data of 1 sensor for 196 days 11

12 Afsluitdijk (IJsselmeer dam) 12

13 Afsluitdijk (IJsselmeer dam) (2) Cross correlation between sensor pairs -Used to validate metadata Trajectory speed vs. point speed - Average speed is 98 Km/h

14 Overall process Transform & Select Frame Clean Aggregate 14

15 Reducing Big Data 15

16 Traffic index (prototype) NUTS 3 area based traffic indexes for the main roads per month / week

17 Example 2: Mobile phones 17 Mobile phone activity as a data source Nearly every person in the Netherlands has a mobile phone Usually on them and almost always switched on! Many people are very active during the day Can data of mobile phones be used for statistics? Travel behaviour (of active phones) Day time population (of active phones) Tourism (new phones that register to network) Data of a single mobile company was used Hourly aggregates per area (only when > 15 events) Especially important for roaming data (foreign visitors)

18 Travel behaviour (of mobile phones) Mobility of very active active mobile phone users during a 14 day period data of a single mob. comp. Based on: Mobile phone activity Location based on phone masts 18 Observed: Includes major cities But the North and South east of the country are much less represented

19 Day time population Hourly changes of mobile phone activity 7 & 8 May 2013 Per area distinguished Only data for areas with > 15 events per hour 19

20 Tourism: Roaming during European league final Hardly any Low Medium High Very high 20

21 Example 3: Social media Dutch are very active on social media! Around 60% according to a surveyna altijd bij zich en staat vrijwel altijd aan Steeds meer mensen hebben een smartphone! Mogelijke informatiebron voor: Welke onderwerpen zijn actueel: Aantal berichten en sentiment hierover Als meetinstrument te gebruiken voor:. 21 Map by Eric Fischer (via Fast Company)

22 Dutch Twitter topics (3%) (7%) (3%) (10%) (7%) (3%) (5%) (46%) 12 million messages 38

23 Sentiment in Dutch social media About the data Dutch firm that continuously collects ALL public social media messages written in Dutch Dataset of more than 3.5 billion messages! Covering June 2010 till the present Between 3 4 million new messages are added per day About sentiment determination Bag of words approach List of Dutch words with their associated sentiment Added social media specific words ( FAIL, LOL, OMG etc.) Use overall score to determine sentiment Is either positive, negative or neutral Average sentiment per period (day / week / month) 23 (#positive #negative)/#total * 100%

24 Daily, weekly, monthly sentiment 24

25 Sentiment per platform (~10%) (~80%)

26 Platform specific results Table 1. Social media messages properties for various platforms and their correlation with consumer confidence Correlation coefficient of Social media platform Number of social Number of messages as monthly sentiment index and media messages 1 percentage of total (%) consumer confidence ( r ) 2 All platforms combined 3,153,002, Facebook 334,854, * 0.85* Twitter 2,526,481, Hyves 45,182, News sites 56,027, Blogs 48,600, Google+ 644, Linkedin 565, Youtube 5,661, Forums 134,98, period covered June 2010 untill November confirmed by visual inspecting scatterplots and additional checks (see text) *cointegrated 26

27 Schematic overview Previous Vorige maand month Current Maand month Day Dag1-7 Dag Day 8-14 Dag Day Dag Day Consumer Confidence 27 Publication date (~20 th ) Sentiment Social media sentiment

28 Results of comparing various periods Consumer Confidence Facebook Facebook Facebook + Twitter * Twitter 0.81* 0.84* 0.86* 0.85* 0.87* 0.89* * 0.85* 0.89* 0.79* 0.82* 0.84* * 0.86* 0.89* 0.79* 0.83* 0.87* 0.75* 0.80* 0.81* 28 *cointegration LOOCV results

29 Overall findings Correlation and cointegration 1 st week of Consumer confidence usually has 70% response Best correlation and cointegration with 2 nd week of the month Highest correlation 0.93* (all Facebook * specific word filtered Twitter) Granger causality Changes in Consumer confidence precede changes in Social media sentiment For all combinations shown! Only tried linear models so far Prediction Slightly better than random chance Best for the 4 th week of month 29

30 Sentiments indicator for NL (beta-version) 30 Displays average sentiment of Dutch public Facebook and Twitter messages

31 Lessons learned (so far) The most important ones: 1) There are many types of Big Data 2) Know how to access and analyse large amounts of data 3) Find ways to deal with noisy and unstructured data 4) Mind set for Big Data Mind set for survey data 5) Need to beyond correlation 6) Need people with the right skills, knowledge and mind set 7) Solve privacy and security issues 8) Data management & costs 31 We are getting more and more grip on these topics

32 Important research topics for ) Profiling Extracting features from units in Big Data sources Use this to study i) population changes and ii) input for ways to correct for selectivity For persons: gender, age, income, education, origin, urbanicity, and household composition For companies: number of employees (size class), turnover, type of economic activity, legal form 2) Data editing at large scale Combination of methods/algorithms and IT 3) Data reduction (dimension reduction) Reduce size of Big Data while retaining similar information content (not sampling!)

33 The Future The future of statistics looks BIG 33

34 @pietdaas Thank you for your attention!