Unlocking the Full Potential of Big Data Lilli Japec, Frauke Kreuter JOS anniversary June 2015 facebook.com/statisticssweden @SCB_nyheter
The report is available at https://www.aapor.org
Task Force Members: Lilli Japec, Co-Chair, Statistics Sweden Frauke Kreuter, Co-Chair, JPSM at the U. of Maryland, U. of Mannheim & IAB Marcus Berg, Stockholm University Paul Biemer, RTI International Paul Decker, Mathematica Policy Research Cliff Lampe, School of Information at the University of Michigan Julia Lane, American Institutes for Research Cathy O Neil, Johnson Research Labs Abe Usher, HumanGeo Group
AAPOR (American Association for Public Opinion Research) a professional organization dedicated to advancing the study of public opinion, broadly defined, to include attitudes, norms, values, and behaviors promotes best practices and transparency works to educate its members as well as policy makers, the media, and the public at large to help them make better use of surveys and survey findings, and to inform them about new developments in the field other task force reports available on https://www.aapor.org
Outline of our presentations What is Big Data? Paradigm shift Big Data activities in different organizations Skills required Big Data process and data quality
three main data sources UNTIL RECENTLY
Survey Data Administrative Data Experiments
NOW
US Aggregated Inflation Series, Monthly Rate, PriceStats Index vs. Official CPI. Accessed January 18, 2015 from the PriceStats website.
Number of vehicles detected in the Netherlands on December 1, 2011 created by Statistics Netherlands (Daas et al. 2013). The vehicle size is shown in different colors; black is small size, red is medium size and green is large size.
Social media sentiment (daily, weekly and monthly) in the Netherlands, June 2010 - November 2013. The development of consumer confidence for the same period is shown in the insert (Daas and Puts 2014).
Big Data http://www.rosebt.com/blog/data-veracity
Hope that found/organic data Can replace or augment expensive data collections More (= better) data for decision making Information available in (nearly) real time
New paradigm New business model Federal agencies no longer major players New analytical model Outliers Finegrained analysis New units of analysis New sets of skills Computer scientists Citizen scientists Different cost structure Source: Julia Lane
Eurostat Big Data Action Plan and Roadmap Pilots exploring the potential of selected big data sources The project will also include activities on: Methodological frameworks, Quality frameworks, Metadata frameworks, IT infrastructures, Communication, Legal frameworks, Ethical frameworks, Skills and training, and Experience sharing.
UNECE and Big Data The Sandbox provides a computing environment to load Big Data sets and tools Consumer price indices experimenting with the computation of price indexes Mobile telephone data statistics on tourism and daily commuting Smart meters statistics on power consumption using data collected from smart meter readings. Traffic loops traffic statistics using data from traffic loops Social media using Twitter data to analyze sentiment and to tourism flows. Job portals computing statistics on job vacancies Web scraping tested methods for automatically collecting data from web sources.
UNECE Big Data Inventory
Statistics Netherlands: Roadmap BIG DATA Two focus projects: the use of traffic loop data for transportation statistics the use of mobile phone data for daytime population and tourism statistics. Six other projects: the use of internet data for price statistics, investigating the use of bank and credit card transactions, the use of social media data for detecting trends in social cohesion, the use of internet data for encoding enterprise purchases and sales, investigating the use of smartcards of public transport for statistics, and the use of internet data for statistics about job vacancies. Source: Pieter Vlag, Statistics Netherlands 18
Examples from Statistics Sweden Scanner data to improve the Household Budget Survey Job vacancy statistics by scraping of the web To evalutate the use of AIS (Automatic Identification System) data. Cooperation between Statistics Sweden and the agency for Transport Analysis (Trafa). Research funding from the Swedish Innovation Agency (Vinnova).
Source: Moström and Justesen, Statistics Sweden One day data
What tasks are required to get there? SKILLS
We have to do this jointly Data Output/Access Example: map visualization / privacy Data Analysis Example: Hadoop MapReduce; High Frequency Data Data Curation/Storage Data Generating Process Research Questions Example: Hadoop Distributed File System Examples: geolocated social media + survey + administrative data Examples: Behavior of interest (migration/political participation/job searches)
Source: Abe Usher
Big words What is big data? What is Hadoop File System? (HDFS) What is Hadoop MapReduce? (MR) How do you link surveys with big data? Source: Abe Usher
Computer scientist Data preparation MapReduce algorithms Python/R programming Hadoop ecosystem System Administrator Storage systems (MySQL, Hbase, Spark) Cloud computing: Amazon Web Services (AWS) Google Compute Engine Hadoop ecosystem Source: Abe Usher
What do we know about the data generating process? RESEARCH
Veracity Who? What? Why? Who is missing? Who is counted repeatedly? What is not said / measured?..and why?
But (at least) one more V http://www.rosebt.com/blog/data-veracity
Terrorist Detector Terrorist Detector Errors in Big Data: An Illustration Suppose 1 in 1,000,000 people are terrorists The Big Data Terrorist Detector is 99.9 accurate The detector says your friend, Jack is a terrorist. What are the odds that Jack is really a terrorist? Source: Paul Biemer 29
Terrorist Detector Terrorist Detector Errors in Big Data: An Illustration Suppose 1 in 1,000,000 people are terrorists The Big Data Terrorist Detector is 99.9 accurate The detector says your friend, Jack is a terrorist. What are the odds that Jack is really a terrorist? Answer: 1 in 1000 i.e., 99.9% of the terrorist detections will be false! Source: Paul Biemer 30
Big Data Process Map Generate Source 1 ETL Extract Analyze Filter/Reduction (Sampling) Source 2 Source K Transform (Cleanse) Load (Store) Computation/ Analysis (Visualization) Source: Paul Biemer 31
Big Data Process Map Generation Source 1 Source 2 Source K ETL Errors include: Extract low signal/noise ratio; lost signals; failure to capture; non-random (or nonrepresentative) sources; metadata that are lacking, absent, or erroneous. Transform (Cleanse) Load (Store) Analyze Filter/Reduction (Sampling) Computation/ Analysis (Visualization) Source: Paul Biemer 32
Big Data Process Map Generation Source 1 Source 2 Source K ETL Extract Transform (Cleanse) Load (Store) Analyze Errors include: specification error (including, errors in meta-data), matching error, Filter/Reduction coding error, editing error, data (Sampling) munging errors, and data integration errors.. Computation/ Analysis (Visualization) Source: Paul Biemer 33
Generation Source 1 Big Data Process Map Data are filtered, sampled or otherwise Errors reduced. include: ETL This sampling may errors, involve selectivity further errors (or lack transformations of representativity), Extract of the modeling data. errors Analyze Filter/Reduction (Sampling) Source 2 Source K Transform (Cleanse) Load (Store) Computation/ Analysis (Visualization) Source: Paul Biemer 34
Big Data Process Map Generation Source 1 ETL Extract Analyze Filter/Reduction (Sampling) Source 2 Source K Errors include: Transform modeling errors, inadequate or (Cleanse) erroneous adjustments for representativity, computation and algorithmic errors. Load (Store) Computation/ Analysis (Visualization) Source: Paul Biemer 35
POTENTIAL
We have to do this jointly Data Output/Access Data Analysis Data Curation/Storage Data Generating Process Research Questions Example: map visualization / privacy Psychology, Law, Math&Comp, Business Example: Hadoop MapReduce; High Frequency Data Economics, Social Sciences, Business, Math&Comp Example: Hadoop Distributed File System Math & Computer Science, Applied Statistics Examples: geolocated social media + survey + administrative data Social Science & Psychology, Humanities, Econ, Business Examples: Behavior of interest (migration/political participation/job searches) Any field
..and think about legal framework