Big Data a big issue for Official Statistics?

Transcription

1 Big Data a big issue for Official Statistics? ASC Conference 26 September 2014 Pete Brodie Session objectives Big Data and Official Statistics The ONS Big Data Project aims Wider engagement and communication Pilots Infrastructure, Innovation Labs Smartmeters Mobile Phones Prices Twitter Emerging Findings Next Steps 1

2 Big Data and Official Statistics Replace existing outputs Produce an entirely new output Complement other sources: Filling in gaps Auxiliary variables for statistical models Improve operational processes Quality assurance ONS Big Data Project A one year project which aims to: investigate the potential for big data in official statistics while understanding the challenges establish an ONS policy and longer term strategy which incorporates ONS s position within Government and internationally in this field Recommend next steps to support the strategy going forward 2

3 Wider engagement and communication International: UNECE / ESS Cross-government: HMG Data Science Community of Interest Group and Cross Profession Working Group GSS data strategy GDS/DfT/DECC/BoE/DSTL Big data for statistics vs other types of analysis Wider engagement and communication Academia: ESRC/RSS University of Southampton/Cardiff/Huddersfield Private sector: Mobile network operators Mysupermarket/Billion Prices Google Privacy groups B2011 Privacy Advisory Group/GDS Privacy and Ethical Committee 3

4 Pilots - Infrastructure Huge and continuously growing data streams, requiring new data architectures and software Feasibility and efficiency of processing, typically requiring parallel computing on a large scale New skills will be required, bringing together statistical and technological expertise Pilots Innovation Labs 4

5 Pilots Innovation Labs A new facility to allow research with datasets and tools without compromising ONS security INDEPENDENT of ONS main systems NOT SECURE so the only data that can go on there is PUBLIC data A private cloud individual machines are pooled together to provide an integrated environment, accessed via web browser Pilots Smart meters Investigating the potential of smart meter electricity data (high frequency 30 mins) to identify household occupancy levels, potentially household structure England and Ireland both conducted pilots of rollout in data now available for research Southampton University commissioned by Beyond 2011 to conduct preliminary research 5

6 Pilots Smart meters Irish smart meter pilot study: Single meter, total daily electricity consumption Christmas 2009 Christmas 2010 Total daily electricity consumption (kwh) J uy l O c t o b e r Feb ru a 2r y0 1 0 Day Ma y Consecutive days w ith low consumption, possibly a week away? A u g u s t De c e m2 b0 e1 r0 Pilots Mobile Phones Investigating using mobile phone data to model population flows, eg travel to work statistics GDS Discussions with mobile phone providers (Telefonica, Vodafone, EE) to provide aggregate data on origin-destination flows Ethical/commercial issues 6

7 Pilots Prices Scraping prices data from the internet for use within price statistics Potential for richer, more frequent and cheaper data collection Focus on grocery prices from three on-line supermarkets Prototype scrapers collecting a selection of CPI/RPI item categories (daily collection) We are purchasing data from MySupermarket.com (linked data, longer time series) for research purposes Webscraping Rendered webpage: HTML code:... </div><div class="productlists" id="endfacets-1"><ul class="cf products line"><li id="p " class=" first"><div class="desc"><h3 class="inbasketinfocontainer"><a id="h " href="/groceries/product/details/?id= " class="si_pl_ title"><span class="image"><img src=" alt="" /><!----></span>warburtons Toastie Sliced White Bread 800G</a></h3><p class="limitedlife"><a href=" the freshest food to your door- Find out more ></a></p><div class="desccontent"><!----><div class="promo"><a href="/groceries/specialoffers/specialofferdetail/default.aspx?promoid=a " title="all products available for this offer" id="flyout promo-a pos" class="promoflyout"><span class="promoimgbox"><img src="/groceries/uiassets/i/sites/retail/superstore/online/product/pos/2for.png" class="promoflyout promo" alt="special Offer" id="flyout promo-a posimg" /></span><em>any 2 for 2.00</em></a><span> valid from 21/1/2014 until 10/2/2014</span></div><div class="tools"><div class="moreinfo"><a href="/groceries/product/details/?id= " class="midiflyout" id="flyout midi-0-"><img class="midiflyout hd" src=" nfoblue.gif" alt="" title="view product information" id="flyout midi-1-" /></a></div><!----><div class="links"><ul><li><a href=" class="shelfflyout active plaintooltip" id="s-tt " title="premium White Bread"> Rest of <span class="hide">premium White Bread <!----></span>shelf </a></li></ul></div></div></div></div><div class="quantity"><div class="content addtobasket"><p class="price"><span class="lineprice"> 1.45<!----></span><span class="linepriceabbr"> ( 0.18/100g)</span></p><h4 class="hide">add to basket</h4><form method="post" id="fmultisearch "... 7

8 The Billion Prices MIT Lehman Brothers files for bankruptcy (15 Sept 2008) Daily Online Price Index (United States) Pilots Twitter Using geo-located tweets from Twitter to provide insights on internal migration: Harvesting geo-located tweets from the Twitter Streaming API Development of clustering methods to group tweets by user and identify significant locations Using Addressbase to classify clusters as (e.g. residential or commercial) Developing rules to determine place of residence and track changes over time. 8

9 Lots of activity in different places but where does this Twitter user live? Density-Based Spatial Clustering with Noise (DBSCAN) 2 parameters: Distance Minimum number of points 9

10 Most likely lives here Cluster_id Northing Easting Count 60033_ _ _ _ _ _ Raw Data Cluster Centroid Noise Prices Pilot Web scrapers Possible purchase of price data eg from Mysupermarket 10

11 Smartmeter Pilot Potential to identify vacant/unoccupied properties Intelligence used in the field/within address register Big Data Technologies Innovation Labs Experience and expertise feed through to IT strategy 11

12 Statistical Precision may not be an issue but bias is Traditional statistical methods must not be forgotten in the hype Crucial role for ONS Ethical Engagement with privacy groups ONS Ethical Committee Research into public attitudes 12

13 Commercial Different commercial models need to be considered Need to develop procurement framework for engagement issues around brand Capability Team sport, cross disciplinary Need for senior focus Chief Data Scientist Time to develop staff, encourage innovation Links with academia support research, attract graduates, placement students Collaboration is key 13

14 Starting to demonstrate tangible benefits and provide evidence that challenges can be overcome But more long term work is needed to build on these initial findings Aligned with ONS and GSS strategy, Government initiatives, UNECE/Eurostat work programmes, academic investment Next steps - aims Support ONS business as usual, projects and programmes with BD Build up a pool of expertise Develop best practise, standards and guidance and training, understand data scientist role Continue to develop partnerships across the GSS Continue to develop partnerships with academics and the private sector Contribute to international and cross Government BD initiatives Implement the BD strategy 14