Big Data a big issue for Official Statistics?

Size: px
Start display at page:

Download "Big Data a big issue for Official Statistics?"

Transcription

1 Big Data a big issue for Official Statistics? ASC Conference 26 September 2014 Pete Brodie Session objectives Big Data and Official Statistics The ONS Big Data Project aims Wider engagement and communication Pilots Infrastructure, Innovation Labs Smartmeters Mobile Phones Prices Twitter Emerging Findings Next Steps 1

2 Big Data and Official Statistics Replace existing outputs Produce an entirely new output Complement other sources: Filling in gaps Auxiliary variables for statistical models Improve operational processes Quality assurance ONS Big Data Project A one year project which aims to: investigate the potential for big data in official statistics while understanding the challenges establish an ONS policy and longer term strategy which incorporates ONS s position within Government and internationally in this field Recommend next steps to support the strategy going forward 2

3 Wider engagement and communication International: UNECE / ESS Cross-government: HMG Data Science Community of Interest Group and Cross Profession Working Group GSS data strategy GDS/DfT/DECC/BoE/DSTL Big data for statistics vs other types of analysis Wider engagement and communication Academia: ESRC/RSS University of Southampton/Cardiff/Huddersfield Private sector: Mobile network operators Mysupermarket/Billion Prices Google Privacy groups B2011 Privacy Advisory Group/GDS Privacy and Ethical Committee 3

4 Pilots - Infrastructure Huge and continuously growing data streams, requiring new data architectures and software Feasibility and efficiency of processing, typically requiring parallel computing on a large scale New skills will be required, bringing together statistical and technological expertise Pilots Innovation Labs 4

5 Pilots Innovation Labs A new facility to allow research with datasets and tools without compromising ONS security INDEPENDENT of ONS main systems NOT SECURE so the only data that can go on there is PUBLIC data A private cloud individual machines are pooled together to provide an integrated environment, accessed via web browser Pilots Smart meters Investigating the potential of smart meter electricity data (high frequency 30 mins) to identify household occupancy levels, potentially household structure England and Ireland both conducted pilots of rollout in data now available for research Southampton University commissioned by Beyond 2011 to conduct preliminary research 5

6 Pilots Smart meters Irish smart meter pilot study: Single meter, total daily electricity consumption Christmas 2009 Christmas 2010 Total daily electricity consumption (kwh) J uy l O c t o b e r Feb ru a 2r y0 1 0 Day Ma y Consecutive days w ith low consumption, possibly a week away? A u g u s t De c e m2 b0 e1 r0 Pilots Mobile Phones Investigating using mobile phone data to model population flows, eg travel to work statistics GDS Discussions with mobile phone providers (Telefonica, Vodafone, EE) to provide aggregate data on origin-destination flows Ethical/commercial issues 6

7 Pilots Prices Scraping prices data from the internet for use within price statistics Potential for richer, more frequent and cheaper data collection Focus on grocery prices from three on-line supermarkets Prototype scrapers collecting a selection of CPI/RPI item categories (daily collection) We are purchasing data from MySupermarket.com (linked data, longer time series) for research purposes Webscraping Rendered webpage: HTML code:... </div><div class="productlists" id="endfacets-1"><ul class="cf products line"><li id="p " class=" first"><div class="desc"><h3 class="inbasketinfocontainer"><a id="h " href="/groceries/product/details/?id= " class="si_pl_ title"><span class="image"><img src="http://img.tesco.com/groceries/pi/121\ \idshot_90x90.jpg" alt="" /><!----></span>warburtons Toastie Sliced White Bread 800G</a></h3><p class="limitedlife"><a href="http://www.tesco.com/groceries/zones/default.aspx?name=quality-andfreshness">delivering the freshest food to your door- Find out more ></a></p><div class="desccontent"><!----><div class="promo"><a href="/groceries/specialoffers/specialofferdetail/default.aspx?promoid=a " title="all products available for this offer" id="flyout promo-a pos" class="promoflyout"><span class="promoimgbox"><img src="/groceries/uiassets/i/sites/retail/superstore/online/product/pos/2for.png" class="promoflyout promo" alt="special Offer" id="flyout promo-a posimg" /></span><em>any 2 for 2.00</em></a><span> valid from 21/1/2014 until 10/2/2014</span></div><div class="tools"><div class="moreinfo"><a href="/groceries/product/details/?id= " class="midiflyout" id="flyout midi-0-"><img class="midiflyout hd" src="http://ui.tescoassets.com/groceries/uiassets/i/../compressed/i_ /sites/retail/superstore/online/product/i nfoblue.gif" alt="" title="view product information" id="flyout midi-1-" /></a></div><!----><div class="links"><ul><li><a href="http://www.tesco.com/groceries/product/browse/default.aspx?notepad=white%20sliced%20loaf%20800g&n= " class="shelfflyout active plaintooltip" id="s-tt " title="premium White Bread"> Rest of <span class="hide">premium White Bread <!----></span>shelf </a></li></ul></div></div></div></div><div class="quantity"><div class="content addtobasket"><p class="price"><span class="lineprice"> 1.45<!----></span><span class="linepriceabbr"> ( 0.18/100g)</span></p><h4 class="hide">add to basket</h4><form method="post" id="fmultisearch "... 7

8 The Billion Prices MIT Lehman Brothers files for bankruptcy (15 Sept 2008) Daily Online Price Index (United States) Pilots Twitter Using geo-located tweets from Twitter to provide insights on internal migration: Harvesting geo-located tweets from the Twitter Streaming API Development of clustering methods to group tweets by user and identify significant locations Using Addressbase to classify clusters as (e.g. residential or commercial) Developing rules to determine place of residence and track changes over time. 8

9 Lots of activity in different places but where does this Twitter user live? Density-Based Spatial Clustering with Noise (DBSCAN) 2 parameters: Distance Minimum number of points 9

10 Most likely lives here Cluster_id Northing Easting Count 60033_ _ _ _ _ _ Raw Data Cluster Centroid Noise Prices Pilot Web scrapers Possible purchase of price data eg from Mysupermarket 10

11 Smartmeter Pilot Potential to identify vacant/unoccupied properties Intelligence used in the field/within address register Big Data Technologies Innovation Labs Experience and expertise feed through to IT strategy 11

12 Statistical Precision may not be an issue but bias is Traditional statistical methods must not be forgotten in the hype Crucial role for ONS Ethical Engagement with privacy groups ONS Ethical Committee Research into public attitudes 12

13 Commercial Different commercial models need to be considered Need to develop procurement framework for engagement issues around brand Capability Team sport, cross disciplinary Need for senior focus Chief Data Scientist Time to develop staff, encourage innovation Links with academia support research, attract graduates, placement students Collaboration is key 13

14 Starting to demonstrate tangible benefits and provide evidence that challenges can be overcome But more long term work is needed to build on these initial findings Aligned with ONS and GSS strategy, Government initiatives, UNECE/Eurostat work programmes, academic investment Next steps - aims Support ONS business as usual, projects and programmes with BD Build up a pool of expertise Develop best practise, standards and guidance and training, understand data scientist role Continue to develop partnerships across the GSS Continue to develop partnerships with academics and the private sector Contribute to international and cross Government BD initiatives Implement the BD strategy 14

ONS Big Data Project. Karen Gask Office for National Statistics

ONS Big Data Project. Karen Gask Office for National Statistics ONS Big Data Project Karen Gask Office for National Statistics Plan for today Introduce the ONS Big Data Project Provide a brief overview of our work to date Provide information about our future work plan

More information

ONS Big Data Project Progress report: Qtr 2 April to June 2014

ONS Big Data Project Progress report: Qtr 2 April to June 2014 Official ONS Big Data Project Qtr 2 Report August 2014 ONS Big Data Project Progress report: Qtr 2 April to June 2014 Jane Naylor, Nigel Swier, Susan Williams Office for National Statistics Background

More information

ONS Big Data Project Progress report: Qtr 3 July to Sept 2014

ONS Big Data Project Progress report: Qtr 3 July to Sept 2014 Official ONS Big Data Project Qtr 3 Report Nov 2014 ONS Big Data Project Progress report: Qtr 3 July to Sept 2014 Jane Naylor, Nigel Swier, Susan Williams, Karen Gask Office for National Statistics Background

More information

ONS Big Data Project Progress report: Qtr 1 Jan to Mar 2014

ONS Big Data Project Progress report: Qtr 1 Jan to Mar 2014 Official ONS Big Data Project Qtr 1 Report May 2014 ONS Big Data Project Progress report: Qtr 1 Jan to Mar 2014 Jane Naylor, Nigel Swier, Susan Williams Office for National Statistics Background The amount

More information

ONS Big Data Project Progress report: Qtr 4 October to Dec 2014

ONS Big Data Project Progress report: Qtr 4 October to Dec 2014 Official ONS Big Data Project Qtr 4 Report Mar 2015 ONS Big Data Project Progress report: Qtr 4 October to Dec 2014 Jane Naylor, Nigel Swier, Susan Williams, Karen Gask, Rob Breton Office for National

More information

ONS Big Data Project Progress report: Qtr 1 January to March 2015

ONS Big Data Project Progress report: Qtr 1 January to March 2015 Official ONS Big Data Project Qtr 1 Report May 2015 ONS Big Data Project Progress report: Qtr 1 January to March 2015 Jane Naylor, Nigel Swier, Susan Williams, Karen Gask, Rob Breton Office for National

More information

Big data coming soon... to an NSI near you. John Dunne. Central Statistics Office (CSO), Ireland John.Dunne@cso.ie

Big data coming soon... to an NSI near you. John Dunne. Central Statistics Office (CSO), Ireland John.Dunne@cso.ie Big data coming soon... to an NSI near you John Dunne Central Statistics Office (CSO), Ireland John.Dunne@cso.ie Big data is beginning to be explored and exploited to inform policy making. However these

More information

big data in the European Statistical System

big data in the European Statistical System Conference by STATEC and EUROSTAT Savoir pour agir: la statistique publique au service des citoyens big data in the European Statistical System Michail SKALIOTIS EUROSTAT, Head of Task Force 'Big Data'

More information

Computer Programming for the Social Sciences

Computer Programming for the Social Sciences Department of Social and Political Sciences Computer Programming for the Social Sciences This two day workshop will teach beginner level, practical computer programming skills for use in social science

More information

International collaboration to understand the relevance of Big Data for official statistics

International collaboration to understand the relevance of Big Data for official statistics Statistical Journal of the IAOS 31 (2015) 159 163 159 DOI 10.3233/SJI-150889 IOS Press International collaboration to understand the relevance of Big Data for official statistics Steven Vale United Nations

More information

Big Data and Official Statistics The UN Global Working Group

Big Data and Official Statistics The UN Global Working Group Big Data and Official Statistics The UN Global Working Group Dr. Ronald Jansen Chief, International Trade Statistics United Nations Statistics Division jansen1@un.org Overview What is Big Data? What is

More information

Online and Scalable Data Validation in Advanced Metering Infrastructures

Online and Scalable Data Validation in Advanced Metering Infrastructures Online and Scalable Data Validation in Advanced Metering Infrastructures Chalmers University of technology Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation

More information

Danny Wang, Ph.D. Vice President of Business Strategy and Risk Management Republic Bank

Danny Wang, Ph.D. Vice President of Business Strategy and Risk Management Republic Bank Danny Wang, Ph.D. Vice President of Business Strategy and Risk Management Republic Bank Agenda» Overview» What is Big Data?» Accelerates advances in computer & technologies» Revolutionizes data measurement»

More information

Hadoop for Enterprises:

Hadoop for Enterprises: Hadoop for Enterprises: Overcoming the Major Challenges Introduction to Big Data Big Data are information assets that are high volume, velocity, and variety. Big Data demands cost-effective, innovative

More information

CSC590: Selected Topics BIG DATA & DATA MINING. Lecture 2 Feb 12, 2014 Dr. Esam A. Alwagait

CSC590: Selected Topics BIG DATA & DATA MINING. Lecture 2 Feb 12, 2014 Dr. Esam A. Alwagait CSC590: Selected Topics BIG DATA & DATA MINING Lecture 2 Feb 12, 2014 Dr. Esam A. Alwagait Agenda Introduction What is Big Data Why Big Data? Characteristics of Big Data Applications of Big Data Problems

More information

Data for the Public Good. The Government Statistical Service Data Strategy

Data for the Public Good. The Government Statistical Service Data Strategy Data for the Public Good The Government Statistical Service Data Strategy November 2013 1 Foreword by the National Statistician When I launched Building the Community - The Strategy for the Government

More information

The Canadian Realities of Big Data and Business Analytics. Utsav Arora February 12, 2014

The Canadian Realities of Big Data and Business Analytics. Utsav Arora February 12, 2014 The Canadian Realities of Big Data and Business Analytics Utsav Arora February 12, 2014 Things to think about for today How Important is Big Data for me? Why do I need to implement Big Data and Analytics

More information

Unlocking the Full Potential of Big Data

Unlocking the Full Potential of Big Data Unlocking the Full Potential of Big Data Lilli Japec, Frauke Kreuter JOS anniversary June 2015 facebook.com/statisticssweden @SCB_nyheter The report is available at https://www.aapor.org Task Force Members:

More information

How to Leverage Big Data in the Cloud to Gain Competitive Advantage

How to Leverage Big Data in the Cloud to Gain Competitive Advantage How to Leverage Big Data in the Cloud to Gain Competitive Advantage James Kobielus, IBM Big Data Evangelist Editor-in-Chief, IBM Data Magazine Senior Program Director, Product Marketing, Big Data Analytics

More information

UNIVERSITY OF INFINITE AMBITIONS. MASTER OF SCIENCE COMPUTER SCIENCE DATA SCIENCE AND SMART SERVICES

UNIVERSITY OF INFINITE AMBITIONS. MASTER OF SCIENCE COMPUTER SCIENCE DATA SCIENCE AND SMART SERVICES UNIVERSITY OF INFINITE AMBITIONS. MASTER OF SCIENCE COMPUTER SCIENCE DATA SCIENCE AND SMART SERVICES MASTER S PROGRAMME COMPUTER SCIENCE - DATA SCIENCE AND SMART SERVICES (DS3) This is a specialization

More information

Research trends relevant to data warehousing and OLAP include [Cuzzocrea et al.]: Combining the benefits of RDBMS and NoSQL database systems

Research trends relevant to data warehousing and OLAP include [Cuzzocrea et al.]: Combining the benefits of RDBMS and NoSQL database systems DATA WAREHOUSING RESEARCH TRENDS Research trends relevant to data warehousing and OLAP include [Cuzzocrea et al.]: Data source heterogeneity and incongruence Filtering out uncorrelated data Strongly unstructured

More information

White Paper. Version 1.2 May 2015 RAID Incorporated

White Paper. Version 1.2 May 2015 RAID Incorporated White Paper Version 1.2 May 2015 RAID Incorporated Introduction The abundance of Big Data, structured, partially-structured and unstructured massive datasets, which are too large to be processed effectively

More information

G-Cloud II Services Service Definition Accenture Cloud SaaS Implementation Services Google Apps

G-Cloud II Services Service Definition Accenture Cloud SaaS Implementation Services Google Apps G-Cloud II Services Service Definition Accenture Cloud SaaS Implementation Services Google Apps 1 Table of Contents 1. Scope of our Services... 3 2. Approach... 4 3. Assets and Tools... 5 4. Outcomes...

More information

Utilizing big data to bring about innovative offerings and new revenue streams DATA-DERIVED GROWTH

Utilizing big data to bring about innovative offerings and new revenue streams DATA-DERIVED GROWTH Utilizing big data to bring about innovative offerings and new revenue streams DATA-DERIVED GROWTH ACTIONABLE INTELLIGENCE Ericsson is driving the development of actionable intelligence within all aspects

More information

BIG DATA FOR DEVELOPMENT: A PRIMER

BIG DATA FOR DEVELOPMENT: A PRIMER June 2013 BIG DATA FOR DEVELOPMENT: A PRIMER Harnessing Big Data For Real-Time Awareness WHAT IS BIG DATA? Big Data is an umbrella term referring to the large amounts of digital data continually generated

More information

Deploying Big Data to the Cloud: Roadmap for Success

Deploying Big Data to the Cloud: Roadmap for Success Deploying Big Data to the Cloud: Roadmap for Success James Kobielus Chair, CSCC Big Data in the Cloud Working Group IBM Big Data Evangelist. IBM Data Magazine, Editor-in- Chief. IBM Senior Program Director,

More information

Visualization and Big Data in Official Statistics

Visualization and Big Data in Official Statistics Visualization and Big Data in Official Statistics Martijn Tennekes In cooperation with Piet Daas, Marco Puts, May Offermans, Alex Priem, Edwin de Jonge From a Official Statistics point of view Three types

More information

Introduction to Big Data! with Apache Spark" UC#BERKELEY#

Introduction to Big Data! with Apache Spark UC#BERKELEY# Introduction to Big Data! with Apache Spark" UC#BERKELEY# So What is Data Science?" Doing Data Science" Data Preparation" Roles" This Lecture" What is Data Science?" Data Science aims to derive knowledge!

More information

Project Outline: Data Integration: towards producing statistics by integrating different data sources

Project Outline: Data Integration: towards producing statistics by integrating different data sources Project Outline: Data Integration: towards producing statistics by integrating different data sources Introduction There are many new opportunities created by data sources such as Big Data and Administrative

More information

IEEE IoT IoT Scenario & Use Cases: Social Sensors

IEEE IoT IoT Scenario & Use Cases: Social Sensors IEEE IoT IoT Scenario & Use Cases: Social Sensors Service Description More and more, people have the possibility to monitor important parameters in their home or in their surrounding environment. As an

More information

Ironfan Your Foundation for Flexible Big Data Infrastructure

Ironfan Your Foundation for Flexible Big Data Infrastructure Ironfan Your Foundation for Flexible Big Data Infrastructure Benefits With Ironfan, you can expect: Reduced cycle time. Provision servers in minutes not days. Improved visibility. Increased transparency

More information

Big Data 101: Harvest Real Value & Avoid Hollow Hype

Big Data 101: Harvest Real Value & Avoid Hollow Hype Big Data 101: Harvest Real Value & Avoid Hollow Hype 2 Executive Summary Odds are you are hearing the growing hype around the potential for big data to revolutionize our ability to assimilate and act on

More information

ACEDS Membership Benefits Training, Resources and Networking for the E-Discovery Community

ACEDS Membership Benefits Training, Resources and Networking for the E-Discovery Community ACEDS Membership Benefits Training, Resources and Networking for the E-Discovery Community! Exclusive News and Analysis! Weekly Web Seminars! Podcasts! On- Demand Training! Networking! Resources! Jobs

More information

Top 10 IT Trends that will shape 2014. David Chin Chair BICSI Southeast Asia

Top 10 IT Trends that will shape 2014. David Chin Chair BICSI Southeast Asia Top 10 IT Trends that will shape 2014 David Chin Chair BICSI Southeast Asia Gartner Hype Cycle: Emerging Technologies 2013 Convergence of cloud, social, mobile, devices, analytic-driven automation and

More information

ESS event: Big Data in Official Statistics. Antonino Virgillito, Istat

ESS event: Big Data in Official Statistics. Antonino Virgillito, Istat ESS event: Big Data in Official Statistics Antonino Virgillito, Istat v erbi v is 1 About me Head of Unit Web and BI Technologies, IT Directorate of Istat Project manager and technical coordinator of Web

More information

Analytics: The real-world use of big data

Analytics: The real-world use of big data Findings from the research collaboration of IBM Institute for Business Value and Saïd Business School, University of Oxford Analytics: The real-world use of big data How innovative enterprises extract

More information

Australian Government Cloud Computing Policy

Australian Government Cloud Computing Policy Australian Government Cloud Computing Policy Maximising the Value of Cloud VERSION 2.0 MAY 2013 AGIMO is part of the Department of Finance and Deregulation Contents Foreword 3 Introduction 4 Australian

More information

PRACTICAL DATA MINING IN A LARGE UTILITY COMPANY

PRACTICAL DATA MINING IN A LARGE UTILITY COMPANY QÜESTIIÓ, vol. 25, 3, p. 509-520, 2001 PRACTICAL DATA MINING IN A LARGE UTILITY COMPANY GEORGES HÉBRAIL We present in this paper the main applications of data mining techniques at Electricité de France,

More information

Split Lane Traffic Reporting at Junctions

Split Lane Traffic Reporting at Junctions Split Lane Traffic Reporting at Junctions White paper 1 Executive summary Split Lane Traffic Reporting at Junctions (SLT) from HERE is a major innovation in real time traffic reporting. The advanced algorithm

More information

Advanced Analytics & IoT Architectures

Advanced Analytics & IoT Architectures Advanced Analytics & IoT Architectures Presented by: Tom Marek and Orion Gebremedhin Use Case: ETL Offloading Have you outgrown your data delivery SLAs? Get the right data at the right time 2 ETL Processing

More information

Data Center Infrastructure Management. optimize. your data center with our. DCIM weather station. Your business technologists.

Data Center Infrastructure Management. optimize. your data center with our. DCIM weather station. Your business technologists. Data Center Infrastructure Management optimize your data center with our DCIM weather station Your business technologists. Powering progress Are you feeling the heat of your data center operations? Data

More information

Tables in the Cloud. By Larry Ng

Tables in the Cloud. By Larry Ng Tables in the Cloud By Larry Ng The Idea There has been much discussion about Big Data and the associated intricacies of how it can be mined, organized, stored, analyzed and visualized with the latest

More information

Building trust in the cloud. Alastair McAulay PA Consulting Group

Building trust in the cloud. Alastair McAulay PA Consulting Group Building trust in the cloud overcoming the inhibitors Alastair McAulay PA Consulting Group PA Knowledge Limited 2011. 2 Introduction Introduction to Alastair McAulay 25 year s experience in the IT Services

More information

Leveraging Data the Right Way

Leveraging Data the Right Way Leveraging Data the Right Way Use It or Lose It Unless organizations use information to drive action, Big Data or any data is more of a liability than an asset. The bottom line: Just get started. Here

More information

YOU VS THE SENSORS. Six Requirements for Visualizing the Internet of Things. Dan Potter Chief Marketing Officer, Datawatch Corporation

YOU VS THE SENSORS. Six Requirements for Visualizing the Internet of Things. Dan Potter Chief Marketing Officer, Datawatch Corporation YOU VS THE SENSORS Six Requirements for Visualizing the Internet of Things Dan Potter Chief Marketing Officer, Datawatch Corporation About Datawatch NASDAQ: DWCH Pioneer in real-time visual data discovery

More information

Delivery Plan for the Long Term Strategy for Population Surveys in Scotland 2009 2019

Delivery Plan for the Long Term Strategy for Population Surveys in Scotland 2009 2019 Delivery Plan for the Long Term Strategy for Population Surveys in Scotland 2009 2019 1. The Scottish Government s Long Term Strategy for Population Surveys in Scotland 2009 2019 was publish in June 2009.

More information

Cisco Data Center Services for OpenStack

Cisco Data Center Services for OpenStack Data Sheet Cisco Data Center Services for OpenStack Use Cisco Expertise to Accelerate Deployment of Your OpenStack Cloud Operating Environment Why OpenStack? OpenStack is an open source cloud operating

More information

Official Statistics in the Age. of Big Data. SAS Forum BeLux 2014. Michail.Skaliotis@ec.europa.eu Albrecht.Wirthmann@ec.europa.eu.

Official Statistics in the Age. of Big Data. SAS Forum BeLux 2014. Michail.Skaliotis@ec.europa.eu Albrecht.Wirthmann@ec.europa.eu. Official Statistics in the Age SAS Forum BeLux 2014 of Big Data Michail.Skaliotis@ec.europa.eu Albrecht.Wirthmann@ec.europa.eu Table of Contents / Storyboard What is "Official Statistics"? Drivers of Big

More information

Internet of Things World Forum Chicago

Internet of Things World Forum Chicago Internet of Things World Forum Chicago John McGagh Head of Innovation October 2014 October 2014 2 About this presentation I will address three points during our time together: 1 Our world 2 Connectivity

More information

Big Data & the Cloud: The Sum Is Greater Than the Parts

Big Data & the Cloud: The Sum Is Greater Than the Parts E-PAPER March 2014 Big Data & the Cloud: The Sum Is Greater Than the Parts Learn how to accelerate your move to the cloud and use big data to discover new hidden value for your business and your users.

More information

The use of Big Data for statistics

The use of Big Data for statistics Workshop on the use of mobile positioning data for tourism statistics Prague (CZ), 14 May 2014 The use of Big Data for statistics EUROSTAT, Unit G-3 "Short-term statistics; tourism" What is the role of

More information

Big Data. Case studies in Official Statistics. Martijn Tennekes. Special thanks to Piet Daas, Marco Puts, May Offermans, Alex Priem, Edwin de Jonge

Big Data. Case studies in Official Statistics. Martijn Tennekes. Special thanks to Piet Daas, Marco Puts, May Offermans, Alex Priem, Edwin de Jonge Big Data Case studies in Official Statistics Martijn Tennekes Special thanks to Piet Daas, Marco Puts, May Offermans, Alex Priem, Edwin de Jonge From a Official Statistics point of view Three types of

More information

Understanding Your Customer Journey by Extending Adobe Analytics with Big Data

Understanding Your Customer Journey by Extending Adobe Analytics with Big Data SOLUTION BRIEF Understanding Your Customer Journey by Extending Adobe Analytics with Big Data Business Challenge Today s digital marketing teams are overwhelmed by the volume and variety of customer interaction

More information

Social Business Intelligence For Retail Industry

Social Business Intelligence For Retail Industry Actionable Social Intelligence SOCIAL BUSINESS INTELLIGENCE FOR RETAIL INDUSTRY Leverage Voice of Customers, Competitors, and Competitor s Customers to Drive ROI Abstract Conversations on social media

More information

Big Data Strategy Issues Paper

Big Data Strategy Issues Paper Big Data Strategy Issues Paper MARCH 2013 Contents 1. Introduction 3 1.1 Where are we now? 3 1.2 Why a big data strategy? 4 2. Opportunities for Australian Government agencies 5 2.1 What the future looks

More information

III Big Data Technologies

III Big Data Technologies III Big Data Technologies Today, new technologies make it possible to realize value from Big Data. Big data technologies can replace highly customized, expensive legacy systems with a standard solution

More information

Urban Big Data Centre. Data services: Guide for researchers. December 2014 Version 2.0 Authors: Nick Bailey

Urban Big Data Centre. Data services: Guide for researchers. December 2014 Version 2.0 Authors: Nick Bailey Urban Big Data Centre Data services: Guide for researchers December 2014 Version 2.0 Authors: Nick Bailey 1 Introduction... 3 UBDC Data Services... 3 Open Data and the UBDC Open Data portal... 4 Safeguarded

More information

Conducting A Preparedness Assessment

Conducting A Preparedness Assessment Conducting A Preparedness Assessment The Next Step In Considering A Big Data Initiative Company Logo Here Prepared For Names of Key Recipients June 1, 2014 2013 Lodestar Advisory Partners All rights reserved.

More information

IT Game Changers: How Cloud, Mobile, Social and Big Data are Redefining Your IT Environment

IT Game Changers: How Cloud, Mobile, Social and Big Data are Redefining Your IT Environment cecmg Annual Meeting / March, 2015 IT Game Changers: How Cloud, Mobile, Social and Big Data are Redefining Your IT Environment Glenn Anderson IBM Lab Services and Training 2013 IBM Corporation 2 (c) Copyright

More information

Collaborations between Official Statistics and Academia in the Era of Big Data

Collaborations between Official Statistics and Academia in the Era of Big Data Collaborations between Official Statistics and Academia in the Era of Big Data World Statistics Day October 20-21, 2015 Budapest Vijay Nair University of Michigan Past-President of ISI vnn@umich.edu What

More information

This survey addresses individual projects, partnerships, data sources and tools. Please submit it multiple times - once for each project.

This survey addresses individual projects, partnerships, data sources and tools. Please submit it multiple times - once for each project. Introduction This survey has been developed jointly by the United Nations Statistics Division (UNSD) and the United Nations Economic Commission for Europe (UNECE). Our goal is to provide an overview of

More information

JISC Progress Report Template - Full

JISC Progress Report Template - Full JISC Progress Report Template - Full Project Identifier Project Title Project Hashtag Project Information To be completed by JISC U-CARE: Universities Collaborative Approach to Reducing Energy Start Date

More information

Using Predictive Maintenance to Approach Zero Downtime

Using Predictive Maintenance to Approach Zero Downtime SAP Thought Leadership Paper Predictive Maintenance Using Predictive Maintenance to Approach Zero Downtime How Predictive Analytics Makes This Possible Table of Contents 4 Optimizing Machine Maintenance

More information

Results of the UNSD/UNECE Survey on. organizational context and individual projects of Big Data

Results of the UNSD/UNECE Survey on. organizational context and individual projects of Big Data Statistical Commission Forty-sixth session 3 6 March 2015 Item 3(a) (iii) of the provisional agenda Items for discussion and decision: Data in support of the Post-2015 Development Agenda: Big Data Background

More information

Could Computing: Concepts and Cost Considerations

Could Computing: Concepts and Cost Considerations Could Computing: Concepts and Cost Considerations Arlene Minkiewicz, Chief Scientist PRICE Systems, LLC arlene.minkiewicz@pricesystems.com Optimize tomorrow today. 1 If computers of the kind I have advocated

More information

Advanced Big Data Analytics with R and Hadoop

Advanced Big Data Analytics with R and Hadoop REVOLUTION ANALYTICS WHITE PAPER Advanced Big Data Analytics with R and Hadoop 'Big Data' Analytics as a Competitive Advantage Big Analytics delivers competitive advantage in two ways compared to the traditional

More information

ECONOMIC COMMISSION FOR EUROPE 20 January 2015

ECONOMIC COMMISSION FOR EUROPE 20 January 2015 UNITED NATIONS ECE/CES/BUR/2015/FEB/11 ECONOMIC COMMISSION FOR EUROPE 20 January 2015 CONFERENCE OF EUROPEAN STATISTICIANS Meeting of the 2014/2015 Bureau Geneva (Switzerland), 17-18 February 2015 For

More information

International Journal of Advanced Engineering Research and Applications (IJAERA) ISSN: 2454-2377 Vol. 1, Issue 6, October 2015. Big Data and Hadoop

International Journal of Advanced Engineering Research and Applications (IJAERA) ISSN: 2454-2377 Vol. 1, Issue 6, October 2015. Big Data and Hadoop ISSN: 2454-2377, October 2015 Big Data and Hadoop Simmi Bagga 1 Satinder Kaur 2 1 Assistant Professor, Sant Hira Dass Kanya MahaVidyalaya, Kala Sanghian, Distt Kpt. INDIA E-mail: simmibagga12@gmail.com

More information

Analysis of Big Data Survey 2015 on Skills, Training and Capacity Building

Analysis of Big Data Survey 2015 on Skills, Training and Capacity Building Analysis of Big Data Survey 2015 on Skills, Training and Capacity Building D R A F T Version 1.0 12 Oct 2015 By UN Global Working Group on Big Data for Official Statistics Task Team on Skills, Training

More information

Big Data for Social Good. Nuria Oliver, PhD Scientific Director User, Data and Media Intelligence Telefonica Research

Big Data for Social Good. Nuria Oliver, PhD Scientific Director User, Data and Media Intelligence Telefonica Research Big Data for Social Good Nuria Oliver, PhD Scientific Director User, Data and Media Intelligence Telefonica Research 6.8 billion subscriptions 96% of world s population (ITU) Mobile penetration of 120%

More information

The Sandbox 2015 Report

The Sandbox 2015 Report United Nations Economic Commission for Europe Statistical Division Workshop on the Modernisation of Official Statistics November 24-25, 2015 The Sandbox project The Sandbox 2015 Report Antonino Virgillito

More information

WHAT DOES BIG DATA MEAN FOR OFFICIAL STATISTICS?

WHAT DOES BIG DATA MEAN FOR OFFICIAL STATISTICS? UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS 10 March 2013 WHAT DOES BIG DATA MEAN FOR OFFICIAL STATISTICS? At a High-Level Seminar on Streamlining Statistical Production

More information

Cloud Computing and Big Data What s the Big Deal

Cloud Computing and Big Data What s the Big Deal Cloud Computing and Big Data What s the Big Deal Arlene Minkiewicz, Chief Scientist PRICE Systems, LLC arlene.minkiewicz@pricesystems.com Optimize tomorrow today. 1 Agenda Introduction Cloud Computing

More information

Exploratory Research Design: Secondary Data

Exploratory Research Design: Secondary Data 1) Overview 2) Primary versus 3) Advantages & Uses of 4) Disadvantages of 5) Criteria for Evaluating 6) Classification of 7) Internal 8) Published External Secondary Sources 9) Computerized Databases 10)

More information

Data Analytics, Management, Security and Privacy (Priority Area B)

Data Analytics, Management, Security and Privacy (Priority Area B) PRIORITY AREA B: DATA ANALYTICS, MANAGEMENT, SECURITY AND PRIVACY ACTION PLAN Data Analytics, Security and Privacy (Priority Area B) Context Data is growing at an exponential rate; information on the web

More information

Using Data Mining and Machine Learning in Retail

Using Data Mining and Machine Learning in Retail Using Data Mining and Machine Learning in Retail Omeid Seide Senior Manager, Big Data Solutions Sears Holdings Bharat Prasad Big Data Solution Architect Sears Holdings Over a Century of Innovation A Fortune

More information

Cloud Computing and Big Data. What s the Big Deal?

Cloud Computing and Big Data. What s the Big Deal? Cloud Computing and Big Data. What s the Big Deal? Arlene Minkiewicz, Chief Scientist PRICE Systems, LLC arlene.minkiewicz@pricesystems.com 2013 PRICE Systems, LLC All Rights Reserved Decades of Cost Management

More information

Economic Directorate s Big Data Overview

Economic Directorate s Big Data Overview Census Scientific Advisory Council Economic Directorate s Big Data Overview April 14, 2016 1 DRAFT FOR INTERNAL USE ONLY Modernizing Economic Statistics FY17 Budget Initiative Customers need more frequent

More information

BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES

BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES Relational vs. Non-Relational Architecture Relational Non-Relational Rational Predictable Traditional Agile Flexible Modern 2 Agenda Big Data

More information

The STC for Event Analysis: Scalability Issues

The STC for Event Analysis: Scalability Issues The STC for Event Analysis: Scalability Issues Georg Fuchs Gennady Andrienko http://geoanalytics.net Events Something [significant] happened somewhere, sometime Analysis goal and domain dependent, e.g.

More information

City Technology Platform Technical Architecture Context

City Technology Platform Technical Architecture Context December 2013 City Technology Platform Technical Architecture Context Intro A major component of the 24m Future City Demonstrator award from the Technology Strategy Board (TSB) has been the design and

More information

Australian Government Cloud Computing Policy

Australian Government Cloud Computing Policy Australian Government Cloud Computing Policy Maximising the Value of Cloud VERSION 2.1 JULY 2013 AGIMO is part of the Department of Finance and Deregulation Contents Foreword 3 Introduction 4 Policy 5

More information

CIS 4930/6930 Spring 2014 Introduction to Data Science Data Intensive Computing. University of Florida, CISE Department Prof.

CIS 4930/6930 Spring 2014 Introduction to Data Science Data Intensive Computing. University of Florida, CISE Department Prof. CIS 4930/6930 Spring 2014 Introduction to Data Science Data Intensive Computing University of Florida, CISE Department Prof. Daisy Zhe Wang Data Science Overview Why, What, How, Who Outline Why Data Science?

More information

Data Isn't Everything

Data Isn't Everything June 17, 2015 Innovate Forward Data Isn't Everything The Challenges of Big Data, Advanced Analytics, and Advance Computation Devices for Transportation Agencies. Using Data to Support Mission, Administration,

More information

move move us Newsletter 2014 Content MoveUs has successfully finished the first year of the project!

move move us Newsletter 2014 Content MoveUs has successfully finished the first year of the project! move us ICT CLOUD-BASED PLATFORM AND MOBILITY SERVICES : AVAILABLE, UNIVERSAL AND SAFE FOR ALL USERS MoveUs has successfully finished the first year of the project! Newsletter 2014 Welcome to MoveUs newsletter.

More information

Event report. Towards self-sustaining data reuse in Europe

Event report. Towards self-sustaining data reuse in Europe Towards self-sustaining data reuse in Europe 18 th of June Madrid, 2015 The EuDEco initial clustering workshop was a highly interactive half-day event held in colocation with the Big Data Value Association

More information

Italian Enterprises Adopt Big Data Solutions. Forrester Consulting

Italian Enterprises Adopt Big Data Solutions. Forrester Consulting Italian Enterprises Adopt Big Data Solutions Forrester Consulting About Forrester Consulting Forrester Consulting provides independent and objective research-based consulting to help leaders succeed in

More information

Title. Enterprise Europe Network & Innovate UK

Title. Enterprise Europe Network & Innovate UK Enterprise Europe Network & Innovate UK Title Practical help for small and medium businesses engaged with research & innovation. Sub-title Roger Hetherington 9 th December 2014 PLACE PARTNER S LOGO HERE

More information

Information Economy Strategy amee s Consultation Response to the Department for Business, Innovation & Skills

Information Economy Strategy amee s Consultation Response to the Department for Business, Innovation & Skills Information Economy Strategy amee s Consultation Response to the Department for Business, Innovation & Skills March 2013 amee 4th Floor, 70/74 City Road London EC1Y 2BJ 1 amee amee s mission is to provide

More information

Hadoop Beyond Hype: Complex Adaptive Systems Conference Nov 16, 2012. Viswa Sharma Solutions Architect Tata Consultancy Services

Hadoop Beyond Hype: Complex Adaptive Systems Conference Nov 16, 2012. Viswa Sharma Solutions Architect Tata Consultancy Services Hadoop Beyond Hype: Complex Adaptive Systems Conference Nov 16, 2012 Viswa Sharma Solutions Architect Tata Consultancy Services 1 Agenda What is Hadoop Why Hadoop? The Net Generation is here Sizing the

More information

Capgemini Big Data Analytics Sandbox for Financial Services

Capgemini Big Data Analytics Sandbox for Financial Services Capgemini Big Data Analytics Sandbox for Financial Services Put your data to use quickly without spending a fortune 2 Capgemini Big Data Analytics Sandbox for Financial Services Table of Contents 1. A

More information

Big Data : New opportunities for EM&V of energy efficiency programs. Annika Todd, Erin Hult

Big Data : New opportunities for EM&V of energy efficiency programs. Annika Todd, Erin Hult Big Data : New opportunities for EM&V of energy efficiency programs Annika Todd, Erin Hult April 2014 Data explosion in other fields Pedabytes of data collected by Google, Netflix, Amazon Development of

More information

GfK Crossmedia Link, Unlocking new insights

GfK Crossmedia Link, Unlocking new insights GfK Crossmedia Link, Unlocking new insights Helsinki, February 5 th 2015 GfK 2015 Intro GfK Crossmedia Link Helsinki, February 5 th, 2015 1 Changing Consumer Behaviour GfK 2015 Intro GfK Crossmedia Link

More information

Big Data, Official Statistics and Social Science Research: Emerging Data Challenges

Big Data, Official Statistics and Social Science Research: Emerging Data Challenges Big Data, Official Statistics and Social Science Research: Emerging Data Challenges Professor Paul Cheung Director, United Nations Statistics Division Building the Global Information System Elements of

More information

ESS event: Big Data in Official Statistics

ESS event: Big Data in Official Statistics ESS event: Big Data in Official Statistics v erbi v is 1 Parallel sessions 2A and 2B LEARNING AND DEVELOPMENT: CAPACITY BUILDING AND TRAINING FOR ESS HUMAN RESOURCES FACILITATOR: JOSÉ CERVERA- FERRI 2

More information

Pre-commercial public procurement and cross-border Living Labs

Pre-commercial public procurement and cross-border Living Labs Pre-commercial public procurement and cross-border Living Labs Innovation Union: the contribution of Europe s regions and cities Brussels, 27 November 2013 Open innovation in Aosta Valley WhatisaLivingLab?

More information

PREDICTIVE MARKETING, DIGITAL ATTRIBUTION, OPTIMIZATION, AND DATA-DRIVEN PERSONALIZATION

PREDICTIVE MARKETING, DIGITAL ATTRIBUTION, OPTIMIZATION, AND DATA-DRIVEN PERSONALIZATION PREDICTIVE MARKETING, DIGITAL ATTRIBUTION, OPTIMIZATION, AND DATA-DRIVEN PERSONALIZATION A m a r t y a B h a t t a c h a r j y & S u n e e l G r o v e r P r i n c i p a l S o l u t i o n A r c h i t e

More information

Ten Mistakes to Avoid

Ten Mistakes to Avoid EXCLUSIVELY FOR TDWI PREMIUM MEMBERS TDWI RESEARCH SECOND QUARTER 2014 Ten Mistakes to Avoid In Big Data Analytics Projects By Fern Halper tdwi.org Ten Mistakes to Avoid In Big Data Analytics Projects

More information

Big Data The Next Phase Lessons from a Decade+ Experiment in Big Data

Big Data The Next Phase Lessons from a Decade+ Experiment in Big Data Big Data The Next Phase Lessons from a Decade+ Experiment in Big Data David Belanger PhD Senior Research Fellow Stevens Institute of Technology dbelange@stevens.edu 1 Outline Big Data Overview Thinking

More information

Global Machine to Machine Smart Services

Global Machine to Machine Smart Services Global Machine to Machine Smart Services Connecting people, devices and systems vodacom.co.za/business M2M communications helping to reduce costs and improving efficiency and service to customers 2 Global

More information