1 Big Data, Applied Overview of Big Data from our experiences at Telefonica s digital services Jose Luis September 2014
2 Agenda 1. What s the big deal with Big Data? A new information business enabled by a new technology 2. Where does all the hype come from? Google -> Yahoo -> Hadoop -> McKinsey -> Boom!! 3. Which are the benefits of Big Data for society and people? Flu spread (MX), Earthquakes (MX) 4. How does Big Data impact people s privacy? From individual user consent, to anonimized & aggregated 5. Why is it relevant for businesses? Better customer insight (BI), and data monetization in new markets 6. What is Telefonica doing to benefit from Big Data? Tef. Digital SmartSteps for Retail & Transport; Big Data Innovation 2
3 What s the big deal with Big Data? Big Data Big Deals McKinsey McKinsey 3
4 Where does all the hype come from? rted it, a t s e l g Goo ed it, c r u o s pen d it, e s u Yahoo o s r d othe n a k o o Faceb but ok it to eport to r s y e s tings e e McKin m d r 00 Boa 5 e n u t r Fo huge Today, marketing budgets are being thrown at those two words, driven by new business no wonder all the noise! 2011: McKinsey Global Institute 2004: Google publishes Map Reduce paper (link: here) 2006: Yahoo s Doug Cutting open sources Hadoop out of his older search engine project Nutch. (Link: here) publishes report on Big Data s market potential for business, reaching out of the tech. world (link: here) 4
5 Big Data A new toolbox for 3-Vs problems New tools to make feasible and cheap to process data that previously was - too Voluminous (think petabytes), - too Varied (think Internet) - too much Velocity (millions per second). Skilled teams also needed : Security, PM, Orchestration Hardcore experts on the tools are needed, of course but several startups are focusing in lowering the entry barrier. Data Visualization: UX & PR demand lots of eye-candy, so, design freaks wanted. Data Quality: rough stuff, you will need ad-hoc tools to be built. 5
6 Big Data a question of scale Storage and Computing Do you really need all that equipment for Big Data? Bandwidth Build it where the data heap is A proof-of-concept phase lets you fine-tune the needs for storage and computing in the final product. (e.g. to avoid ending up with ghost equipment) as big data transfers might take significant bandwidth and time. (e.g. test a new pipeline without risking production data integrity ) 6
7 It is really about Big stuff, so expect some complexity! A typical Data Science pipeline might crunch the data in around a hundred steps Comparing it to a Mobile App, it d be like comparing a Pop band to a Philharmonic Orchestra. VS As in an Orchestra, the string section must know if they go before or after the wind section so a virtual 24x7 Herbert von Karajan is needed. Big Data really needs a first exploratory phase, for testing alternatives to finally select the best data pipeline to extract the insight you want. However, the Big Data architecture should be flexible, to admit new pipelines. And also, not all problems are nails for the Big Data hammer, e.g. Reporting Dashboards 7
8 Does Big Data benefit society and people? H1N1 virus pandemic 2012 Earthquake in Mexico Telefonica used mobile data to measure the spread of a global epidemic ( swine flu ) in Mexico DF Dimensioning emergency services in advance for an optimal response to natural disaster situations After the magnitude 7.4 earthquake in Mexico DF, Telefonica researchers captured modile data records that once anonymized and aggregated allowed building visualizations of the density of calls in the differents part of the city, immediately depicting the areas most affected by the earthquake. With Big Data tools like this, it would be possible for authorities to better anticipate contingency plans, dimensioning emergency services and placing them in those points where there is evidence that will be mostly needed in case of catastrophic events. To understand more about human mobility and the spread of epidemics through society, Telefónica Digital s research team used anonymised and aggregated mobile phone call records to measure numbers of people visiting locations such as airports or universities. The study found successful Mexican Government s decision to shut down key infrastructures, reducing virus propagation by 10%. (Click images for more) (Click images for more) 8
9 How does Big Data impact people s privacy? Bliss Customers largely unaware of what happens with their data, mixed messages (spy stories) Trade Customers make an explicit trade-off for each service Drive? Are customers wanting organizations to use their (personal) data to improve their lives? Key evolving data concepts: Individual, aggregated, anonymized Customer consent (ex/implicit opt-in/out) Legal accepted by society 9
10 Security and Privacy around the new black gold Data might be flowing at some source... but what are the rules to tap into it? On one side, aggregated data has been used for decades (census statistics). On the other side, explicit consent is typically requested by online services (you name it). Data Protection Data Security Data Quality Census-like (anonymous and aggregated) Twitter-like (individual consent a.k.a. user opt-in) Security Requirements (100s) COUNTRY REGULATOR OK Data Quality DOUBLES the work needed vs non-data-centric SW apps 10
11 Why is Big Data relevant for businesses? Different business models, with different maturities, and different risk levels Improve services Mobile advertising Sell access to insights Become a gatekeeper of personal data Data = improved business Data = better advertising Data = business Data = risk =? business Leverage data to understand and improve business (x/up sell, churn) and products Leverage data for targeting users with relevant ads and higher CTR and conversion Insights that help improve businesses and governments Recognize that digital data is delicate (privacy) Turn that into an opportunity M2M Smart cities 11
12 What is Telefonica doing to benefit from Big Data? Smartsteps for Retail, Transport, (dynamicinsights.telefonica.com) Footfall count: Shows the trend in footfall in a specified area hourly, daily, weekly and monthly. Provides a basic profile of the crowd. Catchment area: Shows which postal sectors are your customers coming from by hour, day, week and month. Shows the battleground for two sites. Transport mode: Shows flows of crowds from any two points, segmented by road, air, train, etc. 12
For Big Data Analytics There s No Such Thing as Too Big The Compelling Economics and Technology of Big Data Computing March 2012 By: 4syth.com Emerging big data thought leaders Forsyth Communications 2012.
PREPARED STATEMENT OF ROBERT M. GROVES DIRECTOR U.S. CENSUS BUREAU Census: Planning Ahead for 2020 Before the Subcommittee on Federal Financial Management, Government Information, Federal Services, and
BIG DATA IN LOGISTICS A DHL perspective on how to move beyond the hype December 2013 Powered by Solutions & Innovation: Trend Research PUBLISHER DHL Customer Solutions & Innovation Represented by Martin
1 Contents Introduction. 1 View Point Phil Shelley, CTO, Sears Holdings Making it Real Industry Use Cases Retail Extreme Personalization. 6 Airlines Smart Pricing. 9 Auto Warranty and Insurance Efficiency.
At the Big Data Crossroads: turning towards a smarter travel experience Thomas H. Davenport Visiting Professor at Harvard Business School Amadeus IT Group is committed to minimizing its carbon footprint.
DATA FOR GOOD How big and open data can be used for the common good Edited by Peter Baeck February 2015 2 SUMMARY: THE GEOGRAPHY OF THE UK S CREATIVE AND HIGH TECH ECONOMIES Nesta is an innovation charity
The Massachusetts Open Cloud (MOC) October 11, 2012 Abstract The Massachusetts open cloud is a new non-profit open public cloud that will be hosted (primarily) at the MGHPCC data center. Its mission is
ILM Level 3 Qualifications in Leadership and Management Candidate Handbook 2 Background to ILM The Institute of Leadership & Management (ILM) is Europe s largest independent Leadership and Management Awarding
A HARVARD BUSINESS REVIEW ANALYTIC SERVICES REPORT INTERNET OF THINGS: SCIENCE FICTION OR BUSINESS FACT? Copyright 2014 Harvard Business School Publishing. sponsored by SPONSOR PERSPECTIVE Each year the
April 2013 Operational Intelligence: What It Is and Why You Need It Now Sponsored by Splunk Contents Introduction 1 What Is Operational Intelligence? 1 Trends Driving the Need for Operational Intelligence
Notes: - All dollars in this publication denote U.S. dollars unless otherwise stipulated. - Travel manager and travel buyer are used interchangeably to refer to any manager from any department responsible
E-PAPER March 2014 Big Data & the Cloud: The Sum Is Greater Than the Parts Learn how to accelerate your move to the cloud and use big data to discover new hidden value for your business and your users.
Big Data Getting Value from Big Data: Focus on the Opportunities, Not the Obstacles Table of Contents 2 Embark on Your Big Data Journey with Confidence Getting Started, Keeping Moving 3 Big Data Hype Versus
Q&A: Esri's Jack Dangermond on cloud, big data and Apple vs Google map wars The company today unveiled ArcGIS Online organizational subscriptions Sharon Machlis June 14, 2012 (Computerworld) GIS pioneer
A Forrester Consulting Thought Leadership Paper Commissioned By SAP Real-Time Data Management Delivers Faster Insights, Extreme Transaction Processing, And Competitive Advantage June 2013 Table Of Contents
PHONE CALLS ARE THE NEW CLICK: The True SMB Marketing Opportunity April 2014 Contents Executive Summary... 1 The Age of Smartphones... 3 Local Calls to Action... 6 Calls Calls Calls... 7 SMBs Like It Too...
CIO Roundtable - Big March 13, 2013 Big and its Dimensions Big refers to internal and external data that is multi-structured, generated from diverse sources in near real-time and in large volumes making
Analysis of Data Virtualization & Enterprise Data Standardization in Business Intelligence Laljo John Pullokkaran Working Paper CISL# 2013-10 May 2013 Composite Information Systems Laboratory (CISL) Sloan
White paper The future of Service Desks - vision Service Desks require strategic consideration and innovation to raise user productivity and to support business goals. Fujitsu has the experience and feedback
An Oracle White Paper June 2013 Oracle: Big Data for the Enterprise Executive Summary... 2 Introduction... 3 Defining Big Data... 3 The Importance of Big Data... 4 Building a Big Data Platform... 5 Infrastructure