Vivir en un mar de Datos 2015: Big Data una mirada Global Fundación Telefónica

Transcription

1 RETHINK big Project Consuelo GONZALO MARTÍN UNIVERSIDAD POLITÉCNICA DE MADRID 24 March 2015 Vivir en un mar de Datos 2015: Big Data una mirada Global Fundación Telefónica This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement no

2 MIDAS expertise 20+ years in Data Value Chain: collection, analysis, knowledge extraction Multiple-source data integration and analysis Data Mining on text, image and structured data Data Mining on streaming data High Performance Data Analysis Numerical and agent-based simulations Large scale Heuristic optimization Complex data interaction and visualization Funding: FP7, private (industry) 2 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

3 Medical information Systems: Prediction of patient recovery Early detection of mental decay Pharma applications Drug NSLC effectiveness Pharmaeconomics Mining the IoT: Context aware recommender based on social network analysis Mining portable and wearable monitoring MIDAS technology 3 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica 3

4 MIDAS Project Design of an analytical platform to "monetize electronic medical data EMH Images Genomics Other data (demographic, geographic ) Emergencies: Re-admissions Prioritization of Rx Nuclear Medicine: Automatic identification of tumors Basic research: MEG data (Alzheimer) 4 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

5 Rethink big Project Overview INDUSTRY-DRIVEN The Project: Coordination and Support Action (CSA), 2-year. Coordinated by BSC, Start: 1 Mar 2014 The Mission: To deliver a strategic roadmap for how European technology advancements in hardware, networking and algorithms can be exploited for Big Data analytics, in the next 10 years. 5 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

6 The Partners Rethink big Project Overview 6 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

7 Motivation Big Data a fast growing market with impact on diverse sectors Big Data market is growing six times faster than the overall ICT market (source IDC) Big Data is becoming a key economic asset: Big Data is the new oil (EU N. Kroes) 40,0 30,0 20,0 10,0 0,0 World Wide Big Data Market Forecast EUR Billion Sectors/Domains Public administration Healthcare & Social Care Utilities Transport and Logistics Retail & Trade Geospatial Applications & Services Big Data Value EUR 150 billion to EUR 300 billion in new value (Considering EU 23 larger governments) EUR 90 billion considering only the reduction of national healthcare expenditure in the EU Reduce CO2 emissions by more than 2 gigatonnes, equivalent to EUR 79 billion USD 500 billion in value worldwide in the form of time and fuel savings, or 380 megatonnes of CO2 emissions saved 60% potential increase in retailers operating margins possible with Big Data USD 800 billion in revenue to service providers and value to consumer and business end users USD 51 billion worldwide directly associated to Big Data market (Services and applications) 7 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

8 Motivation Ensure Europe s leading role in the datadriven world addressing competitiveness, innovation, and society covering the all aspects of Big Data Value Skills Social Legal Data Business Technical Application 8 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

9 Big Data Definition 9 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

10 Challenges 10 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

11 Challenges Work with different requirements Velocity Volume Variety Real Time Sensors Power consumption 11 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

12 Challenges Work with different areas Software Tools Systems Network Hardware Applications and end users 12 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

13 Application Challenges Science and Engineering Applications Life Sciences Future Internet and Social Networking Business, Finance, Information Marketplaces 13 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

14 Key: Hardware/Software Holistic Design Hardware needs to be software-aware Software needs to be hardware-aware 14 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

15 What happens if HW does not consider SW Many (supposedly great) changes in HW architecture do not survive Cell processor (Playstation 3 processor) Master-Slave processor model programmed using DMAs -> Extremely difficult for programmers Itanium processor (VLIW) Very Long instruction word explicitly harnesses instruction level parallelism through Compiler -> Compilers could not extract required parallelism 15 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

16 What happens if SW does not consider HW Terasort contest: sorting 100TB data Number 1: Vanilla Hadoop 2100 nodes, 12 cores per node, 64 Gb per node cores 134 Tb memory Vanilla Time: 4300 Hadoop segs is easy to program, but needs Cost 57X in Amazon: more cores, $ X more memory, Number 2: Tritonsort and only gets 2X performance 52 nodes, 8 cores per node, 24 Gb 416 cores 1,2 Tb memory Time: 8300 secs and 6400 secs Cost in Amazon: $ 294 and Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

17 Enabling Technologies Conventional / Unconventional HW and processing technology Distributed Architectures, Devices and Sensors, Memory and Storage Networks Frameworks, SW Models, Algorithms, Data Stuctures and Visualization 17 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

18 Rethink big Methodology Identification of European Big Data Competencies Review& First Working Group Meeting Refine a group of technology and bussines experts Technical and bussines oriented surveys Interactive Working Group Workshop SWOT Elaboration 18 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

19 First Working Group Meeting: 18,19 Sep 2014 Objectives: Identify challenges across European Big Data sectors, Develop a shared language, Engage key strategists Attendees: 70 Experts from 49 Organizations, 38 External INDUSTRY PARTICIPANTS INCLUDED: Academic 13 Project / Programme 2 Research Institution 6 SME 16 Large Company 12 Users Providers THALES, AIRBUS, Boehringer-Ingleheim, AGT International, Capgemini, Cloud&Heat, The Unbelievable Machine Company, NextWorks ARM, THALES, Alcatel Lucent Bell Labs, Telefonica, T-Systems, Bull, TT Tech, Lacie (Seagate), Kalray, Okkam 19 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

20 20 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

21 Initial Expert List 35 Interest in Participation per Area of Expertise Implicit NO Explicit NO YES Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

22 First Working Group Meeting: 9,10 Dec Objectives: to synthesize findings so far and analyzing the hardware and networking situation for Big Data in Europe Attendees: Around 30 partners and external experts participated from seven European countries, representing both those researching and producing the Big Data infrastructure and those who rely on it for their research or business objectives. Conclusions: While Europe may not be less competitive in software and co-design, it holds a leading position in hardware areas such as embedded systems and device design. Software areas such as algorithms and data analytics, domain-specific expertise were also perceived strengths. Opportunities identified include distributed computing, leveraging datasets and real-time analytics. Europe benefits from strong political leadership in this field and the funding to facilitate scaling, although securing cooperation between its vast patchwork of SMEs may prove challenging. Complex bureaucracy and legal frameworks in Europe mean that other regions may move faster to capitalize on such openings. 22 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

23 RETHINK big Project Big Data Value cppp This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement no

24 BDV cppp Content: Multidisciplinary approach 24 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica

25 Thank you 25 Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica