BIG Big Data Public Private Forum

Transcription

1 DATA STORAGE Martin Strohbach, AGT International (R&D)

2 THE DATA VALUE CHAIN Value Chain Data Acquisition Data Analysis Data Curation Data Storage Data Usage Structured data Unstructured data Event processing Sensor networks Streams Multimodality Data preprocessing Semantic analysis Sentiment analysis Data correlation Pattern recognition Realtime analysis Machine learning Trust Provenance Data augmentation Annotation Data validation Redundancy elimination Keep up-to-date Consistency NoSQL NewSQL In-Memory Databases Cloud storage Query interfaces Fast access Integrated Analytics Data Models Consistency Secure Storage Decision support Predictions Simulation Exploration Modelling Control Domainspecific usage Technical Working Groups 2

3 DIGITAL DATA GROWTH 7.9 ZB equalling 1 TB or 210 single sided DVDs for each person in the world Digital content created Available Storage 2.5 Exabytes every day Every two days we create as much information as we did from dawn of mankind through 2013 IDC estimates that in 2009 it would not have been possible to store all created data How can we efficiently store and make the data available for further analysis?

4 DATA STORAGE AS A COMMODITY Big Data Problems are not new Big Data technologies have become commodities It s now easier and more cost-efficient to manage Big Data sets 4

5 EXAMPLES OF DATA PROCESSING Unstructured data MapReduce Structured Data Smart meter data Source: SELECT house_id FROM readings WHERE voltage > OR WHERE voltage < (house_id,voltage, ) Simple example of a energy quality query 5

6 VALUE OF DATA STORAGE Yesterday Today Keep only the necessary data Hard to scale Schema-on write Consistency Paradigm shift: Keep all the data Elasticity Schema-on-read Eventually consistent 6

7 If Big Data Storage Is a commodity WHAT CHALLENGES REMAIN?

8 VOLUME CHALLENGE Adapted from 8

9 ENTERPRISE AMNESIA > 1.8 ZettaBytes 1 What information can be extracted from 100,000+ safety and security cameras that generate petabytes of raw video data and terabytes of extracted video analytics features?

10 ANALYSIS OF DATA VOLUMES Towards Integrated Analytics DBMS Analytics Data Management Analytical Databases Separate Systems for Data Managent and Complex Analytics Different data models May negatively impact performances Analytics executed in the same system Single data model Potentially higher performance 10

11 VARIETY: INTERNET OF THINGS (IOT) >50 billion connected devices in 2020 Cisco Internet of Everything (IoE) Survey: Trillion dollar market for the next 10 years Cross-sectorial use of data Implications for Big Data storage Increased growth of structured data Requires data models that support analysis over spatio-temporal properties and relationships to other real-world entities Heterogeneous nature of data and technologies asks for implementaion of a data virtualization layer 11

12 VARIETY CHALLENGE NoSQL DBMS are highly scalable, but lack standardized interfaces 12

13 TECHNOLOGY SURVEY TWG has surveyed 18 storage technologies Selected based on popularity 21 criteria, e.g. Best used for (e.g. batch processing, high volume write access, low latency queries, ) High availability (yes/no) Support for declarative queries (yes/no, if yes, which?) Schema support (yes/no) Maturity (Good, Bad, medium) License (name of license) Will be included in 2nd Version of Technical Whitepaper CouchDB [3] Neo4J [9] InfiniteGraph [14] Dex [15] AllegroGraph [1] Bigdata [16] Virtuoso [17] Data Model Document Graph Graph Graph RDF/graph RDF Relational + Document + RDF Best used Medium size graphs Big size graphs Medium size graphs Graph and KB related problems Graph and KB related problems Graph and KB related problems Scalability High Restricted High High High High High High availability Yes Restricted Yes Yes Yes Sequential or eventual Sequential Sequential Sequential Sequential Sequential Eventual Sequential consistency Is Partition tolerant no no No Yes No Latency Low Low Low Low transactions MVCC ACID ACID to relax ACID MVCC ACID 13

14 DATA STORAGE USE CASES Media Energy Transport Public Sector 1. Overcoming the data silo culture Health 1. Semantic Data Integration and Storage: 2. Protecting citizens privacy 2. Data Security and Privacy security best practice for non-relational data stores secure data storage and transaction logs 14

15 ENERGY, STANDARDS AND NEWSQL Tenfold data growth impacts power consumption Standards for nosql technologies are missing NewSQL databases my provide alternative for nosql stores Modern form of relational databases aiming at comparable scalability while maintaining transactional guarantees made by traditional DBs Source: IDC's Digital Universe Study, sponsored by EMC, March 2008)

16 SUMMARY Big Data Storage has become commodity business Facilitates the storage and analysis of raw data Open Challenges Scalability relates to complexit of data models used Seamless integration and efficient performace of data analytics IoT: spatio-temporal properties and relationships to other real-world entities, heterogeneous data Surveyed 18 storage technologies (to come) Use Cases hint towards Cross Sectorial Storage Challenges Volume: necessity to scale out Variety: Heterogeneity of data sources Velocity: Low latency queries CouchDB [3] Neo4J [9] InfiniteGraph [14] Dex [15] AllegroGraph [1] Data Model Document Graph Graph Graph RDF/graph Best used Medium size graphs Big size graphs Medium size graphs Graph and KB related problems Scalability High Restricted High High High High availability Yes Restricted Yes Sequential or eventual Sequential Sequential Sequential Sequential Sequential consistency Is Partition tolerant no no No Latency Low Low transactions MVCC ACID ACID to relax ACID 16

17 QUESTIONS THANK YOU? Dr. Martin Strohbach Senior Reseracher, Big Data AGT International