Big Data Technologies. Prof. Dr. Uta Störl Hochschule Darmstadt Fachbereich Informatik Sommersemester 2015

Size: px

Start display at page:

Download "Big Data Technologies. Prof. Dr. Uta Störl Hochschule Darmstadt Fachbereich Informatik Sommersemester 2015"

Christina Carter
8 years ago
Views:

1 Big Data Technologies Prof. Dr. Uta Störl Hochschule Darmstadt Fachbereich Informatik Sommersemester 2015

2 Situation: Bigger and Bigger Volumes of Data Big Data Use Cases Log Analytics (Web Logs, Sensor Logs, Event Logs etc.) RFID Tracking and Analytics Fraud and Risk Management Churn Management Gaming Data Streaming Data Social Network Data Volumes of Data Eric Schmidt (Google CEO, 2010): 5 exabytes of information created between the dawn of civilization through 2003 now created every 2 days, and the pace is increasing. h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

Volumes of Data Eric Schmidt (Google CEO, 2010): 5 exabytes of information created between the dawn of civilization through

3 Characteristics of Big Data Source: L. Haas, IBM h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

4 Emerging Technologies Hype Cycle 2012 Source: Gartner, July 2012 h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

5 Emerging Technologies Hype Cycle 2013 Source: Gartner, July 2013 h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

6 Emerging Technologies Hype Cycle 2014 Source: Gartner, August 2014 h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

7 Scale up vs. Scale out Scale up: big single node system Scale out: many small nodes Source: ibm.com Source: eggmusic.com h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

8 Scale up vs. Scale out Scale up (scale vertically) Advantages: Transparent for database system Management complexity constant Disadvantages: Expensive hardware Limited scalability Scale out (scale horizontally) Advantages: Less expensive hardware High scalability Disadvantages: Load balancing necessary Data distribution necessary Increased error rate Increased management complexity Distributed lock and replication protocols h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

Disadvantages: Expensive hardware Limited scalability Scale out (scale horizontally) Advantages: Less expensive hardware High

9 NoSQL Database Systems: Idea Need for databases being distributed and horizontally scalable. NoSQL Definition: Next Generation Databases mostly addressing some of the points: being non-relational, distributed, open-source and horizontally scalable. The original intention has been modern web-scale databases. The movement began early 2009 and is growing rapidly. Often more characteristics apply such as: schema-free, easy replication support, simple API, eventually consistent / BASE (not ACID), a huge amount of data and more. So the misleading term "nosql" (the community now translates it mostly with "not only sql") should be seen as an alias to something like the definition above. Source: S. Edlich, nosql-database.org Detailed discussion in chapter NoSQL Database Systems h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

The original intention has been modern web-scale databases. The movement began early 2009 and is growing rapidly.

10 NoSQL Database Systems: Classification 1(2) Key-Value Database Systems Key-value pairs Systems: Oracle Berkeley DB, Amazon Dynamo/S3, Redis, Riak, Voldemort, key key key key key value value value value value Document Store Database Systems Key-value pairs with documents as value Document format: JSON, BSON Systems: MongoDB, CouchDB, Couchbase, { } "id": 1, "name": football boot", "price": 199, "stock": { "warehouse": 120, "retail": 10 } h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

$boot", "price": 199, "stock": { "warehouse": 120, "retail": 10 } h_da Prof. Dr.$

11 NoSQL Database Systems: Classification 2(2) Column Familiy Database Systems (Wide Column Stores) Data loosely structured by column families and columns Systems: Google BigTable, HBase, Cassandra, Amazon SimpleDB, Row Key Time Stamp ColumnFamily contents: ColumnFamily anchor "com.cnn.www" t9 anchor:cnnsi.com "CNN" t8 anchor:my.look.ch "CNN.com" t6 t5 "<html> " "<html> " (Graph Database Systems) Graph structures with nodes, edges, and properties Systems: Neo4J, Infinite Graph Source: h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

12 NoSQL Database Systems: Topics of Interest Categorization Data Modeling Application Development Scalability, Availability and Consistency Select the Right DBMS h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

Availability and Consistency Select the Right DBMS h_da

13 Big Data: Other Approaches Will everything be NoSQL in the future? Definitely not! h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

14 Column Store Database Systems: Idea Goal: Reduce the number of disc access / amount of data to read + easy to insert/update a record + only need to read in relevant data might read in unnecessary data + higher compression ratio insert/update require multiple accesses expensive reads on entire records suitable for read-mostly, read-intensive, large data repositories (OLAP) Source: Harizopoulos/Abadi/Boncz: VLDB2009 h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

require multiple accesses expensive reads on entire records suitable for read-mostly, read-intensive, large data

15 Storage Layout Column Store Database Systems: Topics of Interest Columnar Storage Compression Execution Engine Operating on columns (insert / update / read) Operating on compressed data (early vs. late materialization) h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

update / read) Operating on compressed data (early vs.

16 In-Memory Database Systems: Idea Traditionell OLTP & OLAP OLTP ETL OLAP Vision: Information in Real Time: Anything, Anytime, Anywhere OLTP & Real Time OLAP Hardware Trends Steadily declining memory price Many-core processor In-Memory Database Management Systems h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

Trends Steadily declining memory price Many-core processor In-Memory Database

17 In-Memory Database Systems: Topics of Interest In-Memory DBMS Disk-Based DBMS + Tons of Memory Architecture and Implementation Storage Layout: Row, Column, Hybrid? Persistence Design of operations Application Development h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

18 Emerging Technologies Hype Cycle 2014 Source: Gartner, August 2014 h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

19 Course Goals Introduction to the big data technology landscape NoSQL database systems (distributed and horizontally scalable non-relational database systems) Column store database systems In-memory database systems h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

database systems) Column store database systems In-memory database

20 Literature Tiwari, S. Professional NoSQL, Wiley, 2011 Sadalage, P. J., Fowler, M. NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence, Addison Wesley, 2012 Edlich, S., Friedland, A., Hampe, J., Brauer, B., Brückner, M. NoSQL Einstieg in die Welt nichtrelationaler Web 2.0 Datenbanken, Carl Hanser Verlag, 2011 (2 nd ed.) Kudraß, Th. (Ed.), Taschenbuch Datenbanken, Störl, U.: Kap. 12 NoSQL- Datenbanksysteme, Ritter, N.: Kap. 13 Verteilte und föderierte Datenbanksysteme, Carl Hanser Verlag, 2015 (2 nd ed.) Abadi, D., Boncz, P., Harizopoulus, S, Idreos, S., Madden, S. The design and implementation of modern column-oriented database, Transatlantic Publishers, 2013 Harizopoulus, S., Abadi, D., Boncz, P. Column-Oriented Database Systems, VLDB Tutorial, Plattner, H., Zeier, A. In-memory data management: technology and applications, Springer Verlag, 2012 (2 nd ed.) h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

), Taschenbuch Datenbanken, Störl, U.: Kap. 12 NoSQL- Datenbanksysteme, Ritter, N.: Kap. 13 Verteilte und föderierte Datenbanksysteme, Carl Hanser Verlag, 2015 (2 nd ed.) Abadi, D., Boncz, P.

21 Big Data Technologies Introduction NoSQL Database Systems Column Store Database Systems In-Memory Database Systems h_da Prof. Dr. Uta Störl Big Data Technologies: Introduction - SoSe

NOSQL DATABASE SYSTEMS

NOSQL DATABASE SYSTEMS Big Data Technologies: NoSQL DBMS - SoSe 2015 1 Categorization NoSQL Data Model Storage Layout Query Models Solution Architectures NoSQL Database Systems Data Modeling id ti Application