Big Data Management. Big Data Management. (BDM) Autumn Povl Koch September 2,

Transcription

1 Big Data Management Big Data Management (BDM) Autumn 2013 Povl Koch September 2,

2 Overview Today s program 1. Little more practical details about this course 2. Chapter 2 & 3 in NoSQL Distilled 3. Selection of data set 1 (DS1) exercise 1 4. Walkthrough of exercise 2 (storage technologies) 5. New exercise

3 Part 1: Practical details Little more practical details about this course

4 Course Homepage ITU Intranet Course announcements

5 Teaching Assistants Two teaching assistants for now André Aike Baars Ashley Philip Davison-White

6 Course overview Only preliminary for next 4 weeks Lecture Topics covered Litterature 1 Aug Sep. 2 3 Sep. 9 4 Sep. 16 Overview of course. Course details. Big Data use cases. Data Centers. Relational vs. Nonrelational. Exercise 1: Research open datasets Exercise 2: Storage technologies Aggregate data models, graph databases, differences from relational. Selection of Data Set 1 (DS1). Exercise 3: Experiments with DS1. Distribution models, consistency, version stamps. Exercise 4: More experiments with DS1 Map-Reduce Exercise 5: Map-Reduce on DS1 NoSQL Distilled chapter 1 NoSQL Distilled chapter 2-3 NoSQL Distilled chapter 4-6 NoSQL Distilled chapter

7 Course overview Only preliminary for next 4 weeks Lecture Topics covered Litterature 5 Sep. 23 Key-Value Stores Exercise 6: Experiement with Key-Values Exercise 7: Data Set 2 NoSQL Distilled chapter

8 Part 2: NoSQL Distilled Chapters 2 & 3 NoSQL Distilled Chapters 2 &

9 Relational Data Model Relational Data Model Tuples/rows (simple types) Relation

10 Aggregate Data Models Domain-Driven Design s AGGREGATE data models Collection of related objects Complex data types: lists, etc Treated as a unit for data manipulation Unit for consistency, updates with atomic opreations Easier to handle in cluster Natural unit for replication Easier for application programmers

11 Relational model Example of e-commerce order in UML NoSQL Distilled p

12 Relational model Example of e-commerce order NoSQL Distilled p

13 Aggregate data model Example of e-commerce order in aggregate model JSON notation NoSQL Distilled p

14 Aggregate data model Example of e-commerce order in aggregate model NoSQL Distilled p

15 Aggregate orientation Consequences of aggregate orientation Choice of aggregates is crucial to how easy it is to access data later Aggregate-ignorant databases makes it easy to look at data in different ways Aggregate-aware databases makes data easier to distribute in a cluster Transactions is still import to keep consistency In relational database model, ACID may span multipe tables, so can be difficult in cluster environment Consistency more easy in aggregate data model, still ACID properties in some models per aggregate 9/1/

16 Document Databases Document database Document databases can be quieried based on fields, a key or any other field Example of document database: MongoDB Key-Value databases only hold a key and its associated value order_1_name -> Martin order_99_customer -> 1 NoSQL Distilled p

17 Column-Family Store Structure in Column-Family store (inspired by BigTable ) Accessed together Customer-ID 1234 Do not think of this as row but as aggregate Difficult question: how much was sold over last two weeks?

18 Graph Databases Example of graph databases Edges Node Small records with complex interconnections Example: Neo4j Example query: What does Anna and Barbara both like?

19 Schemaless Databases Schema vs. schemaless Schema-based databases Easy to format data for reports, web presentation Difficult to accomodate new data, SQL actually allows for change of tables Schemaless databases (key-value, document, column-store, graph) Easy to add new data, best for non-uniform data But formatting becomes more difficult, there is an implicit schema anyway Can be very difficult to deal with both old data and new data

20 Data access Modelling/optimizing for data access

21 Column Store vs. Graph versions Column Store version Graph version

22 Part 3: selection of data set 1 (DS1) Selection of data set 1 (DS1) exercise

23 Received proposals Received student proposals 1. Instagram 2. Crime of Chicago 3. OpenStreetMap 4. Transport for London 3.aspx# genomes

24 Received proposals Cont. Any kind of weather dataset 6. Datasift social data multiple datasets 7. Facebook stream e/ 8. Twitter 9. Sloan Digital Sky Survey GitHub Archive

25 Received proposals Cont Wikipedia, esp. political purposes Airline ticket prices

26 Proposal 1 Instragram

27 Proposal 2 Crime of Chicago Safety/Crimes-2001-to-present/ijzp-q8t

28 Proposal 3 OpenStreetMap

29 Proposal 4 Transport for London

30 Proposal genomes

31 Proposal 6 Datasift social data

32 Proposal 7 Facebook

33 Proposal 8 Twitter

34 Proposal 9 Sloan Digital Sky Survey

35 Proposal 10 GitHub Archive

36 Proposal 11 Wikipedia

37 Short list Mostly likely to succeed Instagram Transport for London Twitter GitHub Selected data set:?

38 Part 4: Feedback on exercise 2 Storage technologies some takeaways Big data in the size of 128 petabytes is expensive and requires significant amount of space and power Some details compared to 7,200 RPM hard drive 15,000 RPM drives, 10x more expensive, low capacity SATA SSD is 10x more expensive, compact, OK performance and low power PCIe SSD drive, high end, is very fast, but also very costly, 150x SD cards very compact, but slow

39 Exercise 2: Hard disk vs. Solid State Drives

40 Exercise for today First data set exercise

41 Excercise 3: Analysis of Data Set 1 Quick analysis of Data Set 1 (DS1) Your CEO has now changed focus from building the 128 petabytes datacenter due to budget constraints. Instead, the CEO asks you to analyze further the data set that the company s Advisory Group (the BDM class) has selected today. Please write up a new recommendation for the CEO about: - what is the specific big data characteristics of the DS1? - what is the data structure? and will it fit well with relational, key-value, column store, or graph database systems? (overall recommendation is OK) - what other data sources would be relevant to combine DS1 with?