Big Data Management. Big Data Management. (BDM) Autumn Povl Koch September 30,

Size: px

Start display at page:

Download "Big Data Management. Big Data Management. (BDM) Autumn 2013. Povl Koch September 30, 2013 29-09-2013 1"

Daniel Hines
8 years ago
Views:

1 Big Data Management Big Data Management (BDM) Autumn 2013 Povl Koch September 30,

2 Overview Today s program 1. Little more practical details about this course 2. Recap from last time 3. Graph databases - Chapter 11 in NoSQL Distilled 4. New exercises 7 & 8 5. Results from pre-evaluation

3 Part 1: Practical details Little more practical details about this course

4 Course Homepage ITU Intranet Course announcements Use it for exercises and TA help First dataset selected: GitHub Archive or Instagram The three databases selected: MongoDB - Hadoop - hadoop.apache.org/ Neo4j - Intro article:

selected: GitHub Archive or Instagram The three databases selected: MongoDB - http://www.

5 Teaching Assistants Two teaching assistants for now André Aike Baars Ashley Philip Davison-White

6 Course overview Only preliminary for next 4 weeks Lecture Topics covered Litterature 1 Aug Sep. 2 3 Sep. 9 4 Sep. 16 Overview of course. Course details. Big Data use cases. Data Centers. Relational vs. Nonrelational. Exercise 1: Research open datasets Exercise 2: Storage technologies Aggregate data models, graph databases, differences from relational. Selection of Data Set 1 (DS1). Exercise 3: Experiments with DS1. Distribution models, consistency, version stamps. Exercise 4: More experiments with DS1 MongoDB introduction, basics, and Map- Reduce Exercise 5: Map-Reduce on DS1 NoSQL Distilled chapter 1 NoSQL Distilled chapter 2-3 NoSQL Distilled chapter 4-6 NoSQL Distilled chapter 7 and

Exercise 1: Research open datasets Exercise 2: Storage technologies Aggregate data models, graph databases, differences from relational. Selection of Data Set 1 (DS1).

7 Course overview Only preliminary for next 4 weeks Lecture Topics covered Litterature 5 Sep Sep Oct. 7 Nov. 4 Key-Value Stores Exercise 6: Analysis with Key-Values Graph databases Exercise 7: Experiment graph database Exercise 8: Data Set 2 NoSQL Distilled chapter 8 NoSQL Distilled chapter 11 External lecturer: Hadoop (IBM, Søren Ravn) NoSQL Distilled chapter 10 External lecturer: IBM-Vestas case (IBM, Claus Samuelsen) Hanne Breddam (MD of ProphetAnalytics) will make guest lecture on examples of big data projects Philippe Bonnet (currently at INRIA Paris will join for lecture on security and big data)

8 NoSQL Distilled chapter 11 External lecturer: Hadoop (IBM, Søren Ravn) NoSQL Distilled chapter 10 External lecturer: IBM-Vestas case (IBM, Claus Samuelsen)

8 Part 2: Recap from last time Recap from last time:

9 RECAP: Key-Value Stores Features of Riak Key-Value Stores vs. Relational Consistency Eventual consistency Write conflict resolution: either newest win or both loose setup on bucket creation nval = number of replicas, default 3 ( There are no guarantees that the three replicas will go to three separate physical nodes ) W = Number of nodes to respond to write R = Number of nodes to responds to read

setup on bucket creation nval = number of replicas, default 3 ( There are no guarantees that the three

10 RECAP: Key-Value Stores The Riak Ring Source:

11 RECAP: Key-Value Stores Features of Riak Key-Value Stores vs. Relational Transactions Quorum, W value used for writes E.g., replication factor nval = 5, W=3, 2 nodes can be down Query Basically: simple key search, note: key has to be unique per bucket More advanced, e.g., Riak Seach and MapReduce Data structure Value of a Key-Value pair can be any type, blob, text, JSON, XML, etc Riak uses HTTP POST s MIME type Any limitations?

, replication factor nval = 5, W=3, 2 nodes can be down Query Basically: simple key search, note: key has to be

12 RECAP: Consistency Quorums how many replicas need to acknowledge Replica 1 Replica 2 Replica 3 Write requests Reply Coordinator Reply Write/update request Acknowledgement after X replicas reply Example: Cassandra

Reply Coordinator Reply Write/update request

13 RECAP: Consistency Relaxed consistency Different applications can tolerance different consistencies Some NoSQL systems drop transactions/consistency completely due to performance CAP theorem gives the tradeoffs Consistency You can only have 2 out of 3 Availability Partition tolerance every request received by by a nonfailing node in the system must result in a response [Lynch and Gilbert]

tradeoffs Consistency You can only have 2 out of 3 Availability Partition tolerance every request

14 RECAP: Key-Value Stores Features of Riak Key-Value Stores vs. Relational Scaling CAP theorem -like parameters for Riak, per bucket: N = number of nodes to store the key-value replicas, e.g., 5 R = number of nodes to have the data for the read to be successful, e.g., 2 W = number of nodes the writes has to be successful to, e.g.,

nodes to store the key-value replicas, e.g.

15 Part 3: Graph databases Graph databases: NoSQL Distilled Chapter

16 Graph databases Nodes and edges in a graph database

17 Graph databases Examples of graph databases Neo4j - Neo4j is a robust (fully ACID) transactional property graph database. Due to its graph data model, Neo4j is highly agile and blazing fast. For connected data operations, Neo4j runs a thousand times faster than relational databases. OrientDB Enterprise Graph DB OrientDB is the Graph Database with a hybrid model taken from both the Document Database and Object Orientation worlds. It can work in schema-less mode, schema-full or a mix of both. Supports advanced features such as ACID Transactions, Fast Indexes, Native and SQL queries. It speaks the language of the Web: HTTP and JSON

OrientDB Enterprise Graph DB http://www.orientdb.org/ OrientDB is the Graph Database with a hybrid model taken from both the Document Database and Object Orientation worlds.

18 Graph databases Examples of graph databases FlockDB - Twitter runs FlockDB on a large cluster of machines. we use it to store social graphs (who follows whom, who blocks whom) and secondary indices at twitter. FlockDB is a distributed graph database for storing adjancency lists, with goals of supporting: a high rate of add/update/remove operations potientially complex set arithmetic queries paging through query result sets containing millions of entries ability to "archive" and later restore archived edges horizontal scaling including replication online data migration

FlockDB is a distributed graph database for storing adjancency lists, with goals of supporting: a high rate of add/update/remove operations potientially

19 Graph databases Creation of nodes and relationships Node martin = graphdb.createnode() martin.setproperty( name, Martin ) Node pramod = graphdb.createnode() pramod.setproperty( name, Martin ) martin.createrelationshipto(pramod, FRIEND) pramod.createrelationshipto(martin, FRIEND)

createnode() pramod.setproperty( name, Martin ) martin.

20 Graph databases Relationships with properties (the classical M-M problem)

21 Graph databases Features of Neo4j Consistency Within node, fully ACID consistent Within cluster, write on master propagates to slave nodes Writes to slaves propagates to master, then other slaves Nodes can only be deleted with they have no relationships

22 Graph databases Support of transactions in Neo4j

23 Graph databases Avalability Neo4j uses ZooKeeper for managing master ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services

24 Graph databases Queries and indexes in Neo4j First use an index to find starting node, then traversal Indexing service based on Lucene

25 Graph databases Cypher query language

26 Graph databases Cypher query language 1 level

27 Graph databases Friend of a friend of a friend with SQL Six degrees of seperation?

28 Graph databases MySQL vs. Neo4j friend-of-friend traversal 1000 users Source: Neo4j in action, Jonas Partner, et al

29 Graph databases Large scale example: 1 million users

30 Graph databases Application-level sharding

31 Graph databases Use case examples

32 Graph databases Recommendations

33 Graph databases When not to use Operations on all or subset of entries, e.g., a property

34 Part 4: Exercise 7 & 8 Exercise 7 &

35 Exercise 7 Exercise 7 From a famous big data blog, your CEO has read that graph databases perform very well and he asks you to remodel your complete chosen dataset DS1 (Github Archive or Instagram) Describe in detail how you would layout the data in the graph model, as you did in Exercise 5 & 6 Model 3 of your most used (expected) queries in Cypher For what kind of specific queries to a graph version of your dataset would you expect a system like Neo4j to perform very well?

36 Exercise 8 Exercise 8 We need to choose a second data set

37 Section 5: Pre-evaluation Pre-evaluation

38 Pre-evaluation Opinions: Quick Exercises (Good) Idea behind Data entity processing method is good but need ideas (specs) for Real World applications Discussions good... too early

39 Pre-evaluation Suggestions for Change Show examples of exercises from the week before Cloud service overview and usage Real world service use Using other newer techs Outside speakers, from companies using Nosql In class examples of applications and DB usage standard specs for assignments Links to interesting YT videos More activity on Blog posts Past Exam example

Big Data Management. Big Data Management. (BDM) Autumn 2013. Povl Koch September 16, 2013 15-09-2013 1

Big Data Management. Big Data Management. (BDM) Autumn 2013. Povl Koch September 16, 2013 15-09-2013 1 Big Data Management Big Data Management (BDM) Autumn 2013 Povl Koch September 16, 2013 15-09-2013 1 Overview Today s program 1. Little more practical details about this course 2. Chapter 7 in NoSQL Distilled