Big Data Analytics. Lucas Rego Drumond

Transcription

1 Big Data Analytics Lucas Rego Drumond Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany Distributed File Systems and NoSQL Database Distributed File Systems and NoSQL Database 1 / 31

2 Outline 2. NoSQL DataBases Distributed File Systems and NoSQL Database 1 / 31

3 Outline 2. NoSQL DataBases Distributed File Systems and NoSQL Database 1 / 31

4 Why do we need a Distributed File System? Distributed File Systems and NoSQL Database 1 / 31

5 Why do we need a Distributed File System? Distributed File Systems and NoSQL Database 2 / 31

6 Why do we need a Distributed File System? Distributed File Systems and NoSQL Database 3 / 31

7 Why do we need a Distributed File System? Read??? - Whole File? - Specific part? Distributed File Systems and NoSQL Database 4 / 31

8 Why do we need a Distributed File System? Write??? - Append to the end of the file? - Insert content in the middle? Distributed File Systems and NoSQL Database 5 / 31

9 Why do we need a Distributed File System? We want to: Perform multiple parallel reads and writes Have the files available even if one computer crashes (replication) Hide parallelization and distribution details Distributed File Systems and NoSQL Database 6 / 31

10 What is a Distributed File System? File Namespace / /home /home/lucas /home/lucas/big_file Distributed File Systems and NoSQL Database 7 / 31

11 What is a Distributed File System? File Namespace / /home /home/john /home/john/big_file Distributed File Systems and NoSQL Database 8 / 31

12 Examples GFS (Google Inc.) HDFS (Apache Software Foundation) Ceph (Inktank, Red Hat) MooseFS (Core Technology / Gemius) Windows Distributed File System (DFS) (Microsoft) FhGFS (Fraunhofer) GlusterFS (Red Hat) Lustre Ibrix Distributed File Systems and NoSQL Database 9 / 31

13 Components A typical distributed filesystem contains the following components Clients - they do the interface with the user Chunk nodes - stores chunks of files Master node - stores which parts of each file are on which chunk node Distributed File Systems and NoSQL Database 10 / 31

14 Distributed File Systems The Google File System Architecture Distributed File Systems and NoSQL Database 11 / 31

15 Distributed File Systems - Storing files Master node / /home /home/john /home/john/big_file /home/john/big_file Chunk 1 C1 C7 Chunk 2 C3 C5 Chunk 3 C4 C6 Chunk 4 C2 C8 Chunk 4 Chunk 1 Chunk 2 Chunk 3 C1 C2 C3 C4 C5 C6 C7 C8 Distributed File Systems and NoSQL Database 12 / 31

16 Read Example Client Application 1. read(/home/john/big_file, chunk 1) 2. (Chunk 1 handle, {C1, C7}) Master node / /home /home/john /home/john/big_file /home/john/big_file Chunk 1 C1 C7 Chunk 2 C3 C5 Chunk 3 C4 C6 Chunk 4 C2 C8 3. (Chunk 1 handle, byte range) C1 C2 C3 C4 C5 C6 C7 C8 4. Chunk 1 data Distributed File Systems and NoSQL Database 13 / 31

17 Write Example Make sure each replica contains the same data all the time One replica is designated to be the primary replica Master pings the nodes to make sure they are alive Distributed File Systems and NoSQL Database 14 / 31

18 Write Example 1. write(/home/john/big_file, chunk 1) Master node / Chunk 1 /home/john/big_file C1 C7 Client Application 2. (Chunk 1 handle, {C1, C7}) /home /home/john Chunk 2 Chunk 3 C3 C5 C4 C6 /home/john/big_file Chunk 4 C2 C8 3. (Chunk 1 handle, data) C1 C2 C3 C4 C5 C6 C7 C8 6. done 4. (Chunk 1 handle, offset) 5. Return status (success or failure) Distributed File Systems and NoSQL Database 15 / 31

19 Considerations Reads are very efficient operations Writes are efficient if they are appends to the end of the file Write in the middle of a file can be problematic Primary replica decides the order in which to make writes: Data is always consistent in all replicas Distributed File Systems and NoSQL Database 16 / 31

20 GFS vs. HDFS HDFS GFS Chunk Size 128Mb 64Mb Default replicas 2 Files (data and 3 Chunknodes generation stamp) Master NameNode GFS Master Chunk Nodes DataNode Chunk Server Distributed File Systems and NoSQL Database 17 / 31

21 Google File System Distributed File Systems and NoSQL Database 18 / 31

22 Hadoop Distributed File System Distributed File Systems and NoSQL Database 19 / 31

23 2. NoSQL DataBases Outline 2. NoSQL DataBases Distributed File Systems and NoSQL Database 20 / 31

24 2. NoSQL DataBases Databases for Big Data: NoSQL NoSQL: Not only SQL Wide variety of database technologies addressing: Non-relational Distributed storing and processing Dynamic Schema Horizontal Scalability Distributed File Systems and NoSQL Database 20 / 31

25 2. NoSQL DataBases Relational vs NoSQL Databases Relational Databases Structured Data: Relational Tables Vertical Scaling ACID Atomic transaction More Functionality Less Scalability NoSQL Databases Structured and Unstructured Data: Collections Horizontal Scaling BASE Eventual Consistency Less Functionality More Scalability Distributed File Systems and NoSQL Database 21 / 31

26 2. NoSQL DataBases Types of NoSQL Databases Distributed File Systems and NoSQL Database 22 / 31

27 2. NoSQL DataBases Graph databases A graph database is a database that uses graph structures with nodes, edges, and properties to represent and store data. Distributed File Systems and NoSQL Database 23 / 31

28 2. NoSQL DataBases Graph databases Compared with relational databases, graph databases are often faster for associative data sets They map more directly to the structure of object-oriented applications. As they depend less on a rigid schema, they are more suitable to manage ad hoc and changing data with evolving schemas. Graph databases are a powerful tool for graph-like queries. Distributed File Systems and NoSQL Database 24 / 31

29 2. NoSQL DataBases Graph queries Reachability queries Shortest path queries Pattern queries Distributed File Systems and NoSQL Database 25 / 31

30 2. NoSQL DataBases Column Databases Column databases stores a tuples consisting of three elements: Unique name: Used to reference the column. Value: The content of the column. Timestamp: The system timestamp used to determine the valid content. Main Advantage: allows to efficiently add new information about existing entities Distributed File Systems and NoSQL Database 26 / 31

31 2. NoSQL DataBases Example { } street: name: street, value: 1234 x street, timestamp: , city: name: city, value: san francisco, timestamp: , zip: name: zip, value: 94107, timestamp: , Distributed File Systems and NoSQL Database 27 / 31

32 2. NoSQL DataBases Document Databases Designed for storing, retrieving, and managing document-oriented information. In contrast to relational databases and their notions of Relations (or Tables ), these systems are designed around an abstract notion of a Document. Documents inside a document-oriented database are not required to have all the same sections, slots, parts, or keys. Documents are addressed in the database via a unique key that represents that document. Distributed File Systems and NoSQL Database 28 / 31

33 2. NoSQL DataBases Example 1 { } FirstName: Bob, Address: 5 Oak St., Hobby: sailing Distributed File Systems and NoSQL Database 29 / 31

34 2. NoSQL DataBases Example 2 { } FirstName: Jonathan, Address: 15 Wanamassa Point Road, Children: { Name: Michael, Age: 10, Name: Jennifer, Age: 8, Name: Samantha, Age: 5, Name: Elena, Age: 2 } Distributed File Systems and NoSQL Database 30 / 31

35 2. NoSQL DataBases Key Value stores Key Value stores use the associative array as their fundamental data model. In this model, data is represented as a collection of key value pairs. The key value model is one of the simplest non-trivial data models. Example: { Great Expectations : John, Pride and Prejudice : Alice, Wuthering Heights : Alice } Distributed File Systems and NoSQL Database 31 / 31