Data Management in the Cloud

Transcription

1 With thanks to Michael Grossniklaus! Data Management in the Cloud Lecture 8 Data Models Document: MongoDB I ve failed over and over and over again in my life. And that is why I succeed. Michael Jordan 1

2 Trend Towards Specializa=on Non- Datastores ( NoSQL ) New Genera@on OLAP (ver@ca, aster, greenplum) RDBMS (Oracle, MySQL) MongoDB s defini=on of NoSQL : non- rela@onal, next- genera@on opera@onal datastores and databases 2

3 Horizontally Scalable Architectures No joins and no complex row hard to implement both when scaling horizontally abandon the data model No SQL New data models key/value: memcached, Dynamo document- oriented: MongoDB, CouchDB, JSON stores tabular: BigTable Improved ways to develop easier than a rela@onal database management system data modeling: begin with normalized model and refine model by embedding documents 3

4 Spectrum of Systems memcached scalability and performance key/value MongoDB RDBMS depth of func;onality 4

5 What is missing? joins MongoDB vs. RDBMS complex row SQL (but MongoDB has document- based query language ) Terminology database database or schema table collec@on row document column field Database Collec@on2 Doc1 Doc2 Index Collec@on1 Document1 Field1 Field2 Document2 Document3

6 MongoDB vs. RDBMS databases define columns at the table level whereas a document- oriented database defines its fields at the document level - h]p://openmymind.net/mongodb.pdf (p.5) Table - > Collec@on; Row - > Document Shoes Table Shoes Collec=on Brand Size Color Vans 8 Red/Gold DC 8 White/Blue/Green Nike 8.5 YellowGreen { Brand : Vans, Size : 8, Color : Red/Gold } { Brand : DC, Size : 8, Color : White/Blue/Green } { Brand : Nike, Size : 8.5, Color : YellowGreen } 6

7 MongoDB vs. RDBMS databases define columns at the table level whereas a document- oriented database defines its fields at the document level - h]p://openmymind.net/mongodb.pdf (p.5) Table - > Collec@on; Row - > Document Shoes Table Shoes Collec=on Brand Size Color Vans 8 Red/Gold DC 8 White/Blue/Green Nike 8.5 YellowGreen { Brand : Vans, Size : 8, Color : Red/Gold, Note : Looks like mermaid } { Brand : DC, Size : 8, Color : White/Blue/Green } { Brand : Nike, Size : 8.5, Color : YellowGreen } 7

8 MongoDB Data Model & Features Advantages of document model Documents (objects) correspond to types in many programming languages Embedded documents and arrays reduce need for expensive joins Dynamic schema supports fluent polymorphism High Performance High Availability Scaling 8

9 Data Model MongoDB stores JSON- style documents internally represented as BSON denotes an a<ribute { hello : world } of type string \x16\x00\x00\x00\x02hello\x00 \x06\x00\x00\x00world\x00\x00 light- weight and traversable driver converts programming language objects to BSON document size limited to 4 MB Flexible schemas the big difference between a collec@on and a table { author : fred, text :... } { author : mary, text :..., tags : [ mongodb ]} Source: h<p://bsonspec.org 9

10 World Wide Web scaling out high volume caching Less good at highly ad- hoc business intelligence problems that require SQL Use Cases 10

11 Storing Documents post = {author: mike, date: new Date(), text: my blog post..., tags: [ mongodb, intro ]} JavaScript object database db.posts.save(post) collec;on Example uses JavaScript MongoDB supports JavaScript on the client and server side other language bindings exist BSON is the basis for a rich type system e.g., (binary) date values are not possible in JSON Collec@ons are created implicitly on demand 11

12 Special key _id Document Iden=fiers present in all documents: user or system- defined unique across a collec@on can be any type System- defined default _id is added if not specified by user similar to GUID/UUID, lightweight (only 12 bytes) and fast to generate generated at the client ObjectId( 4bface1a e04f3c434 machine id process id incremen@ng counter Source: h<p:// 12

13 Queries Query invoked by method find() on posts by author db.posts.find({author: mike }) query- by- example: problems? Query language query- by- example plus $ modifiers : $gt, $lt, $gte, $lte, $ne, $all, $in, and $nin age between 20 and 40 {author: mike, age: {$gte: 20, $lt: 40}} Advanced queries keyword $where db.posts.find({$where: this.author == mike this.title == dbms }) Source: h<p:// 13

14 Queries involving dates Complex Queries JavaScript starts posts since April 1 coun;ng months at 0 april1 = new Date(2013, 3, 1) db.posts.find({date: {$gt: april1}}) Sor@ng and limit last ten posts db.posts.find() lazy evalua@on (cursors) all posts.sort({date: -1}).limit(10) descending 14

15 Cursors var c = db.posts.find({author: mike }).skip(20) omit first 20 results.limit(10)) c.next() c.next()... next result only return 10 results query first N results and cursor id next with cursor id next N results and cursor id or eof... 15

16 Queries over Text and Collec=ons Use regular expression in find() method posts ending with Tech db.posts.find({text: /Tech$/}) Regular Expression Members of valued a]ributes are queried in the same way as a]ributes with atomic values posts with a specific tag db.posts.find({tags: mongodb }) Use of a mul@- key index speeds up collec@on- oriented queries db.posts.ensureindex({tags: 1}) 16

17 Aggrega=on Func=ons the number of documents in a collec@on total posts db.posts.count() total posts authored by Mike db.posts.find({author: mike }).count() Method distinct() displays the dis@nct values found for a specified key in a collec@on Method group() groups documents in a collec@on by the specified key and performs simple aggrega@on More ways to compute aggregated values aggrega=on framework using aggregate command map/reduce aggrega=on for large data sets using mapreduce command 17

18 Atomic Update Modifiers var c = {author: eliot, date: new Date(), text: Great post! } query selector db.posts.update({_id: post._id}, {$push: {comments: c}}) update value MongoDB has no transac@ons complex func@onality exposed through update modifiers $push adds (appends) an object to an array Client does not need to worry about locking of documents conten@on of updates 18

19 Document Evolu=on post = {author: mike, date: new Date(), text: another blog post..., tags: [ mongodb ], title: Introduction to MongoDB } db.posts.save(post) New A<ribute Flexible schema documents within the same collec@on can have different a]ributes document a]ributes can simply be added or removed evolu@on performed when save() is invoked 19

20 Horizontally Scalable Architectures Remember they said No joins and no complex row hard to implement both when scaling horizontally abandon the data model 20

21 Parallel Database Execu=on - Join SELECT brand, descrip@on, size, shelfnumber, shelfposi@on FROM shoes, shoestorage WHERE shoes.id = shoestorage.id AND color = Green AND lastworn < Case 1: Tuples from shoes and shoestorage are randomly par@@oned (sharded) across the three disks. Bold lines indicate transfer across network Computer 1 JOIN JOIN Computer 2 RePar@@on RePar@@on RePar@@on RePar@@on SELECT SCAN SELECT SCAN shoes shoestorage shoes shoestorage

22 Transac=ons Across Systems T1 Update Converse Black/Red - > Black/Red/White T2 Update Nikes YellowGreen - > ReallyBrightYellow T1 Update Nike Yellow/Green - > NeonYellow T2 Update Nike Yellow/Green - > NeonYellow T1 wai@ng on T2 T2 wai@ng on T1 Distributed Deadlock Brand Size Color T1 T2 Brand Size Color Vans 8 Red/Gold Converse 8 White DC 8 White/Blue/Green Converse 8 Blue Nike 8.5 YellowGreen Converse 8 Black/Red 22

23 Atomic Update Modifiers var c = {author: eliot, date: new Date(), text: Great post! } query selector db.posts.update({_id: post._id}, {$push: {comments: c}}) update value MongoDB has no transac@ons complex func@onality exposed through update modifiers $push adds (appends) an object to an array Client does not need to worry about locking of documents conten@on of updates 23

24 No begin, commit, rollback So, no transac=ons Atomic only at the single document level And, they have Write Concern 24

25 Write Concern Errors Ignored MongoDB does not acknowledge write Client cannot detect failed write including errors, duplicate key But fast performance Unacknowledged MongoDB does not acknowledge the receipt of write Similar to errors ignored; however, drivers a]empt to receive and handle network errors when possible. Used to be the default write concern. Acknowledged Mongodb confirms the receipt of the write Allows clients to catch network, duplicate key, and other errors. Is now the default write concern. h]p://docs.mongodb.org/manual/core/write- concern/ 25

26 Journaled Write Concern Mongodb acknowledges the write only auer commivng the data to the journal. Write is recoverable. Write must wait for next journal commit. To reduce latency, mongodb increases the frequency that it commits to the journal Replica Acknowledged With replica acknowledged you can guarantee that the write propagates to the members of a replica set. See To set replica acknowledged write concern, specify w values greater than 1 to your driver. h]p://docs.mongodb.org/manual/core/write- concern/ 26

27 Indexing 27

28 Indexing MongoDB has support for user- defined secondary indexes Indices defined at the level B- tree indexes for keys and compound keys db.posts.ensureindex({tags: 1}) Support for key indexes index document for every key value in an array text, hash indices; Unique indices Source: h]p:// 28

29 Embedded Documents Document model supports nested or embedded documents {author: mike, Embedded title: Teach Yourself MongoDB, Documents tags: [ mongodb, intro ], comments: [{author: eliot, text: Awesome tutorial! }, {author: fred, text: Thanks Mike! }], text: MongoDB is an open-source... } Dot nota@on to index and query embedded documents create an index on an embedded document db.posts.ensureindex({comments.author: 1}) find posts with a comment by Eliot db.posts.find({comments.author: eliot }) 29

30 Query Op=mizer No algebraic or cost- based query evaluator processes plans in parallel first plan to finish is remembers as most efficient other plans are terminated query plan is recorded re- evaluate plan auer 1000 writes, add, drop or rebuild index Find posts by Joe about database management systems db.posts.find({author: joe, tags: dbms }) linear scan terminate index on tags terminate index on author remember

31 Data File Alloca=on Memory- mapped storage engine files are mapped into memory MongoDB stores each database as a number of files namespace contains the catalog data files contain the collec@ons and documents 16MB blog.ns 64MB blog.0 128MB blog.1 16MB test.ns... double in size (up to 2 GB) allocated for each database h]p://docs.mongodb.org/manual/faq/storage/ 31

32 Extent Alloca=on blog.0 blog.1 blog Extents allocated into namespaces per blog.posts blog.comments blog.users blog.$freelist 000 pre- allocated space Namespace metadata details stored in blog.ns Empty file is kept in order to avoid blocking on 32

33 Drivers Supported Pla[orms and Languages PHP, Perl, Python, Ruby, Java, C++/C, C#, Erlang, Haskell Plazorms Window (x86, x64) Linux (x86, x64) OS X (x86, x64) Solaris FreeBSD 33

34 client driver Replica=on writes reads primary secondary primary primary secondary primary primary secondary secondary secondary secondary primary provides redundancy and increases data availability Replica sets added in version 1.6 master/slave - style replica@on, but roles can change primary receives all writes if primary fails, secondaries elect a new primary h]p://docs.mongodb.org/manual/replica@on/ 34

35 Read Preferences Read Preference Mode primary primarypreferred secondary secondarypreferred nearest Descrip=on Default mode. All read from the current replica set primary. In most read from the primary but if it is unavailable, read from secondary members. All read from the secondary members of the replica set. In most read from secondary members but if nosecondary members are available, read from the primary. read from the nearest member of the replica set, of the member s type. h]p://docs.mongodb.org/manual/core/read- preference/ 35

36 Auto- Sharding Shards Stores the data mongod mongod mongod... Servers mongod mongod mongod mongod mongod Query Routers Supports range- based and hash- based sharding based on a shard key mongod mongos mongos... Stores the cluster s metadata; contains a mapping of the cluster s data set to the shards. Produc@on must have exactly 3 config servers. client Interfaces with client applica@ons and directs opera@ons to the appropriate shard or shards h]p://docs.mongodb.org/manual/sharding/ 36

37 Other Features and map/reduce complex ETL/offline tasks Capped fixed- sized with wrap around (overwrite) e.g., for website logging Unique indexes Mongo shell GridFS file system API built on top of MongoDB stores large binary data by chunking it up into smaller documents Two- dimensional geo- indexing e.g., foursquare 37

38 References M. Dirolf: Introduc=on to MongoDB, O Reilly Webcast, 2010, available at: h]p:// h]p://docs.mongodb.org/manual/ Chapter 9 Sandage & Fowler 38

39 Data Management in the Cloud TO BE CONTINUED 39