26/05/2015. Relational Databases BIG DATA: STORING STRUCTURED INFORMATION. Information Retrieval: Storing Unstructured Information

Transcription

1 CC PROCESAMIENTO MASIVO DE DATOS OTOÑO 2015 Information Retrieal: Storing Unstructured Information Lecture 9: NoSQL I Aidan Hogan aidhog@gmail.com Relational Databases BIG DATA: STORING STRUCTURED INFORMATION Relational Databases: One Size Fits All? 1

2 RDBMS: Performance Oerheads Structured Query Language (SQL): Declaratie Language Lots of Rich Features Difficult to Optimise! Atomicity, Consistency, Isolation, Durability (ACID): Maes sure your database stays correct Een if there s a lot of traffic! Transactions incur a lot of oerhead Multi-phase locs, multi-ersioning, write ahead logging Distribution not straightforward Transactional oerhead: the cost of ACID 640 tps for system with transactional support 12,700 tps for system without logs, transactions or loc scheduling RDBMS: Complexity ALTERNATIVES TO RELATIONAL DATABASES FOR QUERYING BIG STRUCTURED DATA? NoSQL What do you guys now about NoSQL? The Database Landscape Batch analysis of data Not using the relational model Using the relational model Real-time Stores documents (semi-structured alues) Not only SQL Maps Column Oriented Graph-structured data Relational Databases with focus on scalability to compete with NoSQL while maintaining ACID Cloud storage In-Memory 2

3 NoSQL NoSQL: CAP (not ACID) NoSQL CA: Guarantees to gie a correct response but only while networ wors fine (Centralised / Traditional) C CP: Guarantees responses are correct een if there are networ failures, but response may fail (Wea aailability) A P (No intersection) AP: Always proides a best-effort response een in presence of networ failures (Eentual consistency) Distributed! Sharding: splitting data oer serers horizontally Replication Lower-leel than RDBMS/SQL Simpler ad hoc APIs But you build the application (programming not querying) Operations simple and cheap Different flaours (for different scenarios) Different CAP emphasis Different scalability profiles Different query functionality Different data models The Database Landscape Not using the relational model Batch analysis of data Using the relational model Real-time NOSQL: KEY VALUE STORE Stores documents (semi-structured alues) Not only SQL Maps Column Oriented Graph-structured data Relational Databases with focus on scalability to compete with NoSQL while maintaining ACID Cloud storage In-Memory 3

4 Key Value Store Model It s just a Map / Associate Array put(ey,alue) get(ey) delete(ey) Key Afghanistan Albania Algeria Andorra la Vella Angola Antigua and Barbuda Value Kabul Tirana Algiers Andorra la Vella Luanda St. John s. But You Can Do a Lot With a Map Key country:afghanistan country:albania city:kabul city:tirana user:10239 Value capital@city:kabul,continent:asia,pop: #2011 capital@city:tirana,continent:europe,pop: #2013 country:afghanistan,pop: #2013 country:albania,pop: #2013 basedin@city:tirana,post:{103,10430,201} actually you can model any data in a map (but possibly with a lot of redundancy and inefficient looups if unsorted). Products Listings: prices, details, stoc THE CASE OF AMAZON Customer info: shopping cart, account, etc. Recommendations, etc.: 4

5 Amazon customers: Key Value Store: Amazon Dynamo(DB) Databases struggling But many Amazon serices don t need: SQL (a simple map often enough) or een: transactions, strong consistency, etc. Goals: Scalability (able to grow) High aailability (reliable) Performance (fast) Don t need full SQL, don t need full ACID Key Value Store: Distribution Key Value Store: Distribution How might a ey alue store be distributed oer multiple machines? What happens if a machine joins or leaes half way through? Or a custom partitioner Or a custom partitioner 5

6 Key Value Store: Distribution How can we sole this? Or a custom partitioner Consistent Hashing Aoid re-hashing eerything Hash using a ring Each machine pics n psuedo-random points on the ring Machine responsible for arc after its point If a machine leaes, its range moes to preious machine If machine joins, it pics new points Objects mapped to ring How many eys (on aerage) need to be moed if a machine joins or leaes? Amazon Dynamo: Hashing Consistent Hashing (128-bit MD5) Key Value Store: Replication A set replication factor (here 3) Commonly primary / secondary replicas Primary replica elected from secondary replicas in the case of failure of primary A1 B1 C1 D1 E1 Amazon Dynamo: Replication Replication factor of n Easy: pic n next bucets (different machines!) Amazon Dynamo: Object Versioning Object Versioning (per bucet) PUT doesn t oerwrite: pushes ersion GET returns most recent ersion 6

7 Amazon Dynamo: Object Versioning Object Versioning (per bucet) DELETE doesn t wipe GET will return not found Amazon Dynamo: Object Versioning Object Versioning (per bucet) GET by ersion Amazon Dynamo: Object Versioning Object Versioning (per bucet) PERMANENT DELETE by ersion wiped Amazon Dynamo: Model Named table with primary ey and a alue Primary ey is hashed / unordered Primary Key Afghanistan Albania Primary Key Kabul Tirana Countries Value capital:kabul,continent:asia,pop: #2011 capital:tirana,continent:europe,pop: #2013 Cities Value country:afghanistan,pop: #2013 country:albania,pop: #2013 Amazon Dynamo: Model Dual primary ey also aailable: Hash: unordered Range: ordered Countries Hash Key Range Key Value Vatican City 839 capital:vatican City,continent:Europe Nauru 9945 capital:yaren,continent:oceania Amazon Dynamo: CAP Two options for each table: AP: Eentual consistency, High aailability CP: Strong consistency, Lower aailability What s a CP system again? What s an AP system again? 7

8 Amazon Dynamo: Consistency Gossiping Keep alie messages sent between nodes with state Amazon Dynamo: Consistency Two ersions of one shopping cart: Quorums: N nodes responsible for a read write Multiple nodes acnowledge read/write for success At the cost of aailability! Hinted Handoff For transient failures A node coers for another node while its down How best to handle multiple conflicting ersions of a alue (nowing as reconciliation)? Application nows best ( and must support multiple ersions being returned) Amazon Dynamo: Vector Clocs Vector Cloc: A list of pairs indicating a node (i.e., a serer) and a time stamp Used to trac/order ersions Amazon Dynamo: Eentual Consistency using Merle Trees Merle tree is a hash tree Nodes hae hashes of their children Leaf node hashes from data: eys or ranges Amazon Dynamo: Eentual Consistency using Merle Trees Easy to erify regions of the Map Can compare leel-at-a-time Amazon Dynamo: Budgeting Assign throughput per table: allocate resources Reads (4 KB resolution): Writes (1 KB resolution) 8

9 Read More OTHER KEY VALUE STORES Other Key Value Stores Other Key Value Stores Other Key Value Stores NOSQL: DOCUMENT STORE 9

10 The Database Landscape Key Value Stores: Values are Documents Real-time Stores documents (semi-structured alues) Not only SQL Maps Column Oriented Graph-structured data In-Memory Not using the relational model Batch analysis of data Using the relational model Relational Databases with focus on scalability to compete with NoSQL while maintaining ACID Cloud storage Key country:afghanistan Value <country> <capital>city:kabul</capital> <continent>asia</continent> <population> <alue> </alue> <year>2011</year> </population> </country> Document-type depends on store XML, JSON, Blobs, Natural language Operators for documents e.g., filtering, in. indexing, XML/JSON querying, etc. MongoDB: JSON Based Key 6ads786a5a9 o Value (Document) { _id : ObjectId( 6ads786a5a9 ), name : Afghanistan, capital : Kabul, continent : Asia, population : { alue : , year : 2011 } } Document Stores Can inoe Jaascript oer the JSON objects Document fields can be indexed Recap Relational Databases don t sole eerything SQL and ACID add oerhead Distribution not so easy RECAP NoSQL: what if you don t need SQL or ACID? Something simpler Something more scalable Trade efficiency against guarantees 10

11 Recap Recap Key alue stores inspired by Amazon Dynamo Distributed maps Hash eys and range eys Table names Consistent hashing Replication Object ersioning / ector clocs Gossiping / Quorums / Hinted Hand-offs Merle trees Budgeting Document stores: documents as alues Support for JSON, XML alues, field indexing, etc. Questions 11