Speed, scale, query: can NoSQL give us all three? Arun Matthew Couchbase

Transcription

1 Speed, scale, query: can NoSQL give us all three? Arun Matthew Couchbase

2 The project management triangle 2015 Couchbase Inc. Photo by CC-BY 2

3 The data storage triangle star Consistency Ease Speed Availability Distributability Hardware costs Flexibility Query Scale Accuracy 2015 Couchbase Inc. 3

4 The data storage circle Consistency Ease Speed Availability Distributability Hardware costs Flexibility Query Scale Accuracy 2015 Couchbase Inc. 4

5 What affects speed, scale and query? 2015 Couchbase Inc. 5

6 First up: data models 2015 Couchbase Inc. 6

7 What do we mean by NoSQL? Non-relational NoSQL 2015 Couchbase Inc. 7

8 Four popular NoSQL data models Key-value Logo Key Value Profile { } "name": "A Person", "location": "Someplace" 2015 Couchbase Inc. 8

9 Four popular NoSQL data models Document Key { "name": "A Person", "location": "Place", "team": "Team A", "interests": "music" } 2015 Couchbase Inc. 9

10 Four popular NoSQL data models Columnar Author Row Column Title 1 Year Column of release 2 JK Rowling Harry Potter and the Philosopher's Stone 1997 Harry Potter and the Chamber of Secrets 1998 Harry Potter and the NoSQL database Couchbase Inc. 10

11 Four popular NoSQL data models Graph Guitar played by Brian May Queen This is Euler 2015 Couchbase Inc. 11

12 Data model is the first consideration Relational Object Key-value Graph Document Columnar 2015 Couchbase Inc. 12

13 Next up: architecture 2015 Couchbase Inc. 13

14 Master-slave Master Slave 1 Slave 2 Slave 3 Slave Couchbase Inc. 14

15 Master-master: replicated topology Master Master Master 2015 Couchbase Inc. 15

16 Master-master: distributed topology Master Master Master 2015 Couchbase Inc. 16

17 Master-master: replicated v distributed Replicated Dataset must fit on one machine Eventually consistent Distributed Dataset is sharded across machines: can be huge Write to/read from any machine CP or AP 2015 Couchbase Inc. 17

18 Architecture is the second consideration Master-master: distributed Master-master: replicated Master-slave 2015 Couchbase Inc. 18

19 How do data model and architecture influence speed, query and scalability? 2015 Couchbase Inc. 19

20 Four examples Single server key-value store Master-slave document store Multi-master eventually consistent column store Multi-master strongly consistent document store 2015 Couchbase Inc. 20

21 Single server, key-value GET Master 2015 Couchbase Inc. 21

22 Single server GET SET Master 2015 Couchbase Inc. 22

23 Single server, key-value Impact Data model Speed None None Query None None Architecture Scale Need to choose AP or CP Manual sharding on the application layer Master 2015 Couchbase Inc. 23

24 Master-slave, document GET Slave Master Slave 2015 Couchbase Inc. 24

25 Master-slave, document GET SET Slave Master Slave 2015 Couchbase Inc. 25

26 Master-slave, document Impact Data model Architecture Speed None Favours reads, over writes Query Ad-hoc query possible Eventual consistency Slave Master Slave Scale Distinct documents are easily distributed Master is a SPOF 2015 Couchbase Inc. 26

27 Dynamo-like distributed master-master column store SET Master Master Master 2015 Couchbase Inc. 27

28 Dynamo-like distributed master-master column store SET X Master Master Master 2015 Couchbase Inc. 28

29 Dynamo-like distributed master-master column store GET X Master Master Master 2015 Couchbase Inc. 29

30 Dynamo-like distributed master-master column store Impact Data model Architecture Speed Favours writes, over reads Favours reads, over writes Query Favours range-queries Ad-hoc not so easy Eventual consistency complicates queries Scale None No SPOFs High write availability Linear scalability Master Master Master 2015 Couchbase Inc. 30

31 Distributed master-master, strong consistency, document SET Active Active Active Replica Replica Replica 2015 Couchbase Inc. 31

32 Distributed master-master, strong consistency, document SET Active Active Active Replica Replica Replica 2015 Couchbase Inc. 32

33 Distributed master-master, strong consistency, document GET Active Active Active Replica Replica Replica 2015 Couchbase Inc. 33

34 Distributed master-master, strong consistency, document GET Active Active Active Replica Replica Replica 2015 Couchbase Inc. 34

35 Distributed master-master, strong consistency, document FAILURE? Active Replica Active X Replica Active Replica 2015 Couchbase Inc. 35

36 Distributed master-master, strong consistency, document FAIL OVER Replica Active Active X Replica 2015 Couchbase Inc. 36

37 Distributed master-master, strong consistency, document FAIL OVER Replica Active Active X Replica 2015 Couchbase Inc. 37

38 Distributed master-master, strong consistency, document NODE SPECIALISATION Query Query Index Index Data Data Data 2015 Couchbase Inc. 38

39 Distributed master-master, strong consistency, document MULTI-DIMENSIONAL SCALING Query Query Index Index Index Index Data Data Data 2015 Couchbase Inc. 39

40 Distributed master-master, strong consistency, document MULTI-DIMENSIONAL SCALING Data Query Query Index Index Index Data Data Data 2015 Couchbase Inc. 40

41 Distributed master-master, strong consistency, document Impact Data model Architecture Speed Simple GETs and SETs Single read, optionally single write Query Scale Simplifies complex ad-hoc query Distinct documents are easily distributed Strong consistency makes complex query easier Linear scalability No SPOFs No conflicts Active Active Active Replica Replica Replica 2015 Couchbase Inc. 41

42 Diving deeper into query 2015 Couchbase Inc. 42

43 The first NoSQL approach to query Photo by Donarreiskoffer. CC-by Couchbase Inc. 43

44 Manual secondary indexes 2015 Couchbase Inc. 44

45 Map-Reduce was one of the first steps towards query 2015 Couchbase Inc. 45

46 Declarative query for NoSQL 2015 Couchbase Inc. 46

47 Declarative query DB-specific: Neo4J's Cypher or MongoDB's query Attempts at standardisation: Jsoniq SQL reworked for a non-relational model 2015 Couchbase Inc. 47

48 DB-specific query: MongoDB db.staff.find({office: 'London'}) db.staff.find({office: {$in:['london', 'Amsterdam']}}) db.staff.insert({name: 'Matthew Revell', office: 'London'}) db.staff.update({name: 'Matthew Revell', office: 'Amsterdam'}) 2015 Couchbase Inc. 48

49 Attempt at standardisation: JSONiq Based on XQuery Functional language Works with sets, rather than tuples 2015 Couchbase Inc. 49

50 Attempt at standardisation: JSONiq for $p in collection('staff') where $p.serviceyears gt 2 let $name := $p.firstname " " $p.lastname group by $p.office order by $p.serviceyears return { $name, $p.office, $p.serviceyears } 2015 Couchbase Inc. 50

51 SQL for NoSQL: what needs to change? Data is nested Schema is unenforced, so data is heterogenous Data is not normalised 2015 Couchbase Inc. 51

52 SQL for NoSQL: SQL Couchbase Inc. 52

53 SQL for NoSQL: SQL++ Superset of SQL for semi-structured data Handles missing data gracefully and/or explicitly Can query inside nested data Nests and unnests data in results JOINs between documents 2015 Couchbase Inc. 53

54 Introducing N1QL: SQL++ in action 2015 Couchbase Inc. 54

55 Finding all the airports SELECT * FROM `travel-sample` WHERE type='airport'; 2015 Couchbase Inc. 55

56 Limiting and ordering our results SELECT * FROM `travel-sample` WHERE type='airport' ORDER BY COUNTRY LIMIT 10; 2015 Couchbase Inc. 56

57 Working with documents { } "id": 3469, "type": "airport", "airportname": "San Francisco Intl", "city": "San Francisco", "country": "United States", "faa": "SFO", "icao": "KSFO", "tz": "America/Los_Angeles", "geo": { "lat": , "lon": , "alt": 13 } 2015 Couchbase Inc. 57

58 Working with nested data SELECT a.name, s.flight, s.utc, r.sourceairport, r.destinationairport, r.equipment FROM `travel-sample` r UNNEST r.schedule s JOIN `travel-sample` a ON KEYS r.airlineid WHERE r.sourceairport="lhr" AND r.destinationairport = "SFO" AND s.day=1 ORDER BY s.utc; 2015 Couchbase Inc. 58

59 Flying to and from SFO { } { "callsign": "UNITED", "country": "United States", "iata": "UA", "icao": "UAL", "id": 5209, "name": "United Airlines", "type": "airline" { "airline": "UA", "airlineid": "airline_5209", "destinationairport": "SFO", "equipment": "777", "id": 57047, "schedule": [ { "day": 0, "flight": "UA894", "utc": "02:32:00" "id": 3469, "type": "airport", "airportname": "San Francisco Intl", "city": "San Francisco", "country": "United States", }, "faa": "SFO",... "icao": "KSFO", ], "tz": "America/Los_Angeles", "sourceairport": "LHR", "geo": "stops": 0, { "type": "route" "lat": , } "lon": , "alt": 13 } 2015 } Couchbase Inc. 59

60 Prepared statements Optimise frequently-run queries Execution plan happens once, query is run multiple times PREPARE LonSanFran FROM SELECT airline FROM `travel-sample` WHERE sourceairport="lhr" AND destinationairport = "SFO"; 2015 Couchbase Inc. 60

61 Creating indexes CREATE INDEX ukairports ON `travel-sample`(type) WHERE type='airport' AND country='uk' USING GSI; 2015 Couchbase Inc. 61

62 Mutating data DELETE: provide the key to delete the document INSERT: provide a key and some JSON to create a new document UPSERT: as INSERT but will overwrite existing docs UPDATE: change individual values inside existing docs 2015 Couchbase Inc. 62

63 Recapping NoSQL speed 2015 Couchbase Inc. 63

64 Speed and NoSQL Key-value Document Column Graph Data model considerations No CPU load, minimal disk seeks Largely single ops, minimal disk seeks, often relatively simple query Rapid writes Simplifies otherwise expensive queries Architecture considerations Master-slave Master-master replicated Master-master distributed Speeds up reads, slows writes Speeds up reads and writes (with consistency lag) Speeds up reads and writes 2015 Couchbase Inc. 64

65 Recapping NoSQL scale 2015 Couchbase Inc. 65

66 Scale and NoSQL Key-value Document Column Graph Data model considerations Each item is independent, easily distributed Item independence, easily distributed. Indexes might bring cross-node dependencies. Distribute column families, hashed sharding Shard based on data Architecture considerations Master-slave Master-master replicated Master-master distributed Speeds up reads, slows writes Speeds up reads and writes (with consistency lag) Speeds up reads and writes 2015 Couchbase Inc. 66

67 Scale out, scale up or both: multi-dimensional scaling node1 node8 Index Service Query Service Data Service Couchbase Cluster node1 node8 node9 Query Service Index Service Data Service 2015 Couchbase Inc. 67

68 Next steps 2015 Couchbase Inc. 68

69 What next? Developer portal: developer.couchbase.con Forums: forums.couchbase.com Free online training: training.couchbase.com/online Join your local Couchbase meet-up: bit.ly/couchbasemeetups Follow the Couchbase developer community on 2015 Couchbase Inc. 69

70 Thank you Q&A