HOW CLOUD DATABASE ENABLES EFFICIENT REAL-TIME ANALYTICS?

Size: px

Start display at page:

Download "HOW CLOUD DATABASE ENABLES EFFICIENT REAL-TIME ANALYTICS?"

Bruno Quinn
8 years ago
Views:

1 HOW CLOUD DATABASE ENABLES EFFICIENT REAL-TIME ANALYTICS?

2 DATA MANAGEMENT MATTERS Worldwide data volumes keep growing

3 Real time management of big data Return result in milliseconds Deals with TBs to PBs of data FAST CONTRADICTING GOALS? HIGH CAPACITY WHAT? NEED FOR ADVANCED TECHNOLOGY

4 HOW CAN TECHNOLOGY MAKE DATA ACCESS REAL-TIME IN A COST EFFECTIVE WAY? 1. Utilize the right hardware 2. Build advanced indices 3. Cloud Computing HOW? 4. Consistency

5 1. Utilize the right hardware STORAGE MEDIA There are three types of storage media RAM SSD HDD

6 1. Utilize the right hardware STORAGE MEDIA How do they differ? RAM SSD HDD $ / TB 12, Read time / TB 100 ms read size GB 40 s 20 min 3 h 2.5 GB

7 Relational (SQL) vs Document Oriented (NoSQL) Data Model 1. Utilize the right hardware Data represented in complex tabular structure Data is organized in self contained documents distributed among many servers

8 Implications on scaling 1. Utilise the right hardware Relational (SQL) vs Document Oriented (NoSQL) Scales vertically by adding a bigger server, which is disproportionally expensive Scales horizontally by adding a more servers, thus costs growing proportionally with data

(NoSQL) Scales vertically by adding a bigger server, which is

9 1. Utilise the right hardware TYPICAL 30 SERVER CLUSTER RAM SSD HDD Storage, TB Cost, $ 24,000 12,000 5, ms read size GB Read ratio 4% 0.01% 2.3*10-6

100 Cost, $ 24,000 12,000 5,000 100 ms read

10 2. Indexing techniques INDEX An index is an indirect shortcut derived from and pointing into, a greater volume of values, data, information or knowledge. 30 TB TOTAL VOLUME STORED IN CLUSTER 3 GB RELEVANT TO PARTICULAR QUERY TAKES 20 MIN TO READ TAKES 100 MILLISECONDS TO READ

11 2. Indexing techniques GEOSPACIAL DATA Data collected from devices can generate large amount of location based data. Data items with 2 or 3 (incl. time) coordinates Scattered across grid with varying density

location based data. Data items with 2 or 3 (incl.

12 2. Indexing techniques WHY DOES THIS MATTER? 30 TB TOTAL VOLUME OF GEO DATA INDEXED DATA RELEVANT ONLY TO TO A PARTICULAR A AREA AREA OF OF INTEREST CAN BE READ IN REAL-TIME FROM SMALL AREA ON STORAGE MEDIA

13 2. Indexing techniques SPACE FILLING CURVE Can 2 dimensional space be filled with a 1 dimensional curve? Yes, first discovered in 1890 by Giuseppe Peano Most famous space filling curve invented by David Hilbert

14 2. Indexing techniques HILBERT CURVE ALLOWS TRANSFORMING 2D COORDINATES TO 1D WITH SPACE LOCALITY

15 2. Indexing techniques HILBERT CURVE

16 2. Indexing techniques FULL-TEXT SEARCH TEXT INDEX 1: Hickory, dickory, dock. 2: The mouse ran up the clock. 3: The clock struck one, 4: The mouse ran down, 5: Hickory, dickory, dock. clock: 2, 3 dickory: 1, 5 dock: 1, 5 down: 4 hickory: 1, 5 mouse: 2, 4 one: 3 ran: 2, 4 struck: 3 the: 2, 3, 4 up: 2 CLOCK RAN: 2, 3 2, 4 = 2

3: The clock struck one, 4: The mouse ran down, 5: Hickory, dickory, dock.

17 3. Cloud Computing IN-PREMISE VS CLOUD ORG 1 ORG 2 CLOUD PROVIDER ORG 3 ORG 4 Reducing Operational Overheads ORG 5

18 3. Cloud Computing IN-PREMISE VS CLOUD ORG 1 ORG 2 ORG 3 ORG 4 ORG 5

19 3. Cloud Computing IN-PREMISE VS CLOUD CLUSTERPOINT CLOUD EXACTLY THE SAME TOTAL AMOUNT OF WORK EACH QUERY RUNS FASTER DUE TO PARALLELISM

20 3. Consistency Model simple account transfer ACCOUNT A $ 300 ACCOUNT B READ A READ B A = A B = B WRITE A WRITE B'

21 3. Consistency Distributed Architecture CLIENT CLIENT HUB HUB HUB

22 3. Consistency Assign Shards to Nodes A B C D E F G DB1 S0-R1 DB1 S0-R2 DB1 S0-R0 DB2 S1-R1 DB1 S1-R0 DB1 S1-R1 DB3 S1-R2 DB2 S0-R0 DB2 S0-R1 DB3 S0-R0 DB3 S1-R1 DB2 S0-R2 DB2 S1-R2 DB1 S1-R2 DB3 S0-R1 DB3 S0-R2 DB3 S1-R0 DB2 S1-R0

23 3. Consistency ACID-compliant multi-document transactions Hard problem for distributed systems CLIENT HUB Everything has to be in a consistent state HUB S0-R0 S0-R1 S0-R2 S7-R0 S7-R1 S7-R2

24 3. Consistency Solution 1. Enclose operations in a transaction with unique ID 2. Every document/version assigned a transaction_id with which it was added and removed DOC 1001 TID 372 TID 404 DOC 1002 TID 584 TID 703 DOC 1003 TID 672

25 3. Consistency Solution What happens during commit? HUB TID=672 DOC 1001 TLV 1 TLV 2 1: TID 372 DOC 1002 TLV 3 TID 703 2: TID 404 DOC 1003 TID TLV : TID 584 4: TID 672

26 Thank you!

MongoDB and Couchbase

Benchmarking MongoDB and Couchbase No-SQL Databases Alex Voss Chris Choi University of St Andrews TOP 2 Questions Should a social scientist buy MORE or UPGRADE computers? Which DATABASE(s)? Document Oriented