In-Place Computing. For Big and Complex Data

Transcription

1 In-Place Computing For Big and Complex Data

2 In-Place Computing Data objects live (save) and work (compute) in a 8lat and in8inite address space. Data- centric computing (why not moving small code to big data?) Organizing vertical complexity Approximated in- memory system by 2 64 address space

3 In-Place Objects Virtual Memory Space No Swapping or Double Paging CPU In- place object Paging (on- demand) File

4 Dealing With Big and Complex Data The Portraits of Big Data: the 3V model Another V is introduced and helps to determine the complexity - Valence, the degree of interdependency among data components, which normally causes data shufkling. In- place Computing Model is designed to solve high- valence big data problems. Beyond scaling- up and scaling- out, we focus on scaling- in by squeezing out the CPU power of a single machine.

5 Observations Behind In-Place Computing I. History of Code and Data II. Complexity Reduction Traditionally, we handle code and data in different ways and keep them in different spaces. Code size > Data size Transactions Operations Data size > Code size Analytics

6 Scale-Out, Scale-Up, and Scale-In Scale- Up Spread out the data components over a!lat and virtually in!inite address space, where data are ready for the CPU to compute. Scale- Out

7 2 64 : Nearly Infinite The introduction of 64- bit CPU evokes revisit to the software discipline address space is big enough to hold the big data we encounter today. When address space is nearly inkinite, why don t we place the data components over it and make them ready for computations instead of retrieving data to programs?

8 Trade Space with Time Hanoi Tower 2 N - 1 moves Exponential Time O(2 N ) à 2xN- 1 moves Linear Time O(N) With N+1 spaces, we can reduce time complexity from exponential to linear time

9 Macro Data Structure (MDS) Data structure viable to keep, organize, express and compute big data (say size > 4GB) in an efkicient manner. Analogy to (and inspiration from) macromolecule. Intended to be a basic functional unit of computing, given a proper and meaningful set of operations. Like most literatures for data structure, an data structure often refer to an instance of such data structure. For example, a hash table means an instance of hash table structure.

10 Properties of MDS BEPR (Big, In- Place, Persistent and Relocatable) Big virtually inkinite space In- Place ready to compute (no data retrieval) Persistent and long- lived mapped to storage Relocatable - moveable from one space to another TP (ACID) and OO are orthogonal to MDS

11 File-Based MDS File as Memory (mmap) Transferable (relocatable) and splitable Buck operation or matrix/array based computing (e.g. GPU) Paging efkicient in kernel level IO efkicient/asynchronous in kernel level Revision controlled for data consistency Parameterized types for extension for various types of MDS Easy to backup and restore

12 Hierarchical MDS A natural form of human s thinking process Divide and Conquer Split for distributed/parallel computing stationary North accessory pad book mouse usb stationary South accessory 0 1 pad book mouse usb

13 Hierarchical MDS Sales Manager R Hierarchical MDS for Supply Chain Planning, where the hierarchy is dekined by the relations in the relational DB. Store S0 S1 S2 S3 Category C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 p0 p1 p2 p3 p4 p5 p6 p7 p8 p9 p1 0 p1 1 p1 2 p1 3 p1 4 p1 5 p1 6 p1 7 p1 8 p1 9 13

14 Design the right Hierarchical MDS that meets the requirements above.

15 Problems Either no sibling locality or no descendant locality Location- dependent and difkicult to move (sub- trees) around without changing pointers Paging inefkiciency when navigation and split

16 Design Principles Keep both sibling locality and descendant locality in the meantime Brother nodes must be adjacent All nodes in a sub- tree must be together without others Keep paging efkiciency for navigation, split- join, add/remove nodes (use the least paging) Location independent while move around Consistent behavior while evolving

17 Performance: Roll up 1M Data Records 100M Data Records IPC PostgresSQL MySQL NewSQL DB 0 IPC PostgresSQL MySQL NewSQL DB

18 Performance: Log Analysis Query Time (s) IPC MySQL Commmercial Log Analysis Tool Records (Millions)

19 Scalability Scale- up Never Out of Memory BigObject scales even when the data size grows beyond swap space. Linearly Degraded While the others usually degrade exponentially Scale- out Horizontal Scale Partially replicate dimension tables Partition fact tables Perform set operators such as union or inter- sect BigObject

20 Website: Contact: