Divy Agrawal and Amr El Abbadi Department of Computer Science University of California at Santa Barbara

Size: px

Start display at page:

Download "Divy Agrawal and Amr El Abbadi Department of Computer Science University of California at Santa Barbara"

Meghan Gray
8 years ago
Views:

1 Divy Agrawal and Amr El Abbadi Department of Computer Science University of California at Santa Barbara

2 Sudipto Das (Microsoft summer intern) Shyam Antony (Microsoft now) Aaron Elmore (Amazon summer intern) 8/5/2010 2

3 Different from earlier attempts: Distributed Computing Distributed Databases Grid Computing Cloud Computing is REAL Organic growth: Yahoo!, Microsoft, Amazon,Google Poised to be an integral aspect of the global computing Infrastructures 8/5/2010 3

4 Client Site Client Site Client Site HAProxy (Load Balancer) Elastic IP 8/5/2010 MySQL Master DB Replication Database becomes the Scalability Bottleneck MySQL Slave DB Cannot leverage elasticity 4

5 Client Site Client Site Client Site HAProxy (Load Balancer) Elastic IP MySQL Master DB Replication MySQL Slave DB 8/5/2010 5

6 Client Site Client Site Client Site HAProxy (Load Balancer) Elastic IP Scalable and Elastic Key Value Stores But limited consistency and 8/5/2010 operational flexibility 6

7 Scalability Elasticity Fault tolerance Self Manageability Sacrifice consistency? Can be done, but is it a foregone conclusion!!! 8/5/2010 7

8 Separate System and Application State Limit interactions to a single node Decouple Ownership from Data Storage Limited dist synchronization is practical 8/5/2010 8

9 Data Fusion Enrich Key Value stores GStore: Efficient Transactional Multi-key access [ACM SOCC 2010] Data Fission Cloud enabled relational databases ElasTraS: Elastic TranSactional Database [HotClouds2009;Tech. Report 2010] 8/5/2010 9

11 Many applications need multi-key accesses: Online multi-player games Collaborative applications Enrich functionality of the Key value stores 8/5/

12 Apps select any set of keys to form a group A granule of on-demand transactional access Data store provides transactional group access Non-overlapping groups 8/5/

13 Keys located on different nodes Horizontal Partitions of the Keys Key Group A single node gains ownership of all keys in a KeyGroup Group Formation Phase 8/5/

14 Conceptually akin to locking Allows collocation of ownership Transfer key ownership from followers to leader Guarantee safe transfer in the presence of system dynamics: Dynamic migration of data and its control Failures 8/5/

15 Transactional Multi-Key Access Application Clients Grouping Middleware Layer resident on top of a Key-Value Store Grouping Layer Transaction Manager Key-Value Store Logic Grouping Layer Transaction Manager Key-Value Store Logic Grouping Layer Transaction Manager Key-Value Store Logic 8/5/2010 Distributed Storage G-Store 15

17 Designed to make RDBMS cloud-friendly Database viewed as a collection of partitions Suitable for standard OLTP workloads: Large single tenant database instance Database partitioned at the schema level Multi-tenant with large number of small databases Each partition is a self contained database 8/5/

18 Elastic to deal with workload changes Dynamic load balancing of partitions Automatic recovery from node failures Transactional access to database partitions 8/5/

TM Master Health and Load Management OTM OTM Application Clients DB Read/Write Workload Lease Management OTM Durable Writes Metadata Manager

19 TM Master Health and Load Management OTM OTM Application Clients DB Read/Write Workload Lease Management OTM Durable Writes Metadata Manager Master Proxy Application Logic Txn Manager P 1 P 2 P n DB Partitions ElasTraS Client MM Proxy Log Manager Distributed Fault-tolerant Storage 8/5/

20 Performed in Amazon EC2 Used TPC-C for evaluation Cluster size: 10 to 30 nodes Number of concurrent clients: 100 to 1800 Data size: ~1T Each node: 8 cores, 7G RAM, 1.7T disk Max thruput: 0.2M TPC-C Xact/sec on 30 machines using HDFS & Zookeeper software. 8/5/

21 Cloud Computing poses fundamental challenges to database researchers: Scalability, Reliability and Data Consistency Need to understand the new applications Live Data migration is critical. Challenging multi-node/center atomic operations Clear characterization of properties & guarantees. 8/5/

Database Scalabilty, Elasticity, and Autonomic Control in the Cloud

Database Scalabilty, Elasticity, and Autonomic Control in the Cloud Divy Agrawal Department of Computer Science University of California at Santa Barbara Collaborators: Amr El Abbadi, Sudipto Das, Aaron