Aaron J. Elmore, Carlo Curino, Divyakant Agrawal, Amr El Abbadi. cs.ucsb.edu microsoft.com

Transcription

1 Aaron J. Elmore, Carlo Curino, Divyakant Agrawal, Amr El Abbadi cs.ucsb.edu microsoft.com

2 2

3 Moving to the Cloud Why cloud? (are you really asking?) Economy-of-scale arguments Pay-per-use value to customers

4 Moving to DaaS Why Database as a Service (DaaS) for tenants? DB management drama becomes provider s problem (Ideally) high level Service Level Agreement (SLAs / SLOs) Accelerate development lifecycle Why Database as a Service (DaaS) for providers? Internalize a high-cost portion of service (admin) Scale + density + uniformity lower cost

5 The illusion we are aiming for Tenant s view Provider s view

6 Traditional DB deployments Yo the DB is slow! Try now. DB DBAdmin Developer

7 What changes in DBaaS? Developer Developer DB Developer Developer Developer DBAdmin Developer Developer Developer Developer Developer Developer

8 * experiments using OLTPBench: DaaS (managed DBMS) Manually tuned DBMS

9 DaaS: challenges (and agenda)

10 10

11 Shared Hardware (DB-in-a-VM) DBMS (tenant1) DBMS (tenant2) DBMS (tenant3) DBMS (tenant4) Guest OS VM Guest OS VM Guest OS VM Guest OS VM Hypervisor Hardware

12 Shared Process Database (tenant1) Database (tenant2) Database (tenant3) Database (tenant2) Database Process OS Hardware

13 Shared Table Rows (tenant1) Rows (tenant2) Rows (tenant3) Rows (tenant4) Table Database Process OS Hardware

14 Trade-off Shared Hardware Strong Isolation (security, performance) Mechanics (High Availability, Migration) Shared Process Sharing and coordination resource consumption (MEM/CPU/Disk IOps) Shared Table Amortize metadata overheads

15 Multi-tenancy Architectures Shared Hardware SmartSLA, RemusDB, Amazon RDS Shared Process RelationalCloud, CloudDB, SQLAzure, Delphi, Y! cidr2009 (shared storage) ElasTras, DAX Shared Table Force.com, Jacobs/Aulbach

16 DBMS (tenant1) DBMS (tenant2) DBMS (tenant3) DBMS (tenant4) Reusing/Specializing VM technologies for DaaS Guest OS VM Guest OS VM Guest OS VM Guest OS VM Hypervisor Hardware 16

17 Commercial offering: Amazon RDS Amazon RDS Provides pre-configured DBMS (MySQL/Oracle/SQLServer) Addresses much of provisioning issues Strong Isolation / catch-all configuration

18 SmartSLA [Xiong et al. ICDE 2011] Focus Leverage VM-based mechanisms Deliver DB-level SLAs Key Contribution SLA violation vs Resource modeling Actuation of VM-based mechanisms (cpu, ram, replication)

19 SmartSLA Key mechanism Decompose problem in: ML-based model of resource / SLA-penalty Allocation of resource + replication

20 Estimating SLA violation cost and Allocation ML Modeling Build a Map of space (simple ML/features) Allocation algorithm Explore allocation space Models infrastructure cost for replication Models cost of increasing replication

21 REMUSDB [Minhas et al. VLDB 2011 /VLDBJ 2013] Focus High Availability via VM replication OLTP-compatible performance Key Contributions Reuse of mature VM technology (pro of Shared Hardware) Smart DB-specific tricks to improve performance

22 REMUS Leverage Xen VM-replication Snapshots the VM state every few tens of ms Delays network and disk writes until next checkpoint (consistent) Fail-over to secondary and restart from latest checkpoint Problems DBMS bufferpool changes too fast (large deltas to checkpoint) Latency overhead is high for OLTP

23 REMUSDB: DB-specific optimizations Avoid checkpointing clean pages no checkpoint for clean pages bookkeeping so that secondary fetch from disk if needed Limit network delay to Commit/Abort Leverage transactional semantics delay only Commit/Abort messages Reduce impact on throughput 32% goes down to about 10%

24 20:1 consolidation 20:1 consolidation (skewed) Design mismatch DBMS were designed to make full use of dedicate machines Aggressively consume idle resources (especially IOPs) [Curino et al. VLDB 2010]

25 Database (tenant1) Database (tenant2) Database (tenant3) Database (tenant2) The DBMS knows best Database Process OS Hardware 26

26 Commercial offering: SQLAzure [Bernstein et al. ICDE 2011] SQL Azure Shared DBMS process, Dedicated database Shared logging Modified version of SQL Server High-availability via quorum of replicas Support scale-out ACID within a row-group Read-committed across row-group

27 ElasTras Architecture [Das et al. HotCloud 2009] (Shared Storage) Health and Load Management TM Master OTM OTM Lease Management OTM Durable Writes Metadata Manager Master and MM Proxies Txn Manager P 1 P 2 P n Log Manager DB Partitions Distributed Fault-tolerant Storage

28 DAX [Liu et al. VLDB 2013] Scalable and fault tolerant m/t achieved by data layer spanning colos Use Cassandra for storage tier with single owning DB instance Leverage DB and quorum semantics for performance Operation type & R/W/N Epoch-bounded strong consistency

29 RelationalCloud [Curino et al. CIDR 2011]

30 * experiments using OLTPBench:

31 Rows (tenant1) Rows (tenant2) Rows (tenant3) Rows (tenant4) Extreme multi-tenancy Table Database Process OS Hardware 32

32 [Jacobs and Aulbach BTW 2007] Key idea DBMSs don t scale well at the tenant/schema level

33 Force.com and [Aulbach et al. SIGMOD 2008] Focus Target tens of thousands of tenants per server Partially shared schema (polymorphic SaaS apps) Deal with schema-level DBMS scalability limits Key Contribution Clever data design, schema mapping / query rewriting

34 [Aulbach et al. SIGMOD 2008] Many variants Private Table Extension Table Universal Table Pivot Table Chunk Table Chunk Folding

35 Focused on extreme multi-tenancy Middleware-based querying rewriting Ad-hoc security Hard to provide performance isolation Only for small / low-activity tenants

37 39 Chop it and scale it out

38 Schism [Curino et al. VLDB 2010] Positioning Partitioning for shared-nothing DBMSs (RelationalCloud) Focus automatic partitioning of arbitrary schemas (many-to-many) handle access skew, replication Key Contributions Model the problem as graph-partitioning Explain results using decision trees (practical partition functions)

39 Schism: Graph-based Partitioning

40 Schism: Graph-based Partitioning Graph Representation: tuples in the DB are nodes in the graph

41 Schism: Graph-based Partitioning Graph Representation: tuples in the DB are nodes in the graph transactions impose edges among the tuples they access

45 Schism: Graph-based Partitioning Graph Partitioning, NP-Complete, but great heuristics (METIS) Graph Partitioning: find K (close to) balanced partitions of the nodes that minimize the weight of the cut edges (i.e., minimize distributed transactions)

46 Schism: Graph-based Partitioning Natural Classification Problem (Decision Tree) Explanation: compact, predicate-based representation of the graph-partitioning solution

47 SWORD [Quamar et al. EDBT 2013] Key Contributions Repartitioning heuristics Scaling to larger problems by pre-processing (hyper)graph Greater focus on replication for fault-tolerance Use of quorums (not just ROWA)

48 Horticulture [Pavlo et al. 2012] Focus Time-varying skew Handle Store procedures natively Key Contributions Schema and workload-driven partitioning Large neighborhood search (rich cost model + cheap estimation) Horizontal partitioning + table replication + index replication

49

50 Horticulture: Large Neighborhood search Input Best Design DDL Schema Workload Restart Initial Design Relaxation Local Search 53

51 Throughput comparison (for H-Store) (cost estimate) lower is better 60,000 50,000 40,000 (txn/sec) higher is better TPC-C 30,000 20,000 10,000 TPC-C Skewed Horticulture 0 14,000 12,000 10,000 8,000 6,000 4,000 2, Schism

52 Where are we with partitioning? Problems we know how to solve: OLAP (tons of classic work) OLTP (few recent papers, good grasp on the problem) More to do: OLAP-OLTP mixed workloads partitioning Coordinating replication (and erasure codes) for: Performance, Fault-tolerance Geo-distributed placement/replication

54 Managing Resource Contention

55 Finding the Balance Tenant s view Provider s view

56 Contention for Resources t1 t2 t3 t4

57 Enable Performance in a Shared Environment 71

58 Mechanisms to Enforce Isolation Hard Static Provisioning Resource Allocation (Dynamic Provisioning) Soft Smart Placement (Admission Control)

60 Hard Isolation Keeping your word about resource sharing

61 SQLVM [CIDR 2013, SIGMOD 2013, VLDB 2014] Focus Embedding resource allocation in DBMS kernel. How to share critical resources required by DB. How to understand resource allocation. Key Contributions Fine grain resource scheduling (CPU, Memory, I/O). Metering to audit resource promise.

62 SQLVM Motivation WHERE State = Vermont California 10/1/2013 CIDR

63 Resource Governance Mechanism 100 IOPS 50 IOPS Tenant2 Application Tenant is promised reservation of DBMS resources VM inside SQL process CPU utilization, IOPS, Memory, Capacity: 200 IOPS Machine in cluster Database server process Tenant1 database Tenant2 database Resource governance Fine-grained resource sharing Novel mechanisms Metering (auditing) Monitor actual and promised metrics for tenant Determine violations 10/1/2013 CIDR

64 Resource Allocation CPU CPU utilization Memory (Buffer Pool) Hit Ratio Disk I/O: Shaping Traffic issued 10/1/2013 CIDR

65 Challenges 8

66 Soft Isolation

68 Common Patterns Understand workloads Find placement How workloads combine Metrics

69

71 Towards Multi-Tenant Performance SLOs Focus Different hardware configurations (SKU) Multiple tenant performance SLO classes Place to meet SLOs and minimize costs Key Contributions Cost aware server consolidation Tenant placement optimization framework

72 Num of H (100tps) Tenants Heterogeneous SLO Characterization diskc Benchmark server to find max degree multi-tenancy for perf objectives Systematically reduce H tenants, steadily increase L tenant scheduling until a perf objective fails Number of L (10tps) Tenants Server characterizing function: Both perf objectives met Some perf objective fails

73 Approach Assumption Solution

74

76 Robust Tenant Placement Focus In memory databases with temporal changes / etheral DBs Minimize servers while being robust to failures Replication with ability to redirect workload Key Contributions Incremental algorithms to reduce total costs of ownership Maintain replication and respect server load. Migration and existing placement aware solution

77 Request Rate in (%) Workloads Time (in Days)

78 Placing Tenants

79 Solutions Greedy Heuristics Meta-heuristics Exact Solutions Static and incremental solutions.

80 Framework Incremental algorithms follows these steps:

81

83 PMAX Focus Latency response SLOs Workloads are not fixed and vary, history is not available Profit maximization Key Contributions Cost focused placement solution Bounded approximation algorithms & dynamic prog. solution

84 Common Patterns Understand workloads How workloads combine

85 Placement Formulation Each server has a operating costs. Place tenants to minimize costs (occasional violations OK). Two problem formulations:

86 Solution

88

90 Kairos Focus Modeling resource consumption of OLTP workloads Consolidate workloads Key Contributions Method to determine active working set size Model disk I/O for consolidation Find balanced consolidation plan.

91 Buffer Pool Gauging for RAM 953 MB Bufferpool, on TPC-C 5W ( MB/WH) Slide by Sam Madden

92 Disk Model 115

93 Node Assignment via Optimization Problem modeled as: Mixed-integer non-linear optimization problem Implemented in DIRECT non-linear solver; several tricks to make it go fast Slide by Sam Madden

95 118

97 DBSeer Focus Attribute resource consumption to txn classes (and tenants) Attribute at runtime in consolidated process Build models of various DB resources Key Contributions Models for disk I/O, locks, throughput, etc Attribute resources to tenants. Ability for DBAs to play what-if

98 DBSeer From ft What-if questions

99 Transaction Clustering SQL Logs time connection sql stmt 1:92 C1 BEGIN TRANSACTION 1:93 C2 SELECT * FROM. New Order Payment 1. Extract features of each transaction - number of rows read/written to each table 2. Run DBSCAN clustering algorithm Build Access Distributions Delivery

100 Predicting Disk I/O Disk Reads = Disk Writes = Key Observation: Predict # of dirty pages

101 Other Components of DBSeer

103 131

105 Pythia Focus Tenant workloads are unknown, disk-based, and dynamic Use supervised learning to model tenants and colocation Leverage models to resolve performance crisis Key Contributions Method for empirically learning how tenant classes colocate End to end framework for tenant placement

106 Tenant Model

107 Describe Resource Consumption Disk Heavy Disk Medium Disk Light CPU Heavy CPU Light T Feature 1 Feature 2 Feature 3 ( ) =

108 Learn which classes colocate well Boundaries set by administrator. Under Good Over Uses resources and latency SLOs. Control over consolidation. Incrementally learned through observation

109 Things Don t Always Go To Plan node node model

110 Searching for a solution.

112 Migration for Load Balancing

113 Migration Forms

114 Migration Goals

115 Albatross [Das et al. VLDB 2011] Focus Live migration in a shared storage transactional DB Migration TM state and cache Key Contributions First live migration for shared storage. Minimal strain on destination

116 Live Migration for Shared Storage BMS Node Cached DB State Transaction State Cached DB State Transaction State Tenant/DB Partition Source Destination Tenant/DB Partition ersistent mage 147

117 Albatross Live Migration Ownership Source (N src ) Destination (N dst ) Steady State Time 1. Begin Migration 2. Iterative Copying 3. Atomic Handover Steady State Initiate Migration Snapshot cache at N src Initialize tenant at N dst N src continues executing transactions Synchronize and Catch-up Track changes to DB State at N src Iteratively synchronize state changes 148 Finalize Migration Stop serving Tenant at N src Synchronize cache Migrate transaction state Transfer ownership to N dst

118 Zephyr [Elmore et al. SIGMOD 2011] Focus Live migration in a shared nothing transactional DB (H2) No heavy-weight synchronization protocols or replication. No downtime, some aborted transactions. Key Contributions First live migration for shared nothing DBMS. Minimal strain on source (scale up)

119 Init Mode Freeze index wireframe and migrate P 1 P 1 P 2 P 2 Owned Pages P 3 P 3 Un-owned Pages P n P n Active transactions T S1,, T Sk Source Destination Page owned by Node Page not owned by Node

120 Dual Mode Requests for un-owned pages can block P 1 P 2 P 3 P 3 accessed by T Di P 1 P 2 P 3 P n P 3 pulled from source P n Old, still active transactions T Sk+1,, T Sl T D1,, T Dm New transactions Source Destination Index wireframes remain frozen Page owned by Node Page not owned by Node

121 Finish Mode Pages can be pulled by the destination, if needed Completed P 1 P 2 P 3 P n Source P 1, P 2, pushed from source P 1 P 2 P 3 P n T Dm+1,, T Dn Destination Page owned by Node Page not owned by Node

122 Cut Me Some Slack : Latency-Aware Live Migration for Databases [Barker et al. EDBT 2012] Focus Interference aware live migration Key Contributions Throttles migration to minimize impact Implementation with no internal modification

123 Slacker Approach

124 ProRea Live Database Migration for Multi-tenant RDBMS with Snapshot Isolation [Schiller et al. EDBT 2013] Focus Overcome some Zephyr shortcomings Key Contributions A proactive and reactive live migration

125 ProRea - Approach

126 In Closing 157

127 Many Other Issues 158

128 Future Challenges Additional resource isolation controls SLOs / SLAs Data sharing Better workloads Analytics

129 Thanks!