CIT 668: System Architecture

Transcription

1 O t p s 1 t p s 4/12/2011 CIT 668: System Architecture Distributed Databases Topics 1. Scaling databases 2. Concurrency 3. Database replication 4. Replication architectures 5. Database partitioning Database Scaling Techniques B a s e c a s e a 1 T P S s y s te m S c a le u p to a 2 T P S c e n tra liz e d s y s te m U s e rs 1 T P S s e rv e r U s e r s 2 T P S s e rv e r P a r t it io n in g T w o 1 T P S s y s te m s R e p lic a t io n T w o 2 T P S s y s te m s U s e rs 1 T P S s e rv e r O t p s U s e r s 2 T P S s e rv e r 1 t p s U s e rs 1 T P S s e rv e r U s e r s 2 T P S s e rv e r 1

2 Distributed System Types Shared Memory Shared Disk Shared Nothing All CPUs share memory/disk Scalability limited by memory contention (vertical scaling only) CPUs share storage, not RAM Scalability limited by disk contention (vertical scaling only) Each CPU has its own RAM and disks Very high (horizontal) scalability since no contention for shared resources Concurrency The classic example Code to withdraw funds from bank account withdraw(account, amount) { balance = get_balance(account); balance -= amount; put_balance(account, balance); return amount; } What happens if you setup automatic bill pay and two withdrawals are made simultaneously? 2

3 Create a separate thread for each withdrawal, each running the same code. withdraw(account, amount) { balance = get_balance(account); balance -= amount; put_balance(account, balance); return amount; } withdraw(account, amount) { balance = get_balance(account); balance -= amount; put_balance(account, balance); return amount; } Interleaved schedules Execution of the two threads can be interleaved, with preemptive scheduling: Execution sequence as seen by CPU balance = get_balance(account); balance -= amount; balance = get_balance(account); balance -= amount; put_balance(account, balance); put_balance(account, balance); context switch context switch What s the account balance after this sequence? Race Conditions A race condition is a bug in which the result of a process depends on the sequence or timing of other events. 3

4 Mutual Exclusion To synchronize access to shared objects, we can use mutual exclusion. Code that uses mutual exclusion to synchronize its execution is called a critical section, which is a section of code such that: 1. Only one thread at a time can execute in the critical section. 2. All other threads must wait to enter the section. 3. When a thread leaves the critical section, another thread can enter. Critical section requirements Mutual exclusion At most one thread is in the critical section. Progress If a thread is outside the critical section, it cannot prevent another thread from entering the critical section. Bounded waiting If a thread is waiting on the critical section, it will eventually enter the critical section. Performance The cost of entering and leaving the critical section is small with respect to the work done within it. Locking A lock is an object that provides two operations: acquire(): thread calls this before entering critical section release(): thread calls this after leaving critical section A thread holds the lock btw acquire() and release(). 4

5 Deadlocks Deadlocks A deadlock is a situation where two or more actions are waiting for the other to finish, and thus neither ever completes. Database replication 5

6 Purposes of Replication Data distribution Maintains a copy of DB at another geographic site to lower latency at site or for DR. Load balancing Allows application to access data on multiple servers. Backup and recovery Backups of replicated DB can be performed without impacting performance of original production DB. High availability Application can failover to replicated DB. ACID Properties Atomicity All data modifications within a transaction must happen completely or not at all. No partial transactions can be recorded. Consistency All changes to an instance of data must be reflected in all instances of that data. Isolation The Elements of a transaction should be isolated to the user performing the transaction until it is completed. Durability When a system failure occurs, the data in the DB must be accurate up to the last committed transaction before failure. Replication techniques Eager (synchronous) All replicas updated as part of original transaction. Data is always consistent, but ensuring data consistency can lead to long waits or deadlocks. Lazy (asynchronous) Original transaction completes on node, then updates propagated to other nodes as separate transactions. Can result in conflicts when transactions modify same object on different nodes before replicas are updated. Must have reconciliation protocol to resolve conflicts. 6

7 Eager and Lazy Transactions A s in g le -n o d e T ra n s a c tio n W rite A W rite B W rite C C o m m it A th re e -n o d e Eager Transaction W rite A W rite A W rite A W rite B W rite B W rite B W rite C W rite C W rite C C o m m it C o m m it C o m m it A th re e -n o d e Lazy Transaction (actually 3 Transactions ) W rite A W rite B W rite C C o m m it W rite A W rite B W rite C C o m m it W rite A W rite B W rite C C o m m it Replication architectures Master/Slave Replication Slave DBs only accept read operations for application. Flickr.com DBs logged 13 SELECTS for each write. Master DB Accepts write operations. Copies write operations to slave DBs. Single point of failure! 7

8 Load-Balanced Master-Slave Architecture Pyramid Master/Slave Architecture Flickr Database Architecture 2006 Databases use MySQL s InnoDB tables for speed. Data replicated to search slaves, which change table type to MyISAM to get full text search. 8

9 Master/Slave Replication Scales reads, not writes. Good for read-centric applications. Master is a single point of failure. Difficult to reparent slaves on master failure. Multimaster Replication 1. All data is located at multiple nodes. 2. All nodes can process any transaction. ACID problems: Atomicity: transactions must commit successfully on all nodes or must fail on all nodes. Consistency: requires a global ordering of commits across the cluster. Durability: must wait for data to be written to disk on all nodes. Two-phase commit 1. Request phase: transaction coordinator asks nodes to try to commit. Nodes execute transaction up to point where they will be asked to commit. Each node returns commit/abort. 2. Commit phase: if any node returns abort, coordinator tells all nodes to rollback; if all nodes return commit, then coordinator tells all to commit and nodes send acknowledgement to coordinator when done. 9

10 Two-phase Commit Problems Coordinator failures If coordinator goes down, nodes can block indefinitely, waiting for commit request. Node failures If a node fails, coordinator can wait forever to get unanimous commit decision. If coordinator can mark a node as down and ignore its vote, then nodes must be resynchronized when are once again available. Combined failures If both a node and the coordinator go down after coordinator starts sending commit messages, then it is possible the down node committed, but no other node received the message. Dual-Master Replication for HA Dual Master Replication High reliability Two identical copies of DB. Easy maintenance Set only one DB to be active. Update inactive server. Synchronize. Flip to other DB as active one, then update it. 10

11 MySpace Case Study 3000 web servers 800 cache servers 440 database servers hosting >1000 databases Each DB server has 4 2-core CPUs + 64GB RAM Database Partitioning Partitioning A partition is a division of a logical database into distinct independent parts. Horizontal partitioning divides the database by rows, with groups of rows stored on diff nodes. Horizontal scaling is highly scalable. A horizontal partition is called a shard. Vertical partitioning divides the database by columns, with sets of columns stored on different nodes. Vertical scaling is limited by the number of columns that are accessed independently. 11

12 Partitioning Criteria Range partitioning selects a partition by determining if the partition key is within a certain range. List partitioning assigns each partition a list of values. Hash partitioning uses a hash function to determine partition membership. Wikipedia Shard Architecture LiveJournal Sharding Architecture

13 Sharding Advantages Faster Queries Since each shard has a fraction of the whole DB, queries are faster than they would be on whole. Higher Write Bandwidth Can have master/slave configuration for each shard, so each shard has its own dedicated master to do writes. High Scalability Can continue to divide DB into more shards to scale out indefinitely. Sharding Disadvantages Rebalancing To scale, you need to split shards, which can require substantial manual effort and downtime. Google and Flickr s shards auto-rebalance, which requires a mechanism to invalidate references, so underlying data can be moved while in use. Cross-shard joins are slow If a request requires data from multiple shards, it s slower than accessing a single DB. Social networks need to find relationships among as many dimensions and users as possible. Poor support Less mature tools and documentation than replication. Key Points Database scaling techniques 1. Use caching to avoid need to scale DB 2. Scale up with faster hardware 3. Replicate DB to multiple servers 4. Split DB across multiple servers (partitions) Concurrency Critical section, lock, deadlock Replication Master/slave scales reads but not writes; has SPOF Multi-master either loses ACID or doesn t scale well Partitioning Partition tables by rows and put shards on diff nodes. 13

14 References 1. John Allspaw, The Art of Capacity Planning, O Reilly, Brad Fitzpatrick, LiveJournal: Behind the Scenes, USENIX, Jim Gray et. al. The Dangers of Replication and a Solution. In Proceedings of the 1996 ACM SIGMOD international conference on Management of data (SIGMOD '96). 4. James Hamilton, Scaling at MySpace, Cal Henderson, Building Scalable Web Sites, O Reilly, Cal Henderson, Flickr and PHP presentation, Theo Schlossnagle, Scalable Internet Architectures, Sams Publishing, Wikipedia, Server Layout Diagrams, 9. Jeremy D. Zawodny and Derek J. Balling, High Performance MySQL, O Reilly,