Infrastructure Technologies for Large- Scale Service-Oriented Systems. Kostas Magoutis

Transcription

1 Infrastructure Technologies for Large- Scale Service-Oriented Systems Kostas Magoutis

2 SMARTER: High performance RSMs Is Paxos too expensive to use in a high-performance replicated data store? How does it compare to commonly used primarybackup replication schemes?

3 Paxos protocol

4 Gaios architecture Sparse array of bytes 128-bit GUID Create, delete, read, write, truncate Checksums (CRC-32) State machine state SMARTER internal state Record Paxos proposals

5 SMARTER Changes over SMART Pluggable interface for storage and log providers Zero-copy data path (ideally single-write) I/O prefetching at proposal time Batch client operations Parallel network transport Handle hardware errors and non-determinism More efficient protocol for read-only requests Operations Client, leadership, interaction with logging and RSM Managing RSM state and sending replies to clients View change, state transfer, log trimming

6 SMARTER Pipelining requests Depth a = 6 Batching requests Combine several client requests into proposal for a single free slot Leader does not need to be among majority that has logged the proposal Run at speed of median member Majority sets for different operations may differ When leader s network bandwidth is scarce, fall into chain replication

7 Local stream store (LSS) Maps stream to single, large file in NTFS Non-cached I/O Batch-mode writes Atomically checkpoints entire LSS state upon request Overlaps checkpointing with in-memory operation execution Only brief pause when checkpointing is initiated Checksums CRC32: 4 bytes for each 4KB of data (0.1% overhead) Store separately from data Separate I/Os needed to protect against misdirected I/O Checksums written next to each other and accessed in batch Deterministic free space

8 Minimize duplicate writes to disk Original SMART wrote same data to disk four times Paxos log was implemented as SQL table State was implemented as another SQL table Implemented proper Paxos proposal log Removed one write op Left state transaction log and in-place state update Combined proposal log and transaction log to single write Transaction log contains pointers to proposal log Valid only when proposal was agreed upon Single-write system would require a write-once store

9 Efficient disk request processing Disks work best when presented with batch of I/Os E.g., SCAN algorithm Read-write requests Writes exclusively into in-memory cache Cache cleaned at checkpoint time in disk efficient order Read-only requests Ordinarily run on only one replica As they arrive, they are re-ordered and sent to disk Executed when disk reads complete, in any order

10 Read-write requests ACID Atomicity: logging requests Durability: waiting for log writes to complete before replying to client Isolation/Consistency: One request at a time in RSM Prefetch API Inspect an operation prior to execution, at propose time Prefetch if needed (no harm if proposal not agreed) Ordinarily Reads will hit in the cache, writes will not block Memory pressure may necessitate some blocking

11 Read-only processing Five key techniques Execute a particular read op on only one replica, guaranteeing read of last committed write seen by any client Does not require clock synchronization (leases) Use agreement protocol that does not require logging Reorder reads into disk-efficient schedule Spread reads across replicas to leverage disk arms Direct reads away from replicas that checkpoint an LSS

12 Consistency A read-only op should execute after any read/write operation that completed before the read was issued The state against which the read is run must reflect any operation that any client has seen as completed It may or may not reflect subsequent writes Client Read-only request L: a leader WHATS_MY_VIEW (there may be another leader, that has committed most recent write) Stamp with S = max of Highest op # committed in sequence by this leader Highest op # leader reproposed when view begun Wait for at least half of all replicas to ack L is their leader Dispatch read-only request to a replica R, include slot # S R waits for S to execute; if still in same configuration, exec read

13 Non-determinism RSM assumes state machines are deterministic Cannot use replica local time (rtc) Other causes of non-determinism: memory allocation failures, hardware faults on disk, network or memory Replica inconsistencies may be handled as Byzantine faults Gaios converts Byzantine faults to restarts or crash failures

14 Experiments 3 x AMD Opteron 2216 (2 dual-core CPUs) old 1 x AMD Opteron 2216 (4 dual-core CPUs) client 2 x AMD Opteron 2382 (2 quad-core CPUs) new Windows Server 2008 R2 Enterprise Edition Cisco Catalyst 3560G gigabit switch

15 Simulating Primary-Backup Reads dispatched without quorum check Read/write operation quorums must include leader All read/write replies come only from the leader Useful only for experimentation (missing leases) Consequences of reading from single leader may be mitigated by sharding data over replica groups Worst case (PB1): read from only a single node Best case (PBN): sends reads to all nodes

16 Random I/O performance sqlio over 10GB Gaios drive 8KB random read/writes 35 concurrent readers Gaios write, PW write Gaios read, PBN read PB1 read Old servers New servers

17 OLTP 64-bit MS SQL Server 2008 Enterprise Edition Order-entry workload, ~50-50% reads/writes 3GB log file, 53GB table file on separate disks I/O bound, negligible CPU load on SQL Server Unable to direct reads away from checkpointing nodes Unable to extract read parallelism