INFO5011 Advanced Topics in IT: Cloud Computing Week 10: Consistency and Cloud Computing

Transcription

1 INFO5011 Advanced Topics in IT: Cloud Computing Week 10: Consistency and Cloud Computing Dr. Uwe Röhm School of Information Technologies! Notions of Consistency! CAP Theorem! CALM Conjuncture! Eventual Consistency! Properties! Dynamic Data Placement Outline! Data Consistency Properties and the Trade-offs in Commercial Cloud Storages INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-2

2 Revisit: The CAP Theorem Consistency [Brewer, PODC2000] Availability Partitioning Tolerance! Theorem: You can have at most two of these properties for any shared-data system. INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10v-3 Notions of Consistency! Strong Consistency (aka 1-copy-Serializability)! behavior as if there is only one copy of each data item, and! only serializable accesses permitted! Weak Consistency! No guarantee that a subsequent (read) access from this or any other client will return a just updated value! Might mean several versions of data (i.e. not 1-copy), might mean not serializable! Updates will be propagated eventually to all replicas after some delay ( inconsistency interval ) => Eventual Consistency INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-4

3 Eventual Consistency! A model originally proposed for disconnected operation (e.g., mobile computing)! Different nodes keep replicas and each update is eventually propagated to each replica! And eventually, there is agreement on which update is the latest! DNS is the most well-known system implementing eventual consistency! Usual definition is counterfactual: once updating ceases, and the system stabilizes, then after a long enough period, all replicas will have the same value INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-5 Why Eventual Consistency?! Argument 1: Availability is King, Network Partitioning is Fact! Cloud infrastructure, especially the lowest level (data storage) has a crucial always on requirement! But networking can fail (temporarily) even within one rack or between different racks of same data center! Algorithms for strong consistency would block till network is up again => seen as unacceptable! Argument 2: Latency Penalty and Costs too high! In some applications, there s replication between different data centers needed. Having this synchronous (strong consistent) would impose huge performance penalty! Typical latency within one data center: 2-3 ms! Typical latency between continents: >100ms! Also: Replication needs bandwidth which translates into costs INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-6

4 Example from Yahoo! [VLDB2011]! Scenario: a social networking application that uses this Figure distributed 1: database Globally replicated database that asynchronously distributed database, propagates with replicas updateskept to in-sync remotevia datacen- an! ters. asynchronous replication mechanism! inter data center communication needed! east Users coast typically of theshow U.S., some and inlocality France. which For should clarity direct in ourthe discussion, replication we mechanism will focus on to a avoid single costs table and containing improve perf. records, but our techniques generalize directly to multiple tables or INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-7 other data models. Each replica location stores a full or partial copy of the table. Because of the high latency for communicating between datacenters, replication is typically done asynchronously. Usually, Sidenote: writes are Partial persistedreplication one or more localto servers save and acknowledged Bandwidth to the while applications increasing (e.g., made 1-safe). Latency Later, updates are sent to other replica locations. An example of! Goal: replication algorithm with low bandwidth and low latency this architecture is shown in Figure 1. As the figure shows,! we Side-Constraints: can think of the system as having two distinct components:! Latency a database SLAs system, (at least which local access manages must reads be read and fast) writes of data! Legal records, Constraints and a(not replication all data is system, allowed to which be stored manages everywhere) replication Approach: of updates between replica locations. In real systems! these! Asynchronous, componentsprimary-copy might be on replication the same server (as in MySQL replication! Any write [3]) is or guaranteed different to be servers done at (as the master in PNUTS first [9]). The replication! Partial replication system must ensure reliable delivery of updates to remote! full-copies datacenters vs. stubs despite failures. Individual servers might! fail Read-everywhere, (and even but lose if stub, data), it is actually but local a remote or read remote (latency copies penalty) can! Policy be used Constraints for recovery. that define In! each the allowable location, and mandatory a given locations recordfor exists full replicas either of each as record, a fulland replica! the orminimum as a stub. number Aof full replicas for is each a normal record copy of the record,! Dynamic possibly placement enhanced algorithm with that metadata takes local reads/writes to supportinto selectivefor replication, promoting stubs suchto as full acopies list of and other vice versa full replicas. A stub account INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-8 contains only the record s primary key and metadata, but no data values. Note that we do not consider selective replication at the field or column level in this paper. are only updated warded scheme, is delete notify th A rec databas wise, th the stub ably the penalty been ser now nee sponse l As we i forward It may example promote replicas promoti terns, b This allo number convert replicas forward data mu 3.2 O Interespecial ity. The width u tant. H ing send duce the bandwid Inter- Repl

5 Dynamic Placement! Dynamic demotion/promotion of stubs! Reading a stub: stub promoted to full replica! Update on replica: if after retention interval -> stub INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-9 Extra Properties for Eventual Consistency! Programming an application is much harder if storage supports only eventual consistency! What to do until everything settles down??! Handling inconsistency in the sequence of reads! Cf Hellerstein et al PODS 10/CIDR 11: eventual consistency data model supports monotonic programs (a very limited class)! A range of extra properties, which (if the storage provides this) can make programming not quite so hard! e.g., read-your-own-writes, monotonic reads, INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-10

6 Properties of Eventual Consistency! Causal Consistency! If client A has communicated with client B that an item has been updated by A, B will see the updated value of A! Read-your-writes Consistency! Special form of Causal Consistency for A=B! Session Consistency! Practical version of Read-your-writes Consistency that guarantees this property just within one session, but not between separate sessions! Monotonic Read Consistency! A subsequent read will never return an older version of a data item than a previous read! Monotonic Write Consistency! Serializability just for writes INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) The CALM Conjuncture: Consistency and Logical Monotonicity! Observation 1: monotonic eventually consistent! Observation 2: coordination at every nonmonotonic operation eventual consistent! Conjecture: non-monotonic and uncoordinated inconsistent [Hellerstein, PODS 2010 Keynote] INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-12

7 State-of-the-Art with Current Offerings for Cloud Storage! CIDR 2011 Paper: Data Consistency Properties and the Trade-offs in Commercial Cloud Storages: the Consumers Perspective! The following slides are from this talk, available in the original from INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) CIDR 2011: Consistency from the Consumer s Perspective! Paper investigates consistency models provided by commercial cloud storages! If weak consistency, which extra properties supported?! How often and in what circumstances is inconsistency (stale values) observed?! Any differences between what is observed and what is announced from the vendor?! Investigation of the benefits for consumer of accepting weaker consistency models! Are the benefits significant to justify consumers effort?! When vendor offers choice of consistency model, how do they compare in practice? INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-14

8 Observed Platforms! A variety of commercial cloud NoSQLs that are offered as storage service! Amazon S3! Two options: Regular and Reduced redundancy (durability)! Amazon SimpleDB! Two options: Consistent Reads and Eventual Consistent Reads! Google App Engine datastore! Two options: Strong and Eventual Consistent Reads! Windows Azure Table and Blob! No option available in data consistency INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) Frequency of Observing Stale Data! Experimental Setup! A writer updates data once each 3 secs, for 5 mins! On GAE, it runs for 27 seconds! A reader(s) reads it 100 times/sec! Check if the data is stale by comparing value seen to the most recent value written! Plot against time since most recent write occurred Execute the above once every hour On GAE, every 10 min For at least 10 days Done in Oct and Nov, 2010 INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-16

9 SimpleDB: Read and Write from a Single Thread! With eventual consistent read, 33% of chance to read freshest values within 500ms! Perhaps one master and two other replicas. Read takes value randomly from one of these? First time for eventual consistent read to reach 99% fresh is stable 500ms Outlier cases of stale read after 500ms, but no regular daily or weekly variation observed INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) Stale Data in Other Cloud Data Stores Cloud NoSQL and Accessing Source SimpleDB (access from one thread, two threads, two processes, two VMs or two regions) S3 (with five access configurations) GAE datastore (access from a single app or two apps) What Observed Accessing source has no affect on the observable consistency. Eventual consistent reads have 33% chance to see stale value, till 500ms after write. No stale data was observed in ~4M reads/config. Providing better consistency than SLA describes. Eventual consistent reads from different apps have very low (3.3E -4 %) chance to observe values older than previous reads. Other reads never saw stale data. Azure Storages (with five access configurations) No stale data observed. Matches SLA described by the vendor (all reads are consistent). INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-18

10 Additional Properties: Read-Your-Writes?! Read-your-writes: a read always sees a value at least as fresh as the latest write from the same thread/session! Our experiment: When reader and writer share 1 thread, all reads should be fresh! SimpleDB with eventual consistent read: does not have this property! GAE with eventual consistent read: may have this property! No counterexample observed in ~3.7M reads over two weeks INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) Additional Properties: Monotonic Reads?! Monotonic Reads: Each read sees a value at least as fresh as that seen in any previous read from the same thread/session! Our experiment: check for a fresh read followed by a stale one! SimpleDB: Monotonic read consistency is not supported! In staleness, two successive eventual consistent reads are almost independent! The correlation between staleness in two successive reads (up to 450ms after write) is , which is very low 1 st Stale 39.94% (~1.9M) 1 st Fresh 23.36% (~1.1M) 2 nd Stale 2 nd Fresh 21.08% (~1.0M) 15.63% (~0.7M)! GAE with eventual consistent read: not supported! 3.3E -4 % chance to read values older than previous reads INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-20

11 Additional Properties: Monotonic Writes?! Monotonic Writes: Each write is completed in a replica after previous writes have been completed! Programming is notoriously hard if monotonic write consistency is missing! W. Vogels. Eventually consistent. Commun. ACM, 52(1), 2009.! This is an implementation property, not directly visible to consumer. But we explore what happens when we do successive writes, and try to read the data INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) SimpleDB s Eventual Consistent Read: Monotonic Write! A data has value v0 before each run. Writing value v1 and then v2 there, then read it repeatedly v1!= v2 v1 = v2 When v1!= v2, writing v2 pushes v1 to replicas immediately (previous value v0 is not observed) Very different from the only writing one value case When v1 = v2, second write does not push (v0 is observed) Same as the only writing one value case INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-22

12 SimpleDB s Eventual Consistent Read: Further exploration -- Inter-Element Consistency SimpleDB s Data Model Domain Item Attribute Value SimpleDB s API Write a value Write multiple values in an item Write multiple values across items in a domain Read a value Read multiple values in an item Read multiple values across items in a domain! Consistency between two values when writing and reading them through various combinations of APIs Reading two values independently Writing two at once and reading two at once Writing two in the same domain independently INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) Each read has 33% chance of freshness. Each read operation is independent Both are stale or both are fresh. Seems batch write and batch read access to one replica The second write pushes the value of the first write (but only if two values are different) 10- Trade-Off Analysis of SimpleDB: A Benefit for Consumer from Weak Consistency? No significant difference was observed in RTT, throughput, failure rate under various readwrite ratios If anything, it favors consistent read! Financial cost is exactly same * Each INFO5011 client "Cloud sends Computing" 1 rps. All obtained (U. under Röhm 99:1 and Y. read-write Zhou) ratio 10-24

13 What Consumers Can Observe (as of the state of these experiments)! SimpleDB platform showed frequent inconsistency! It offers option for consistent read. No extra costs for the consumer were observed from our experiments! At least under the scale of our experiments (few KB stored in a domain and ~2,500 rps)?? Maybe the consumer should always program SimpleDB with consistent reads? INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) What Consumers Can Observe (contd) (as of the state of these experiments)! Some platforms gave (mostly) better consistency than they promise! Consistency almost always (5-nines or better)! Perhaps consistency violated only when network or node failure occurs during execution of the operation?? Maybe the chance of seeing stale data is so rare on these platforms that it need not be considered in programming?! There are other, more frequent, sources of data corruption such as data entry errors! The manual processes that fix these may also be used for rare errors from stale data INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-26

14 Implications of these Experiments for Consumers?! Can a consumer rely on our findings in decision-making? NO!! Vendors might change any aspect of implementation (including presence of properties) without notice to consumers.! e.g., Shift from replication in a data center to geographical distribution! Vendors might find a way to pass on to consumers the savings from eventual consistent reads (compared to consistent ones)! The lesson is that clear SLAs are needed, that clarify the properties that consumers can expect INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) Summary! Strong Consistency! is most desirable for consumers, but seen as not achievable with network partitioning tolerance! Cf. CAP Theorem! Eventual Consistency:! Allows nodes to disagree on current data value! But current algorithms/systems differ widely in how this is achieved! Current Cloud Storage Implementations! Provide different variants of eventual consistency without disclosing the implementation details or clear SLAs! Currently missing SLAs (observable, but no guarantees):! Rate of inconsistent reads, time to convergence, performance under variety of workloads, availability, costs INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-28

15 References! Werner Vogels: Eventually Consistent. Communications of the ACM, Volume 52, No. 1, Jan 2009.! H. Wada, A. Fekete, L. Zhao, K.. Lee and A. Liu: Data Consistency Properties and the Trade-offs in Commercial Cloud Storages: the Consumers Perspective. In CIDR 2011.! Sudarshan Kadambi, Jianjun Chen, Brian F. Cooper, David Lomax, Raghu Ramakrishnan, Adam Silberstein, Erwin Tam, Hector Garcia- Molinn: Where in the World is My Data? In: VLDB 2011.! Joseph M. Hellerstein: The Declarative Imperative Experiences and Conjunctures on Distributed Logic. SIGMOD Record 39:1, March INFO5011 "Cloud Computing" (U. Röhm and Y. Zhou) 10-29