9.6 Multimaster replication

Transcription

1 9.6 Multimaster replication "All citizens are equal" D D D D D May results in unsolvable conflicts Detected when updates are propagated Need auxiliary data which reflect the update of object x at different sites Not only an academic study: in some applications data may be updated according to geo location: "Employees in Berlin / New York ". Updates primarily at home location. 01-TA-Repl- 47

2 Multimaster Also a multimaster scenario, if the disconnected devices may update independently. Master DB DB Master responsible for shipping updates it learns from replicas to all other replicas DB DB Disconnected copies More general case: every replica can synchronize at any time with any other replica (node). 01-TA-Repl- 48

3 Versions and ordering Independent updates and synchronisation -----> x x'' x' v Repl R x' No problem: versions of x follow each other (happens before, precedence relation) x''' Repl S Update and sync anywhere at any time conflict if there was a version which has been overwritten independently by two (replication) nodes. 01-TA-Repl- 49

4 System model Transactions read / write at arbitrary replica No abort (simplifying assumption) Only the first update of an object x in a TA defines version id Objects may be tables, files, rows... Version id identifies last value of x written by TA Since x and y updated by R are causally related: Version of x is update count of R e.g [R,7] is version of R, [R,8] is version of y if x and y updated subsequently 01-TA-Repl- 50

5 Version id Version id for objects x [R,updateCount] not sufficient for ordering :... x8 y9 x7 R S replica How do y9 and x10 compare?? Which order?? How can conflicts be detected? 01-TA-Repl- 51

6 Multimaster Sync Task Find data structures and sync algorithms which allow to detect conflicts, i.e. There are transactions T1 at R and T2 at S which have not seen the output of each other, but produced a new version. Order of versions xi directly precedes xj if a) there are TA t1, t2 and t1 reads x and writes xi, t2 reads xi and writes xj b) xi precedes xj if there xi directly precedes xj or there is a sequence of versions xi,xi+1,...,xj and xi directly precedes xi+1,... (transitive closure) 01-TA-Repl- 52

7 Multimaster: : data structures For each replica Ri a version ID vector: [[R1,c1],...[Rn,cn]] : number of updates at Ri from every other replica Ri has received. Reflects updates R received from the other replica. and an update count [R, c]. Node Ri ordered version vector = vector of update counts e.g. R,S,T, R has update count 8, R version vector: [8, 5,10] has seen all but one S : [4, 5, 9] update of T, and all of S etc T : [5, 4, 11] Each data item x has a version id [ Ri,c] : x has been Ri with update count c. 01-TA-Repl- 53

8 Multimaster Sync Let R, S be nodes with version vectors VR =[c1,...,cn], VS = [d1,...dn] If R wants to synchronize with S, (1) R sends VR to S, (2) S sends VS and all updates of all objects x which satisfy: let VR[i] = k, version id of x = [Ri,c] and k < c... because R has not seen the update of x made by Ri (3) R updates its version vector an objects x received, if no conflict! 01-TA-Repl- 54

9 Multimaster Partial order on version vectors : VR < VS if for all i VR[i] VS[i] VS < VR if for all i VS[i] VR[i] else incomparable. Update rules (1) TA t executes at R with update count [R,c]. For each modified x gets version [R,c]; c++ (2) Sync: sending x from S to R... (3) conflict? Goal of the rules: if version xi overwrites version xj then xj precedes xi 01-TA-Repl- 55

10 Multimaster Sync Update rules (cont) (2) x sent from S to R, let version id of R [Rk, d] version id of S [Ri, c] If VR[i] > c then discard the version of x sent (since R has received from Ri already 'higher' update) If VS[k] > d then replace x with version received from S with version id [Ri,c] (since S received version of x produced by Rk before R) Update version vector. (3) VR and VS incomparable: conflict 01-TA-Repl- 56

11 Multimaster / version vectors Conflicts must be resolved by application Except for some possible strategies: last update wins,... Better solution: Replica retain conflicting updates (versions of x ) and present them to application. Correctness of replica update? Easy to see with version vectors for each object (!) More subtle only with version of object and version vector of replica. Show that goal of the rules achieved: xi overwrites version xj only if xj precedes xi 01-TA-Repl- 57

12 Example Conflict situation: x has been updated independently by R1 and R2 Example by Bernstein / Newcomer 01-TA-Repl- 58

13 Example R3 receives T2's update and it can tell whether it ran before or after R2 received T1's update if version vectors are used. 01-TA-Repl- 59

14 9.7 Replication in the real world Typically simpler solutions oriented towards most important scenarios Asynchronous mode Terminology of vendors differs Typical global architecture: [source:oracle] Data changes are captures and staged to target for consumption 01-TA-Repl- 60

15 Important scenarios High availability Master e.g. msg. queue, or redo log device Hot standby Supported by most systems; Oracle: specific multi-master configuration (both [a]synchronous) Might be synchronous, but would slow down TA processing at master Transfer of log data, replay at standby Insert into message queue may be part of TA at primary No TA lost Take over within seconds needed for replay of pending TAs 01-TA-Repl- 61

16 Scenario: scaling Scaling of read workload TA / command log device, Msg queue MySQL replication: - slaves read command master - restart of slave: use numbering of commands Primary copy/ master Read only copies/ slaves Oracle: Read-only materialized view Low update traffic, unidirectional refresh, failure of slave slight read performance decrease 01-TA-Repl- 62

17 Scenario: Clients with update right Typical situation: Mobile clients Oracle: - updatable materialized view Low update traffic, bidirectional refresh, frequently trigger-based update on both sides, acceptable if low update rate, e.g. msg queue based communication, conflicts may have to be solved manually 01-TA-Repl- 63

18 Replication Manager Dedicated server for coordination replication specific tasks IBM: "Data Propagator" Sybase "Replication server" MS: "SQL Server Synchronization Mgr" Oracle "Replication Mgr" (Siebel) Different types of data refreshment policies Different kind of technical data exchange, e.g. msg-queues, publish, subscribe etc Replication server ("staging server") Typically hierachically structured 01-TA-Repl- 64

19 Oracle 8i Can replicate updates to table fragments or stored proc calls at the master copy Uses triggers to capture updates in a deferred queue Updates are row-oriented, identified by primary key Can optimize by sending keys and updated columns only Group updates by transaction, which are propagated: Either serially in commit order or in parallel with some dependent transaction ordering: each read reads the commit number of the data item; updates are ordered by dependent commit number Snapshots (= materialized view) are updated in a batch refresh. Pushed from master to snapshots, using queue scheduler 01-TA-Repl- 65

20 Oracle replication overall picture slide by G. Alonzo, ETH Very flexible solution, (nearly) everything allowed! not shown (and not required!?): replication manager 01-TA-Repl- 66

21 Multimaster Peer-to-Peer Replication keeps all copies up to date transactional guarantees How? Conclusion from experiments and talks and personal communication: table locks (!) May be ok in particular situations, but in general? 01-TA-Repl- 67

22 Multimaster replication: peer-to-peer Multi-master replication without a primary: Wingman Each row of a table has 4 additional columns globally unique id (GUID) generation number, to determine which updates from other replicas have been applied version number = a count of the number of updates to this row array of [replica, version number] pairs, identifying the largest version number it received for this row from every other replica. Used in Microsoft Access 7.0 and Visual Basic 4.0 adapted from Phil Bernstein 01-TA-Repl- 68

23 Multimaster replication: "MS-Wingman" Each replica has a current generation number A replica updates a row s generation number whenever it updates the row A replica remembers the generation number it had when it last exchanged updates with R, for every replica R. A replica increments its generation number every time it exchanges updates with another replica. So, when exchanging updates with R, it should send all rows with a generation number larger than what it had when last exchanging updates with R. adapted from Phil Bernstein 01-TA-Repl- 69

24 Wingman update processing Use Thomas Write Rule to process an update from another replica Compare the update s and row s version numbers The one with larger version number wins (use replica id to break ties) Yields the same result at both replicas, but maybe not serializable 01-TA-Repl- 70

25 Wingman: not serializable Suppose two replicas perform updates to x Replica A does 2 updates, incrementing version number from 1 to 3 Replica B does 1 update, incrementing version number from 1 to 2 When they exchange updates, replica A has higher version number and wins, causing replica B s update to be lost For this reason, rejected updates are retained in a conflict table for later analysis 01-TA-Repl- 71

26 Wingman: rejecting duplicate update Some rejected updates are duplicates To identify them - When applying an update to x, replace x s array of [replica, version#] pairs by the update s array. To avoid processing the same update via many paths, check version number of arriving update against the array Consider a rejected update to x at R from R, where [R, V] describes R in x s array, and V is the version number sent by R. If V V, then R saw R s updates If V < V, then R didn t see R s updates, so store it in the conflict table for later reconciliation 01-TA-Repl- 72

27 9.8 Replication and Google GFS Big chunks of data (64 MB) blocks heavily replicated controlled by master replicated as well Important status data e.g. who is primary held in master data structure These data are persistently replicated Chubby Lock service based on Paxos consensus, locks them according to reader-writer locking: n readers one writer, no reader 01-TA-Repl- 73

28 Chubby Use cases GFS: Elect a master BigTable: master election, client discovery, table service locking Well-known location to bootstrap larger systems Partition workloads Locks should be coarse: held for hours or days build your own fast locks on top 01-TA-Repl- 74

29 Chubby All client traffic One Chubby Cell replica Master replica replica replica replica Master: has all the information about chunks, node failures, locks etc. Readers / writer have to lock chunks before read / write Loss of Master = disaster! 01-TA-Repl- 75

30 Chubby Typical 5 Chubby cells (servers) in different racks Responsible for a data center Master election using Paxos Master Lease: promise not to elect a new master for some time (see below) Clients will access master or replicas found in DNS but all reads / writes forwarded to master Write requests propagated to replica by consensus protocol 01-TA-Repl- 76

31 Fault tolerant locking service Client network Rplica network Paxos File transfer / snapshot Chubby protocol RPC locking: reader /writer -model: many reads, at most one write Local file system IO Lock service used by GFS, BigTable etc. Holds all kinds of metadata Replicated for fault tolerance, not performance 01-TA-Repl- 77

32 Why leases? Goal: Make reads cheaper Read request would need a consensus of sufficient (3) replica New master cold have been elected! Value to be read may be different in different replica. Master lease: a promise not to elect a new master as long as lease is valid But writes TA-Repl- 78

33 Write requests write requests performed on master propagated to replica using Paxos In case of agreement (3 replica of 5 living) ack to client Log entries propagated for the values to be written One instance of Paxos started for each log entry Multipaxos = agreement on a sequence of values Many subtle engineering problems, see reader 01-TA-Repl- 79

34 9.8 Mobile Databases, a brief overview IBM DB2 Everyplace Oracle 9i Lite Sybase UltraLite Tamino Mobile Pointbase Micro extremedb Differences Synchronisation with base stations Application Development (Tools, platforms, ) see: Mutschler, Specht: Mobile Datenbankysteme, Springer TA-Repl- 84

35 Application Architectures Standard Client Server Drahtlose Netzwerkanbindung Festnetzverbindung Mobile Clients Anwendung DB Middleware (Synchronisations-Server) DB-Server Systems DBS separated from application IBM DB2 EveryPlace Oracle 9i Lite slides adaptes from Mutschler / Specht 01-TA-Repl- 85

36 Application Architectures Integrated Mobile DB Mobile Anwendung Drahtlose Netzwerkanbindung Festnetzverbindung Integrierte Datenbank Optionale Middleware DB-Server DBS and application integrated saves memory space, only functions needed are binded Systems Sybase UltraLite Pointbase Micro extremedb (Main mempry DB") 01-TA-Repl- 86

37 Database Engine Typically Relational Functionally differs considerably Top end: DB2Everyplace, full fledged DBS Systems configurable (100 - ~500 KB) 01-TA-Repl- 87

38 Synchronisation of Replica DB2 Everyplace: Mirror-DB Spiegeldatenbank Quelldatenbank Mobiler Client 4 Mirror Table 2 Change Data Table 1 Source Table Synchronisationsantwort Input Queue 3 Source System Mid-Tier-System 01-TA-Repl- 88

39 Synchronisation of Replica Oracle Lite Snapshot based Snapshot = materialized view Full Refresh transmit all tuples of snapshot query Fast Refresh use snapshot logs Force Refresh mixed full / fast 01-TA-Repl- 89

40 Mobile Middleware Client Middleware Server Mobile Server (Middleware) Apache Oracle HTTP Server Oracle 9i AS (WE) MS Module MS Module MS Module Oracle Lite Datenbank Mobile Server Standalone Oracle 9i Datenbank Should be non-proprietary! How to connect Client with server from different vendor? -> Standards 01-TA-Repl- 90

41 SyncML Platform no only c/s sync but also client / client 01-TA-Repl- 91

42 Summary Replication is intended for availability rather than for throughput / response time enhancement (more or less) Transactional guarantees are costly Atomicity and prevention of lost updates may be ok in many application, i.e. Isolation level Read uncommitted more update performance (e.g. asynchronous update propagation possible) Replication of tables with high frequency updates does not make much sense (in general), but backup Sophisticated (and confusing!) solutions by vendors Formidable task for the DB Administrator to decide on when and what to replicate 01-TA-Repl- 92