Using Tungsten Replicator to solve replication problems

Transcription

1 Using Tungsten Replicator to solve replication problems Neil Armitage, Cluster implementation Engineer, Continuent Giuseppe Maxia, QA Director, Continuent 1 1

2 ABOUT US Neil Armitage Continuent Tungsten Deployment and Support Engineer, Continuent, Inc 20 years development and DB experience Giuseppe Maxia, a.k.a. "The Data Charmer" QA Director, Continuent, Inc 25 years development and DB experience long timer MySQL community member. Oracle ACE Director 2 2

3 Tungsten replicator Global transaction ID Multiple masters Multiple sources Flexible topologies Parallel replication Heterogeneous replication... and more 3 3

4 What Tungsten Replicator is NOT Automated management Automatic failover Transparent connections All the above (and more) are available with a commercial solution named Continuent Tungsten (a.k.a. Tungsten Enterprise) 4 4

5 What are we talking about? Requirements Components Installation Topologies Administration Troubleshooting 5 5

6 Tungsten Replicator Concepts Replicator The replication engine Role Master, slave, direct slave service A.k.a. "pipeline" stage extract,queue,apply 6 6

7 Tungsten Replicator Components THL Transaction History Log service schema Makes the node crash proof properties file service definition tools Ruling from a centralized location 7 7

8 Tungsten Replicator in a nutshell host1 master host2 slave binlog global transaction ID THL THL trep_commit_seqno origin seqno eventid trep_commit_seqno origin seqno eventid 8 8

9 Planning Hosts Topology Stand-alone or taking over 9 9

10 master-slave fan-in slave MySQL Oracle Oracle MySQL all-masters Heterogeneous Oracle MySQL MySQL Oracle star 10

11 Installation 11 11

12 Installation System Requirements Validate!rst Deploying from a single location 12 12

13 Installation - tools tools/ tungsten-installer tools/ con!gure-service tools/update (Using the cookbook recipes, you hardly see them) 13 13

14 Tungsten in practice Installation 14 14

15 Installation Check the requirements Get the binaries Expand the tarball Run cookbook 15 15

16 REQUIREMENTS Java JRE or JDK (Sun/Oracle or Open-jdk) Ruby 1.8 (only during installation) ssh access to the same user in all nodes MySQL user with all privileges 16 16

17 Installation - Choices --master-slave --direct 17 17

18 master-slave host1 master host2 slave binlog THL THL host3 slave THL 18 18

19 direct host1 master host2 slave relay log binlog host3 THL slave relay log THL 19 19

20 Overview of Virtual Machines Copy zip!les from USB Key Expand on local disk Start all 4 Machines in VirtualBox 20 20

21 Virtual Machines 4 Nodes host1->host4 Running centos 6.3 and Percona 5.5 Root and tungsten password = password localhost port 2222 redirects to 22 on hosts ssh - p 2222 tungsten@localhost 21 21

22 VERY important de!nitions Staging directory: Where you unpack the software and run the installer. There is generally only one, in one host; Can be discarded after installation Installation directory: Where your installed software will go; There is one for every host; 22 22

23 Example host1 Staging directory: $HOME/tungsten-replicator Installation directory: /opt/replication Installation directory: /opt/replication host2 Installation directory: /opt/replication host

24 Requirements : how to step by step: how it happened 24 24

25 installing VMs Step-by-step demo 25 25

26 Overview of Tungsten cookbook 26 26

27 tungsten cookbook tungsten-replicator /cluster-home +--/cookbook +--/tools +--/tungsten-replicator 27 27

28 tungsten cookbook tungsten-replicator /cookbook +--COMMON_NODES.sh +--USER_VALUES.sh +--NODES_MASTER_SLAVE.sh +--install_master_slave +--show_cluster +--test_cluster

29 tungsten cookbook tungsten-replicator /cookbook +--COMMON_NODES.sh +--USER_VALUES.sh +--NODES_ALL_MASTERS.sh +--install_all_masters +--show_cluster +--test_cluster

30 tungsten cookbook tungsten-replicator /cookbook +--COMMON_NODES.sh +--USER_VALUES.sh +--NODES_STAR.sh +--install_star +--show_cluster +--test_cluster

31 tungsten cookbook tungsten-replicator /cookbook +--COMMON_NODES.sh +--USER_VALUES.sh +--NODES_FAN_IN.sh +--install_fan_in +--show_cluster +--test_cluster

32 tungsten cookbook $ cat COMMON_NODES.sh export NODE1=host1 export NODE2=host2 export NODE3=host3 export NODE4=host

33 tungsten cookbook $ cat USER_VALUES.sh # User defined values for the cluster to be installed. export TUNGSTEN_BASE=$HOME/installs/cookbook export DATABASE_USER=tungsten export BINLOG_DIRECTORY=/var/lib/mysql export MY_CNF=/etc/my.cnf export DATABASE_PASSWORD=secret export DATABASE_PORT=3306 export TUNGSTEN_SERVICE=cookbook export RMI_PORT=10000 export THL_PORT=2112 export START_OPTION=start 33 33

34 Getting started: VALIDATE FIRST export VERBOSE=1./cookbook/check_cookbook./cookbook/validate_cluster 34 34

35 sample master-slave installation edit cookbook/common_nodes.sh edit cookbook/user_values.sh run cookbook/install_master_slave and then: run cookbook/show_cluster run cookbook/test_cluster 35 35

36 What does the installation do 1: Validate all servers host4 host1 host2 host3 Report all errors 36 36

37 What does the installation do 1: (again) Validate all servers host4 host1 host2 host

38 What does the installation do 2: install Tungsten in all servers $HOME/ tinstall/ config/ releases/ relay/ thl/ tungsten/ backups/ host4 host1 host2 host

39 example (from manual installation) ssh r2 chmod 444 $HOME/tinstall./tools/tungsten-installer \ --master-slave --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$home/tinstall \ --thl-directory=$home/tinstall/logs \ --relay-directory=$home/tinstall/relay \ --cluster-hosts=r1,r2,r3,r4 --start ERROR >> qa.r2.continuent.com >> /home/tungsten/ tinstall is not writeable 39 39

40 example ssh r2 chmod 755 $HOME/tinstall./tools/tungsten-installer \ --master-slave --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$home/tinstall \ --thl-directory=$home/tinstall/logs \ --relay-directory=$home/tinstall/relay \ --cluster-hosts=r1,r2,r3,r4 --start # no errors 40 40

41 After installation. A tour of the cookbook utilities 41 41

42 General principles (1) Scripts without extension are designed to be launched by users e.g../cookbook/help./cookbook/install_master_slave Scripts with extension ".sh" are either for internal use only or deprecated../cookbook/install_* scripts can be used before installing. Most everything else require an installed topology 42 42

43 General principles (2) After installation there is a!le CURRENT_TOPOLOGY in the staging directory cookbook scripts can be used either from the staging directory or from the installation directory

44 Cookbook tour: help and checks./cookbook/check_cookbook./cookbook/help./cookbook/readme 44 44

45 Cookbook tour: Getting information./cookbook/show_cluster./cookbook/paths./cookbook/backups./cookbook/services./cookbook/query_node {node} {query}./cookbook/query_all_nodes {query} 45 45

46 Cookbook tour: Inspecting replication./cookbook/replicator./cookbook/trepctl./cookbook/thl./cookbook/show_conf./cookbook/edit_conf./cookbook/show_log./cookbook/vimlog./cookbook/emacslog 46 46

47 Cookbook tour: testing tools./cookbook/test_cluster./cookbook/start_load [start stop]./cookbook/test_all_topologies 47 47

48 Cookbook tour: powerful admin tools./cookbook/heartbeat./cookbook/switch./cookbook/add_node_master_slave./cookbook/add_node_star./cookbook/copy_backup./cookbook/clear_cluster # <--- CAUTION! 48 48

49 More installation 49 49

50 DRY-RUN Method to simulate installation; Does NOT perform installation; Does NOT even do validation; It only shows the commands used to install; Allows you to get the commands and do an installation manually (e.g. when you can't ssh between nodes) 50 50

51 DRY-RUN export DRYRUN=1./cookbook/install_master_slave 51 51

52 Intro to multi-master installation 52 52

53 How tungsten-installer Works for Basic Master/Slave Deployment Staging copy of files db1 db2 check prereqs copy code db3 configure 53 53

54 From Master/Slave Replication... db2 db1 Replicator Service alpha Replicator Service alpha db3 Replicator tungsteninstaller Service alpha Install master and slaves on the whole cluster 54 54

55 To Multi-Master db1 Replicator Replicator db2 Service alpha Service alpha Service bravo Service bravo tungsteninstaller tungsteninstaller con!gureservice con!gureservice Install master on db1 install master on db2 install slave service on db1 install slave service on db

56 tungsten-installer master 1 TUNGSTEN_HOME=/home/tungsten/installs/cookbook./tools/tungsten-installer --master-slave --master-host=$master1 --datasource-port= datasource-user=tungsten --datasource-password=secret --datasource-log-directory=/var/lib/mysql --service-name=alpha --home-directory=$tungsten_home --cluster-hosts=$master1 --start creating service 'alpha' Notice: --cluster-hosts has only one host 56 56

57 tungsten-installer master 2 TUNGSTEN_HOME=/home/tungsten/installs/cookbook./tools/tungsten-installer --master-slave --master-host=$master2 --datasource-port= datasource-user=tungsten --datasource-password=secret --datasource-log-directory=/var/lib/mysql --service-name=bravo --home-directory=$tungsten_home --cluster-hosts=$master2 --start creating service 'bravo' Notice: --cluster-hosts has only one host 57 57

58 Con!gure Service master 1 TUNGSTEN_HOME=/home/tungsten/installs/cookbook $TUNGSTEN_HOME/tungsten/tools/configure-service -C --quiet --host=$master1 --datasource=$master1 --local-service-name=alpha --role=slave --service-type=remote --release-directory=$tungsten_home/tungsten --skip-validation-check=thlstoragecheck --master-thl-host=$master2 --master-thl-port= svc-start bravo Notice: bravo is the master service in host

59 Con!gure Service master 2 TUNGSTEN_HOME=/home/tungsten/installs/cookbook $TUNGSTEN_HOME/tungsten/tools/configure-service -C --quiet --host=$master2 --datasource=$master2 --local-service-name=bravo --role=slave --service-type=remote --release-directory=$tungsten_home/tungsten --skip-validation-check=thlstoragecheck --master-thl-host=$master1 --master-thl-port= svc-start alpha Notice: alpha is the master service in host

60 From Master/Slave Replication... db2 db1 Replicator Service db1 Replicator Service db1 db3 Replicator Service db1./cooobook/install_master_slave 60 60

61 How Do I Install Fan-In Replication? db1 Replicator Service db1 Replicator db2 Replicator Service db1 Service db2 Service db2 db3./cooobook/install_fan_in 61 61

62 How Do I Install Multi-Master? db1 Replicator Service db1 Service db2 db2 Service db1 Service db2 Replicator./cooobook/install_all_masters 62 62

63 How Do I Extend Multi-Master? db1 Replicator Service db1 Service db2 Service db3 Replicator Service db1 db3 db2 Service db2 Service db3 Service db1 Service db2 Service db3 Replicator 63 63

64 How Do I Extend Multi-Master? db1 Replicator Service db1 Service db2 Service db3 Service db4 Replicator Service db1 Service db2 Service db3 Service db4 db3 db2 Service db1 Service db1 db4 Service db2 Service db2 Service db3 Service db3 Service db4 Service db4 Replicator Replicator 64 64

65 How Do I Install a Star Topology? db1 Replicator Service db1 Service db3 Replicator Service db1 db3 HUB db2 Service db2 Service db3 Service db2 Service db3 Replicator./cooobook/install_star 65 65

66 How Do I Extend a Star Topology? db1 Replicator Service db1 Service db3 Service db1 Service db2 db3 HUB db2 Replicator Service db2 Service db3 Service db4 Service db3 db4 Service db3 Service db4 Replicator 66 66

67 How Do I Extend a Star Topology? db1 Replicator Service db1 Service db3 Service db1 Service db2 db3 HUB db2 Replicator Service db2 Service db3 Service db4 Service db5 Service db3 db5 Replicator Service db5 Service db3 db4 Service db3 Service db4 Replicator 67 67

68 BI-DIR: the painless way edit cookbook/common_nodes.sh edit cookbook/user_values.sh remove two nodes edit the variables in cookbook/ NODES_ALL_MASTERS.sh cookbook/install_all_masters 68 68

69 Multiple masters fan-in Steps: install a master service in each node install a slave service for each master in the fanin node or : cookbook/install_fan_in 69 69

70 Multiple masters star topology Steps: install a master service in each server in the hub, install a slave service for each spoke in each spoke, install a slave service for the hub, using bypass option cookbook/install_star 70 70

71 Taking Over from Standard Replication cookbook/install_standard_replicaton cookbook/takeover 71 71

72 Replication Management 72 72

73 Common Commands replicator trepctl thl the Tungsten service schema 73 73

74 replicator It s the service provider You launch it once when you start You may restart it when you change con!g 74 74

75 trepctl Tungsten Replicator ConTroLler It s the driving seat for your replication You can start, update, and stop services You can get speci!c info 75 75

76 trepctl Tungsten Replicator Controller put services online or o"ine check status skip events inspect internals change roles heartbeat backup/restore... and a lot more 76 76

77 thl Transaction History List Gives you access to the Tungsten transaction history logs 77 77

78 thl Transaction History Log info index list (total or a speci!c event, or by range) purge 78 78

79 Tungsten service schema one for each service named "tungsten_service_name" e.g. tungsten_alpha, tungsten_dragon Most important table: trep_commit_seqno 79 79

80 Looking at the tungsten service db select * from tungsten_dragon.trep_commit_seqno\g ******************* 1. row ******************* task_id: 0 seqno: 102 fragno: 0 last_frag: 1 source_id: qa.r1.continuent.com epoch_number: 0 eventid: mysql-bin : ;0 applied_latency: 0 update_timestamp: :56:12 shard_id: tungsten_dragon extract_timestamp: :56:

81 Where are the tools in the tungsten directory: $TUNGSTEN_BASE/tungsten/tungsten-replicator/bin replicator trepctl thl # the daemon # replicator controller # transaction history log tool 81 81

82 Starting and stopping the replicator cd $TUNGSTEN_BASE/tungsten/tungsten-replicator/bin./replicator status Tungsten Replicator Service is running (PID:32400)../replicator stop Stopping Tungsten Replicator Service... Stopped Tungsten Replicator Service../replicator start Starting Tungsten Replicator Service or./cookbook/replicator

83 checking replicator vitals trepctl services Processing services command... NAME VALUE appliedlastseqno: -1 # bad sign? appliedlatency : -1.0 role : slave servicename : dragon servicetype : local started : true state : ONLINE Finished services command

84 sending a heartbeat trepctl -host $MASTER_HOST heartbeat trepctl services Processing services command... NAME VALUE appliedlastseqno: 102 appliedlatency : role : slave servicename : dragon servicetype : local started : true state : ONLINE Finished services command

85 replicator status (1) trepctl status Processing status command... NAME VALUE appliedlasteventid : mysql-bin : ;0 appliedlastseqno : 102 appliedlatency : clustername : default currenteventid : NONE currenttimemillis : dataserverhost : qa.r4.continuent.com extensions : latestepochnumber : 0 masterconnecturi : thl://qa.r1.continuent.com:2112/ masterlistenuri : thl://qa.r4.continuent.com:2112/ maximumstoredseqno : 102 minimumstoredseqno : 0 [...] 85 85

86 replicator status (2) [...] offlinerequests : NONE pendingerror : NONE pendingerrorcode : NONE pendingerroreventid : NONE pendingerrorseqno : -1 pendingexceptionmessage: NONE resourceprecedence : 99 rmiport : role : slave seqnotype : java.lang.long servicename : dragon servicetype : local simpleservicename : dragon sitename : default sourceid : qa.r4.continuent.com state : ONLINE timeinstateseconds : uptimeseconds : Finished status command

87 A failover scenario 1: MySQL native replication 87 87

88 1. one Master, two slaves Loading the employees test database 88 88

89 2. Master goes away * Stop replication * Slaves are updated at di"erent levels # 2 select count(*) from titles 333,145 # 3 select count(*) from titles 443,

90 3. Look into Slave #2 binary logs!nd the last transaction 90 90

91 4. Look into Slave #3 binary logs 1.!nd the transaction that was last in slave #2 2. Recognize that last transaction in the log of slave #3 (This can actually take you a LOOOONG TIME) 3. Get the position immediately after this transaction 4. (e.g in!le mysql-bin ) 91 91

92 5. promote Slave #3 to master * in slave #2 CHANGE MASTER TO master_host= slave_3_ip, master_user= slavename, master_password= slavepassword, master_log_file= mysql-bin , master_log_pos=134000; 92 92

93 A failover scenario 1I: Tungsten Replicator 93 93

94 1. one master, two slaves loading the employees test database 94 94

95 2. Master goes away * Stop replication * Slaves are updated at di"erent levels # 2 select count(*) from titles 333,145 # 3 select count(*) from titles 443,

96 3. no need to!nd the last transaction # simply change roles trepctl -host slave3 setrole -role master trepctl -host slave2 setrole \ -role slave -uri thl://slave3 trepctl -host slave3 online State: ONLINE trepctl -host slave2 online State: GOING-ONLINE:SYNCHRONIZING 96 96

97 4. Check that the slave has synchronized # new master select seqno from tungsten.trep_commit_seqno; 78 # new slave select seqno from tungsten.trep_commit_seqno;

98 4. Tell the replicator to hurry up # new master trepctl -node slave3 flush Master log is synchronized with database at log sequence number: 78 # new slave trepctl host slave2 wait -applied 78 ONLINE select seqno from tungsten.trep_commit_seqno;

99 4.... and we re done # new master select count(*) from employees.titles count(*) # new slave: count(*)

100 planned role switch cookbook/install_master_slave cookbook/switch

101 Switching roles in master/slave replication (1) online db2 db1 Replicator Service db1 Replicator Service db1 db3 online Replicator Service db1 online

102 Switching roles in master/slave replication (2) online db2 db1 Replicator Service db1 Replicator Service db1 db3 o"ine Replicator Service db1 online

103 Switching roles in master/slave replication (3) online db2 db1 Replicator Service db1 Replicator Service db1 db3 o"ine Replicator Service db1 Wait for transactions to be applied online

104 Switching roles in master/slave replication (4) o"ine db2 db1 Replicator Service db1 Replicator Service db1 db3 o"ine Replicator Service db1 Slaves go offline o"ine

105 Switching roles in master/slave replication (5) o"ine db2 db1 Replicator Service db1 Replicator Service db1 db3 o"ine Replicator Service db1 Slave is promoted. Notice: 2 masters, but o"ine o"ine

106 Switching roles in master/slave replication (6) o"ine db2 db1 Replicator Service db1 Replicator Service db1 db3 o"ine Replicator Service db1 old master becomes slave o"ine

107 Switching roles in master/slave replication (7) o"ine db2 db1 Replicator Service db1 Replicator Service db1 db3 o"ine Replicator Service db1 slaves are directed to new master o"ine

108 Switching roles in master/slave replication (8) online db2 db1 Replicator Service db1 Replicator Service db1 db3 online Replicator Service db1 all nodes go online, using new master online

109 Tungsten GTID vs MySQL 5.6 GTID What is GTID How it works in Tungsten How it works (or not) in MySQL

110 without global transaction ID commit commit commit commit A master binlog position binlog binlog slave B C slave position 110 position 110

111 with global transaction ID commit commit commit commit A master id#200 slave B C slave id# id#

112 Tungsten and global transaction ID: activation (none) active by default

113 Tungsten and global transaction ID: status trepctl status Processing status command... NAME VALUE appliedlasteventid : mysql-bin : ;0 appliedlastseqno : 6 appliedlatency : clustername : default currenteventid : NONE currenttimemillis : dataserverhost : qa.r4.continuent.com

114 Tungsten and global transaction ID: seeing transactions thl list -seqno 6 SEQ# = 6 / FRAG# = 0 (last frag) - TIME = :44: EPOCH# = 0 - EVENTID = mysql-bin : ;0 - SOURCEID = qa.r1.continuent.com - SQL(0) = insert into test.v1 values (1, 'inserted by node #1') /* SERVICE = [cookbook] */

115 Tungsten and global transaction ID: changing master connection trepctl offline trepctl online -seqno

116 Tungsten and Global transaction ID: crash-safe slave tables mysql -e 'select * from tungsten_cookbook.trep_commit_seqno\g' *************************** 1. row *************************** task_id: 0 seqno: 6 fragno: 0 last_frag: 1 source_id: qa.r1.continuent.com epoch_number: 0 eventid: mysql-bin : ;0 applied_latency: 0 update_timestamp: :44:35 shard_id: test extract_timestamp: :44:

117 Tungsten and Global transaction ID: crash-safe tables and parallel replication mysql -e 'select seqno, source_id, shard_id,update_timestamp from tungsten_cookbook.trep_commit_seqno' seqno source_id shard_id update_timestamp qa.r1.continuent.com db :54:14 8 qa.r1.continuent.com db :54:14 9 qa.r1.continuent.com db :54:14 10 qa.r1.continuent.com db :54:14 11 qa.r1.continuent.com db :54:14 12 qa.r1.continuent.com db :54:14 13 qa.r1.continuent.com db :54:14 14 qa.r1.continuent.com db :54:14 15 qa.r1.continuent.com db :54:14 16 qa.r1.continuent.com db :54:

118 MySQL 5.6 and global transaction ID activation mysqld --log-slave-updates \ --gtid-mode=on \ --enforce-gtid-consistency WARNING: before MySQL , it was --disable-gtid-unsafe-statements

119 MySQL 5.6 and global transaction ID seeing transactions # :15:49 server id 1 end_log_pos 344 CRC32 0x45b25c8f GTID [commit=yes] '7A77A490-3D3A-11E2-8CC9-7DCF B: 2'/*!*/; # at 344 # :15:49 server id 1 end_log_pos 423 CRC32 0x873c8fac Query thread_id=3 exec_time=0 error_code=0 SET TIMESTAMP= /*!*/; BEGIN /*!*/; # at 423 # :15:49 server id 1 end_log_pos 522 CRC32 0xb4bf4372 Query thread_id=3 exec_time=0 error_code=0 SET TIMESTAMP= /*!*/; insert into t1 values (1)

120 MySQL 5.6 and global transaction ID status show slave status\g *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: Master_User: rsandbox Master_Port: Connect_Retry: 60 Master_Log_File: mysql-bin Read_Master_Log_Pos: 1837 Relay_Log_File: mysql_sandbox13234-relay-bin Relay_Log_Pos: 2047 Relay_Master_Log_File: mysql-bin Retrieved_Gtid_Set: 46E B28-11E2-BF47-6C626DA07446:1-7 Executed_Gtid_Set: 46E B28-11E2-BF47-6C626DA07446:

121 MySQL 5.6 and global transaction ID changing master connection CHANGE MASTER TO master_log_file='mysql-bin ', master_log_pos='1234' # No global transaction ID is used

122 MySQL 5.6 and global transaction ID crash-safe slave table select * from slave_relay_log_info\g ********************* 1. row ******************** Number_of_lines: 7 Relay_log_name:./mysql_sandbox13234-relay-bin Relay_log_pos: 2047 Master_log_name: mysql-bin Master_log_pos: 1837 Sql_delay: 0 Number_of_workers: 5 Id: 1 # NO Global transaction ID is used!

123 MySQL 5.6 and global transaction ID crash-safe slave table + parallel select * from mysql.slave_worker_info\g Id: 12 Relay_log_name:./mysql_sandbox13234-relay-bin Relay_log_pos: 4299 Master_log_name: mysql-bin Master_log_pos: 7155 Checkpoint_relay_log_name:./mysql_sandbox13234-relay-bin Checkpoint_relay_log_pos: 1786 Checkpoint_master_log_name: mysql-bin Checkpoint_master_log_pos: 4642 Checkpoint_seqno: 9 Checkpoint_group_size: 64 Checkpoint_group_bitmap:? # NO Global transaction ID is used!

124 Filters

125 Tungsten Replication Service Pipeline Stage Extract Filter Apply Stage Extract Filter Apply Stage Extract Filter Apply Master DBMS Transaction History Log In-Memory Queue Slave DBMS

126 Restrict replication to some schemas and tables./tools/tungsten-installer \ --master-slave -a \... --svc-extractor-filters=replicate \ "--property=replicator.filter.replicate.do=test,*.foo" \... --start-and-report # test="test.*" -> same drawback as binlog-do-db in MySQL # *.foo = table 'foo' in any database # employees.dept_codes,employees.salaries => safest way

127 Exclude some schemas and tables from replication./tools/tungsten-installer \ --master-slave -a \... --svc-extractor-filters=replicate \ "--property=replicator.filter.replicate.ignore=test,*.foo" \... --start-and-report # test="test.*" -> same drawback as binlog-ignore-db in MySQL # *.foo = table 'foo' in any database # employees.dept_codes,employees.salaries => safest way # DO NOT MIX.do and.ignore! # (you can do it, but it may not do what you mean)

128 Change name of replicated schema -a --svc-applier-filters=dbtransform \ --property=replicator.filter.dbtransform.from_regex1=stores \ --property=replicator.filter.dbtransform.to_regex1=playground # from_regex1=stores -> name of the schema in the master # to_regex1=playground -> name of the schema in the slave # WARNING: requires "USE schema_name" to work properly

129 Multi-master: Con#ict prevention

130 CONFLICTS Continuent

131 What's a con#ict Data modi!ed by several sources (masters) Creates one or more : data loss (unwanted delete) data inconsistency (unwanted update) duplicated data (unwanted insert) replication break

132 Data duplication 4 Matt 140 alpha id name amount 1 Joe Frank Sue 100 bravo charlie 4 Matt 130 BREAKS REPLICATION

133 auto_increment o$sets are not a remedy A popular recipe auto_increment_increment + auto_increment_offset They don't prevent con#icts They hide duplicates

134 Hidden data duplication 11 Matt 140 INSERT alpha o$set 1 id name amount 1 Joe Frank Sue 100 bravo o$set 2 INSERT charlie o$set 3 13 Matt

135 Data inconsistency 3 Sue 108 UPDATE alpha id name amount 1 Joe Frank Sue 100 bravo charlie UPDATE 3 Sue

136 Data loss 3 Sue 108 UPDATE alpha id name amount 1 Joe Frank Sue 100 bravo charlie DELETE record #3 MAY BREAK REPLICATION

137 con#ict handling strategies resolving after the fact planned for future use Needs information that is missing in async replication avoiding requires synchronous replication with 2pc preventing setting and enforcing a split sources policy Transforming and resolving all records are converted to INSERTs used by Tungsten planned for future use con"icts are resolved within a given time window Continuent

138 Multi-master: Con!ict prevention

139 Tungsten con#ict prevention in a nutshell 1. de!ne the rules (which master can update which database) 2. tell Tungsten the rules 3. de!ne the policy (error, drop, warn, or accept) 4. Let Tungsten enforce your rules

140 Tungsten Con#ict prevention facts Sharded by database De!ned dynamically Applied on the slave services methods: error: make replication fail drop: drop silently warn: drop with warning

141 Tungsten con#ict prevention applicability unknown shards The schema being updated is not planned actions: accept, drop, warn, error unwanted shards the schema is updated from the wrong master actions: accept, drop, warn, error whitelisted shards can be updated by any master

142 Con#ict prevention directives --svc-extractor-filters=shardfilter replicator.filter.shardfilter.unknownshardpolicy=error replicator.filter.shardfilter.unwantedshardpolicy=error replicator.filter.shardfilter.enforcehomes=false replicator.filter.shardfilter.allowwhitelisted=false

143 con#ict prevention in a star topology alpha updates employees A C Host1 master: alpha database: employees A B C Host3 master: charlie (hub) database: vehicles Host2 master: bravo database: buildings B C

144 con#ict prevention in a star topology alpha updates vehicles A C Host1 master: alpha database: employees A B C Host3 master: charlie (hub) database: vehicles Host2 master: bravo database: buildings B C

145 con#ict prevention in a all-masters topology alpha updates employees B A C Host1 master: alpha database: employees Host2 master: bravo database: buildings B A C A B C Host3 master: charlie database: vehicles

146 con#ict prevention in a all-masters topology charlie updates vehicles B A C Host1 master: alpha database: employees Host2 master: bravo database: buildings B A C A B C Host3 master: charlie database: vehicles

147 con#ict prevention in a all-masters topology bravo updates employees B A C Host1 master: alpha database: employees Host2 master: bravo database: buildings B A C A B C Host3 master: charlie database: vehicles

148 con#ict prevention in a all-masters topology charlie updates employees B A C Host1 master: alpha database: employees Host2 master: bravo database: buildings B A C A B C Host3 master: charlie database: vehicles

149 setting con#ict prevention rules trepctl -host host1 -service charlie \ shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host

150 setting con#ict prevention rules trepctl -host host2 -service charlie \ shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host

151 setting con#ict prevention rules trepctl -host host3 -service alpha \ shard -insert < shards.map trepctl -host host3 -service bravo \ shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # alpha and bravo are slave services in host

152 Con#ict prevention demo reminder Server #1 can update "employees" Server #2 can update "buildings" Server #3 can update "vehicles"

153 Sample correct operation (1) mysql #1> create table employees.names(... ) # all servers receive the table # all servers keep working well

154 Sample correct operation (2) mysql #2> create table buildings.homes(... ) # all servers receive the table # all servers keep working well

155 Sample incorrect operation (1) mysql #2> create table employees.nicknames(... ) # Only server #2 receives the table # slave service in hub gets an error # slave service in #1 does not receive anything

156 sample incorrect operation (2) #3 $ trepct services simple_services alpha [slave] seqno: 7 - latency: ONLINE bravo seqno: [slave] -1 - latency: OFFLINE:ERROR charlie [master] seqno: 66 - latency: ONLINE

157 sample incorrect operation (3) #3 $ trepct -service bravo status NAME VALUE appliedlasteventid : NONE appliedlastseqno : -1 appliedlatency : -1.0 (...) offlinerequests : NONE pendingerror : Stage task failed: q-to-dbms pendingerrorcode : NONE pendingerroreventid : mysql-bin : ;0 pendingerrorseqno : 7 pendingexceptionmessage: Rejected event from wrong shard: seqno=7 shard ID=employees shard master=alpha service=bravo (...)

158 Fixing the issue mysql #1> drop table if exists employees.nicknames; mysql #1> create table if exists employees.nicknames (... ) ; #3 $ trepct -service bravo online -skip-seqno 7 # all servers receive the new table

159 Sample whitelisted operation mysql #2> create table test.hope4best(... ) mysql #1> insert into test.hope4best values (... ) # REMEMBER: 'test' was explicitly whitelisted # All servers get the new table and records # But there is no protection against conflicts

160 administration

161 Viewing THL Events thl info log directory = /home/tungsten/installs/master_slave/thl/dragon/ min seq# = 0 max seq# = 101 events =

162 viewing THL events thl index LogIndexEntry thl.data (0:102)

163 viewing THL events thl index [...] LogIndexEntry thl.data (0:18) LogIndexEntry thl.data (19:33) LogIndexEntry thl.data (34:35) LogIndexEntry thl.data (36:3641) LogIndexEntry thl.data (3642:3712) LogIndexEntry thl.data (3713:3838) LogIndexEntry thl.data (3839:3949) LogIndexEntry thl.data (3950:4011) LogIndexEntry thl.data (4012:4039) LogIndexEntry thl.data (4040:4057) LogIndexEntry thl.data (4058:4067) LogIndexEntry thl.data (4068:4073) LogIndexEntry thl.data (4074:4085) LogIndexEntry thl.data (4086:4095) LogIndexEntry thl.data (4096:4101) LogIndexEntry thl.data (4102:4111)

164 viewing THL events thl list -seqno 102 [...] SEQ# = 102 / FRAG# = 0 (last frag) - TIME = :56: EPOCH# = 0 - EVENTID = mysql-bin : ;0 - SOURCEID = qa.r1.continuent.com - METADATA = [mysql_server_id=10;is_metadata=true;service=dragon;shard=tung sten_dragon;heartbeat=none] - TYPE = com.continuent.tungsten.replicator.event.repldbmsevent - OPTIONS = [##charset = ISO8859_1, autocommit = 1, sql_auto_is_null = 1, foreign_key_checks = 1, unique_checks = 1, sql_mode = 'IGNORE_SPACE', character_set_client = 8, collation_connection = 8, collation_server = 8] - SCHEMA = tungsten_dragon - SQL(0) = UPDATE tungsten_dragon.heartbeat SET source_tstamp= " :56:09", salt= 2, name= "NONE" WHERE id= 1 /* SERVICE = [dragon] */

165 Skipping a THL Event trepctl online -skip-seqno 1092 trepctl online -skip-seqno 1092,1093,1094 # see example

166 Adding a Member Let's see the cookbook, and use it

167 parallel replication

168 Replicator Pipeline Architecture Tungsten Replicator Process Pipeline Stage Stage Stage Extract Assign Shard ID Apply Extract Apply Parallel Queue Extract Extract Extract Apply Apply Apply channels Transaction History Log MySQL Binlog THL shard.list file Slave DBMS 168

169 Parallel replication facts Sharded by database Good choice for slave lag problems Bad choice for single database projects

170 Parallel Replication test STOPPED binary logs MySQL slave Concurrent sysbench on 30 databases Tungsten slave OFFLINE direct: alpha (slave) running for 1 hour TOTAL DATA: 130 GB RAM per server: 20GB replicator alpha Slaves will have 1 hour lag 170

171 measuring results START binary logs MySQL slave Tungsten slave ONLINE direct: alpha (slave) Recording catch-up time replicator alpha 171

172 MySQL native replication slave catch up in 04:29:30 172

173 Tungsten parallel replication slave catch up in 00:55:40 173

174 Parallel replication made simpler FROM HERE

175 Parallel replication made simpler TO HERE 175

176 Parallel replication made simpler 176

177 parallel replication direct slave facts No need to install Tungsten on the master Tungsten runs only on the slave Replication can revert to native slave with two commands (trepctl offline; start slave) Native replication can continue on other slaves Failover (either native or Tungsten) becomes a manual task

178 installing parallel replication MORE_OPTIONS='--channels=10'./cookbook/install_master_slave

179 Checking parallel replication trepctl status trepctl status -name tasks trepctl status -name shards trepctl status -name stores

180 Parallel replication demo

181 Troubleshooting

182 Identify the Failed Component Steps 1. trepctl services 2. trepctl -service SVC_NAME status 3. look at the logs 4. Take action

183 reading the logs ls $TUNGSTEN_BASE/tungsten/tungsten-replicator/logs/ trepsvc.log user.log... or./cookbook/show_log # let's see it in practice

184 Parting thoughts

185 Open source Tungsten Replicator now includes Oracle-to-MySQL and Oracle-to-Oracle extractors and appliers!

186 560 S. Winchester Blvd., Suite 500 San Jose, CA Tel +1 (866) Fax +1 (408) Our Blogs: Continuent Website: Tungsten Replicator 2.0: Continuent