MySQL High Availability. MMM & MHA in DP 卢钧轶

Transcription

1 MySQL High Availability MMM & MHA in DP 卢钧轶

2 Agenda Why we need a HA solution MMM in DP MHA in DP

3 When We Need to Switch Server Server down Server maintenance Logical : DDL on big table Physical: memory replacement

4 Why We Need A Failover Solution Failover without tool: Notify DBA for MySQL Failure Kill old Find the latest Slave as Candidate Execute change to on each Slave Direct APP to new IP & restart

5 Difficulties For MySQL Failover file2:3000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:3000

6 MySQL HA options Stand-by MMM MHA + VIP DRDB + (heartbeat,keepalive) Proxy MHA + HAProxy Cluster MySQL Cluster Galera

7 Why We Chose MMM in 2010 MMM MHA Entire HA solution Yes only failover VIP Built-in support Needs user defined script CLI interface All included in mmm_control Several Perl script Deployment CPAN++Agent CPAN+Manager+Agent + SSH authorization

8 MMM in DP Used in Production since M-M / M-M-S Clusters Modifications in Source Code

9 MMM Is VIP based HA solution Message between & Agent Auto Failover for both M/S Is Not: SQL router Load Balancer Active ( latest code release in )

10 Simple Usage Web Server vip1 Standby

11 Basic Usage vip1 Slave vip2 Standby Slave vip3 Write Read Web Server

12 Role Based Machine Group Job Group vip5 vip1 vip2 Frontend Group vip1 & vip2 Slave vip3 / vip5 Slave vip4 Backend Group vip3 & vip4

13 MMM architecture vip1 vip2 checker replication DB1(online):role1 DB2(online):role2 DB3(online):role3 DB4(online):role4 role status Slave vip3 Slave vip4

14 MMM Logic

15 MMM Slave Switch process_command mmm_control set_offline DB4 vip1 vip2 OFFLINE DB1(online):role1 DB2(online):role2 DB3(online):role3 DB4(offline) Slave vip3 Slave

16 MMM Slave Switch distributed_roles vip1 vip2 Slave vip3 Slave DB1(online):role1 DB2(online):role2 DB3(online):role3,role4 DB4(offline)

17 MMM Slave Switch send_status_to_agents vip1 vip2 Slave vip3 Slave DB1(online):role1 DB2(online):role2 DB3(online):role3,role4 DB4(offline) role4 added

18 How MMM Do Mark-Down vip1 vip2 Slave vip3 Slave vip4

19 How MMM Do Mark-Down set global read_only=1 vip1 vip2 Slave vip3 Slave vip4

20 How MMM Do Mark-Down remove VIP vip2 Slave vip3 Slave vip4

21 How MMM Do Mark-Down vip2 Slave vip3 Slave vip4 select MASTER_POS_WAIT()

22 How MMM Do Mark-Down vip2 Slave vip3 Slave vip4 show master status

23 How MMM Do Mark-Down vip2 Data Consistency??? change master to Slave vip3 Slave vip4

24 How MMM Do Mark-Down vip1&vip2 Slave vip3 Slave vip4 vip1 online

25 Cons of MMM Global read_only is hard to get on busy serv ers Slow slave bring more downtime Change master on slaves is sequential Communication between agent & monitor is based on message MMM is fundamentally broken -- Baron Schwartz:

26 Con #1 Slow slave brings more downtime VIP1 cannot be accessed during this period vip2 Slave vip3 Job Slave vip4 select MASTER_POS_WAIT()

27 Eliminate Replication Lag Logical Methods: Chop oak-chunk-update Avoid peak hour jobs Physical Methods: Prefetch Disk upgrade

28 Shannon Systems Direct-IO PCIe SSD 兼具可靠性和安全性 Smart FTL Flash Translation Layer 6.4TB 全球单卡最大容量 - 热 冷数据动态跟踪 - GC/WL 综合动态平衡 写放大因子最小化 - 双重数据保护机制 - 最大化NAND Flash 寿命 完善容错数据保护机制 - 高达 40bit/1KB ECC或更高 - 读写 擦除出错处理及数据保护 - 页面 坏块出错处理及数据保护 端到端数据保护 - 企业级端到端数据链路保护 (Data path protection) - 多重数据完整性及正确性校 验 掉电数据保护 67/9µs 超低随机读/写延迟 - 完善的突发掉电数据保护机制 - 防止系统不正常关机的数据完整性和安全 性 25W 更小峰值功耗 节省运维成本 过热保护机制 - 防止系统过热对系统造成不可恢复 损伤

29 Shannon Direct-IOTM PCIe SSD G2T 测试数据 Capacity 800GB 1.2TB 1.6TB 3.2TB/6.4TB Flash 闪存类型 MLC MLC MLC MLC Read Bandwidth 读宽带 1.4GB/s 2.0GB/s 2.6GB/s 2.6GB/s Write Bandwidth 写宽带 1.2GB/s 1.8GB/s 1.8GB/s 1.9GB/s Random Read Latency(4KB) 随机读延迟 67us 67us 67us 67us Random Write Latency (4KB) 随机写延迟 9us 9us 9us 9us Random Read IOPS(4KB) 随机读IOPS 300, , , ,000 Random Write IOPS(4KB) 随机写IOPS 310, , , ,000 Endurance 寿命 10PB 15PB 20PB 40PB

30 What Have We Done However replication lag can still be exist Improvement: Only wait for new master to catch up

31 Con #2 Global read_only is hard to get on busy servers 1. Invalidates the Query Cache. 2. Waits for all in-flight updates to complete and at the same time it blocks all incoming updates. 3. Closes all open tables and expels them from the table cache. Wait for all SELECT queries to complete. Incoming SELECT queries will get blocked. 4. Blocks COMMITs.

32 What Have We Done Manually check long sql before switch Improvement: remove master vip before read_only to kill sql before read_only

33 What Have We Done

34 How DP-MMM Do Failover Clients remove VIP vip2 Slave vip3 Slave vip4

35 How DP-MMM Do Failover Clients Kill Active Trx vip2 Slave vip3 Slave vip4

36 How DP-MMM Do Failover Clients set global read_only=1 vip2 Slave vip3 Slave vip4

37 How DP-MMM Do Failover Clients vip2 Slave vip3 Slave vip4 MASTER_POS_WAIT() & SHOW MASTER STATUS

38 How DP-MMM Do Failover Clients vip2&vip1 Slave vip3 Slave vip4 VIP1 added

39 How DP-MMM Do Failover Clients vip2 Slave vip3 Slave vip4 CHANGE MASTER TO

40 Con #3 Why MMM is fundamentally broken: Communication based on message between monitor & agent Agent manage its own state Data Loss

41 Example of Broken set_offline db1 success!

42 Cause of Broken Instance 1 State: DB1: ONLINE (role: write) DB2: ONLINE (role: reader) DB1 You are ONLINE Instance 2 State: DB1: ONLINE (role: write) DB2: ONLINE (role: reader) You are ONLINE DB2 VIP2

43 Cause of Broken mmm_control set_offline db2 Instance 1 State: DB1: ONLINE (role: write,reader) DB2: OFFLINE DB1 You are OFFLINE VIP2 Instance 2 State: DB1: ONLINE (role: write) DB2: ONLINE (role: reader) You are ONLINE DB2

44 What Have We Done Fix multiple monitor instance bug. But we cannot fix its architecture mmm_control set_online db1 mean NOTHING but confirmation of command

45 Cons of MMM Global read_only is hard to get on busy serv ers Slow slave bring more downtime Change master on slaves is sequential Communication between agent & monitor is based on message

46 Why We Chose MHA in 2014 MMM MHA Data Loss Every Time after failover Almost none Communicate with DAL N/A User defined script HA Solution without N/A VIP Supported

47 How MHA Do Failover Slave1 Slave2 Slave3

48 How MHA Do Failover Slave1 Slave2 Slave3 Connect to all nodes and check: master is really dead all slaves is the slave of dead master last failover is ealier than $last_failover_minute

49 How MHA Do Failover Slave1 Slave2 Slave3

50 Call master_ip_failover_script master_ip_failover_script Slave1 file1:1000 Force Shutdown Slave2 file2:3000 Slave3 file1:3000

51 Find Latest Slave Relay Log Position Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:3000

52 Save Binlog if Alive file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:3000

53 Find Relay Log Base Slave file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:3000

54 Find Candidate file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:3000

55 Candidate Configuration candidate_master: this server is prioritized in election no_master: this server will not be new master in election latest_priority: use order by block_name instead relay_log position in election

56 Candidate Election Rule IF nothing special configured elect relay log base slave IF $latest_priority=1 elect latest slave with candidate_master=1 ELSE elect cacndidate_master by name IF $latest_priority=1 elect latest slave ELSE elect remaining slave

57 Recover Candidate file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:2000 file1: 2000-end file2:

58 Recover Candidate file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:2000 file1: 2000-end file2: file2:

59 Recover Candidate file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:2000 file1: 2000-end file2: file2:

60 Recover Candidate file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file1:end file1: 2000-end file2: file2:

61 Recover Candidate file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file2:3000 file1: 2000-end file2: file2:

62 Recover Candidate file2: file2:4000 Slave1 file1:1000 Slave2 file2:3000 Slave3 file2:4000 file1: 2000-end file2: file2:

63 Recover Candidate file2: file2:4000 show master status Slave1 file1:1000 Slave2 file2:3000 Slave3 file2:4000 file1: 3000-end file2: file2:

64 Parallel Recover Slaves file2: Slave3 file2:4000 Slave1 file1:1000 Slave2 file2:3000 file1: 1000-end file2: file2: file2:

65 Cons of MHA No Slave Management CLI Not So Friendly compared to MMM SSH login without password within all nodes

66 What Have We Done MHA::SlaveFailover MHA::NodeOnline masterha_wrapper User Defined Script interact with DAL

67 Interact With DAL Slave1 Slave2 Mark Down DB3 Web Slave3

68 Interact With DAL Slave1 Slave2 Mark Down DB3 Web Slave3

69 Interact With DAL Slave1 Slave2 Kill Connection Web Slave3

70 Interact With DAL Slave1 Slave2 Stop Slave Web Slave3

71 Interact With DAL read_only=on Slave1 read_only=on read_only=on Slave2 Slave3 Mark Down Web read_only=off

72 Interact With DAL read_only=on Slave1 read_only=on read_only=on Slave2 Slave3 Web Searching For Writable Server

73 Interact With DAL read_only=on Slave1 read_only=on read_only=on Slave2 Slave3 set read_only=on Web Searching For Writable Server

74 Interact With DAL read_only=on read_only=on Slave2 Slave3 read_only=on Slave1 Web Searching For Writable Server

75 Interact With DAL read_only=on read_only=on Slave2 Slave3 read_only=off Slave1 set read_only=off Web Searching For Writable Server

76 Future GTID + mysqlfailover Percona lock_binlog_for_backup

77 GTID Possible to identify a transaction uniquely across the replication servers. Make the automation of failover process much easier. No calculations! At application level it is easier to do WRITE/READ split. Development of new automation tools isn t a pain now.

78 GTID

79 Flush Tables With Read Lock 1. Invalidates the Query Cache. 2. Waits for all in-flight updates to complete and at the same time it blocks all incoming updates. 3. Closes all open tables and expels them from the table cache. Wait for all SELECT queries to complete. Incoming SELECT queries will get blocked. 4. Blocks COMMITs. Reference:

80 Lock Binlog For Backup Unlike FTWRL, the LOCK TABLES FOR BACKUP statement: * does not invalidate the Query Cache; * never waits for SELECT queries to complete regardless of the storage engines involved; * never blocks SELECTs, or updates to InnoDB, Blackhole and Federated tables.

81 Conclusion Tips for takeaway

82 When to Use MMM Simple -- MySQL deployment environment Can afford several seconds data loss VIP based HA solution already exist

83 When to Use MMM - DP version Already using MMM Slow Slaves included in cluster Massive load on master

84 When to Use MHA Cannot afford data loss No requirement for Slave High Availability Proxy already exists in current deployment Interact with client side needed when switch/failover

85 Q&A Thanks! dianping.com