PRM and DRBD tutorial Yves Trudeau October 2012
Agenda Introduction to Pacemaker PRM principle PRM Hands-on HA over shared storage What is DRBD? Impacts of DRBD on MySQL DRBD Hands-on
About me
Pacemaker history Heartbeat V1 Heartbeat V2 OpenAIS Pacemaker Heartbeat Corosync
Pacemaker stack Resources Resource-agents Pacemaker Communication
Pacemaker communication Heartbeat or Corosync UDP based Broadcast* / multicast / unicast
Pacemaker features CIB Group Stonith Location and colocation Policy engine Clone set and ms Election of DC Scores
Resource agents Scripts implementing an API Can read and write cluster wide attributes Wide selection Fairly easy to create new ones
PRM
PRM principle of operation (1)
PRM principle of operation (2)
PRM principle of operation (3)
PRM principle of operation (4)
PRM principle of operation (5)
PRM principle of operation (6)
PRM principle of operation (7)
PRM principle of operation (8)
PRM principle of operation (9)
PRM principle of operation (10)
PRM highlights Handles slaves configuration Master promotion based on master_score Reader_VIP controlled by the readable attribute writer_vip follows the master
Enough talking! Hands-on
PRM hands-on: VMs setup 1. Install Virtualbox 2. Vms: http://goo.gl/p0uj9 3. File -> import appliance, repeat for the 3.ova files 4. user = root, password = pacemaker
PRM hands-on: The VMs ssh access port 22222, 22223, 22224 eth0 is Virtualbox NAT, not interesting eth1 is private VM network, 172.30.212.x Installed software: MySQL, DRBD, Corosync, pacemaker (all stopped)
PRM hands-on: corosync config /etc/corosync/corosync.conf secauth: on (see /etc/corosync/authkey) rrp_mode: none --> single ring service.d/pacemaker --> link with pacemaker
PRM hands-on: corosync startup /etc/init.d/corosync start To check: corosync-objctl grep members To check: tcpdump -n -i eth1 port 5405
PRM hands-on: Pacemaker No configuration file!! /etc/init.d/pacemaker start crm status crm configure show
PRM hands-on: Properties property $id="cib-bootstrap-options" \ stonith-enabled="false" \ no-quorum-policy="ignore" crm configure load update /root/prm/properties.crm crm configure show
PRM hands-on: MySQL config /etc/mysql/my.cnf [mysqld] user=mysql pid=/var/lib/mysql/mysqld.pid server_id=1 # Must be different between hosts log-bin=binlog relay-log=relay mysql_install_db --user=mysql /etc/init.d/mysql start
PRM hands-on: MySQL config (2) cat /root/prm/grants.sql mysql /etc/init.d/mysql stop
PRM hands-on: mysql_repl.crm primitive p_mysql ocf:heartbeat:mysql \ params config="/etc/mysql/my.cnf" \ pid="/var/lib/mysql/mysqld.pid" \ socket="/var/run/mysqld/mysqld.sock" \ replication_user="repl_user" \ replication_passwd="whatapassword" \ max_slave_lag="15" \ evict_outdated_slaves="false" \ binary="/usr/sbin/mysqld" \ test_user="test_user" test_passwd="test_pass" \ op monitor interval="5s" role="master" OCF_CHECK_LEVEL="1" timeout="30s" \ op monitor interval="2s" role="slave" OCF_CHECK_LEVEL="1" timeout="30s" \ op start interval="0" timeout="900s" \ op stop interval="0" timeout="900s"
PRM hands-on: mysql_repl.crm (2) crm configure load update /root/prm/mysql_repl.crm crm configure show crm status
PRM hands-on: ms.crm ms ms_mysql p_mysql \ meta master-max="1" master-node-max="1" \ clone-max="2" clone-node-max="1" \ notify="true" globally-unique="false" \ target-role="master" is-managed="true" crm configure load update /root/prm/ms.crm crm configure show crm status crm configure show --> mysql_replication property
PRM hands-on: VIP_repl.crm primitive writer_vip ocf:heartbeat:ipaddr2 \ params ip="172.30.212.100" nic="eth1" \ op monitor interval="10s" primitive reader_vip_1 ocf:heartbeat:ipaddr2 \ params ip="172.30.212.101" nic="eth1" \ op monitor interval="10s" primitive reader_vip_2 ocf:heartbeat:ipaddr2 \ params ip="172.30.212.102" nic="eth1" \ op monitor interval="10s" crm configure load update /root/prm/vip_repl.crm crm configure show crm status
PRM hands-on: writervip_rules.crm Is that all? crm resource demote ms_mysql crm resource promote ms_mysql colocation writer_vip_on_master \ inf: writer_vip ms_mysql:master order ms_mysql_promote_before_vip \ inf: ms_mysql:promote writer_vip:start
PRM hands-on: writervip_rules.crm (2) crm configure load update writervip_rules.crm crm configure show crm status Better?
PRM hands-on: readervip_rules.crm What happens if we break replication? cibadmin -Q grep readable location No-reader-vip-1-loc reader_vip_1 \ rule $id="no-reader-vip-1-rule" -inf: readable eq 0 location No-reader-vip-2-loc reader_vip_2 \ rule $id="no-reader-vip-2-rule" -inf: readable eq 0 colocation reader_vip_1_dislike_reader_vip_2 \ -200: reader_vip_1 reader_vip_2 crm configure load update /root/prm/readervip_rules.crm
PRM hands-on: readervip_rules.crm (2) crm status Better? Fix replication set global sql_slave_skip_counter = 1; start slave; crm status
PRM hands-on: Name resolution property $id="mysql_replication" \ p_mysql_repl_info="pacemaker-1 binlog.000008 190" Implies that host names resolve from all hosts cat IP_attributes.crm node pacemaker-1 \ attributes p_mysql_mysql_master_ip="172.30.212.1" node pacemaker-2 \ attributes p_mysql_mysql_master_ip="172.30.212.2" crm configure load update /root/prm/ip_attributes.crm
PRM hands-on: Name resolution (2) crm node standby pacemaker-1; crm node standby pacemaker-2 crm node online pacemaker-1; crm node online pacemaker-2 crm configure show property $id="mysql_replication" \ p_mysql_repl_info="172.30.212.1 binlog.000009 107"
PRM hands-on: Adding a node On pacemaker-3: mysql_install_db --user=mysql /etc/init.d/mysql start cat /root/prm/grants.sql mysql Half configure replication mysqldump on master, and restore on pacemaker-3
PRM hands-on: Adding a node (2) On pacemaker-3: start slave; /etc/init.d/mysql stop rsync -av 172.30.212.2:/etc/corosync/* /etc/corosync/ /etc/init.d/corosync start /etc/init.d/pacemaker start
PRM hands-on: Adding a node (3) Is there a 2 nd slave? Why? crm configure edit --> increment clone-max Result?
PRM hands-on: Troubleshooting System log /tmp/mysql.ocf.ra.debug/log crm resource cleanup... crm resource reprobe
PRM hands-on: Cleanup crm configure property stop-all-resources=true on pacemaker-3: crm_node -i --> record the number stop pacemaker-3 VM crm_node -R node_id number from above cibadmin --delete --obj_type nodes --crm_xml '<node uname="pacemaker-3"/>'
PRM hands-on: Cleanup (2) cibadmin --delete --obj_type status --crm_xml '<node_state uname="pacemaker-3"/>' crm configure erase
DRDB
What is DRBD? A Linux kernel module A wrapper around any block device Writes ops sent over the network to the replica 2 states, primary and secondary Effectively a SAN emulator
DRBD principle of operation (2)
DRBD principle of operation (3)
DRBD principle of operation (4) Pacemaker contribution: Manage DRBD (master-slave set) Mount the file system Start MySQL Setup the VIP
DRBD implication on MySQL MyISAM Binary log bind option innodb_log_file_size / innodb_checkpoint_age_target innodb_buffer_pool_restore_at_startup
DRDB Hands-on
DRBD hands-on: /etc/drbd.d/mysql.res resource mysql { device /dev/drbd0; meta-disk internal; disk /dev/pacemaker-1/lvmysql; on pacemaker-1 { address 172.30.212.1:7788; } on pacemaker-2 { address 172.30.212.2:7788; }
DRBD hands-on: DRBD prep. steps drbdadm create-md all /etc/init.d/drbd start cat /proc/drbd drbdadm -- --overwrite-data-of-peer primary all mkfs.ext3 /dev/drbd0 drbdadm secondary all
DRBD hands-on: Properties property $id="cib-bootstrap-options" \ stonith-enabled="false" \ no-quorum-policy="ignore" crm configure load update /root/drbd/properties.crm crm configure show
DRBD hands-on: drbd.crm primitive p_drbd_mysql ocf:linbit:drbd \ params drbd_resource="mysql" \ op start timeout="240" \ op stop timeout="100" \ op monitor interval="10" role="master" \ op monitor interval="30" role="slave" ms ms_drbd_mysql p_drbd_mysql \ meta notify="true" crm configure load update /root/drbd/drbd.crm crm configure show / crm status
DRBD hands-on: fs.crm primitive p_fs_mysql ocf:heartbeat:filesystem \ params device="/dev/drbd0" \ directory="/var/lib/mysql" fstype="ext3" \ options="noatime" \ op start timeout="60" \ op stop timeout="60" \ op monitor interval="30" timeout="40" colocation c_fs_on_drbd inf: p_fs_mysql \ ms_drbd_mysql:master order o_drbd_before_fs inf: \ ms_drbd_mysql:promote p_fs_mysql:start crm configure load update /root/drbd/fs.crm
DRBD hands-on: fs.crm (2) crm configure show / crm status cat /proc/drbd mount mysql_install_db --user=mysql
DRBD hands-on: mysql.crm primitive p_mysql ocf:heartbeat:mysql \ params binary="/usr/sbin/mysqld" \ config="/etc/mysql/my.cnf" \ op start interval="0" timeout="120" \ op stop interval="0" timeout="120" \ op monitor interval="10" timeout="30" group g_mysql p_fs_mysql p_mysql crm configure load update /root/drbd/mysql.crm
DRBD hands-on: ip.crm primitive p_ip_mysql ocf:heartbeat:ipaddr2 \ params ip="172.30.212.100" cidr_netmask="24" \ flush_routes="true" nic="eth1" \ op monitor interval="10" group g_mysql p_fs_mysql p_mysql p_ip_mysql crm configure load update /root/drbd/ip.crm crm configure show / crm status
Enter some text to highlight an upcoming Percona Live.