High Availability Solutions with MySQL best OpenSystems Day Fall 2008 Ralf Gebhardt Senior Systems Engineer MySQL Global Software Practice ralf.gebhardt@sun.com 1
HA Requirements and Considerations HA Solutions from MySQL Third-Party HA Solutions for MySQL Additional HA Resources plus Q & A 2
HA Requirements and Considerations Service Level Agreements (SLAs) Budgets & Timeliness to Delivery Fail Over Requirements Fault Tolerance Single Point of Failures Open Source, Proprietary & Mixed Tech Stacks Security Hardware Requirements Storage Considerations RAID SAN NAS Shared-Nothing Operating System Interoperability Geographic Redundancy What impact does a hardware or software failure have on the business? 3
What is Replication? The duplication of data to more then one location The use of additional hardware or software for Availability Reliability Fault tolerance Performance Asynchronous vs. Synchronous Safeguarding data loss Database Replication Statement vs Row-based Replication File-Based Replication Block-level (i.e. DRBD) 4
What is Replication? The duplication of data changes to more than one location Asynchronous Replication update customer Synchronous Replication update customer 5
What is Clustering? Clusters are often used for: Operating System Databases Application servers Load balancers Web servers Just about any service that needs to be highly available Redundant nodes provide service when components fail Eliminate single points of failure Multiple network connections Redundant data storage (SAN, RAID) Shared-Nothing vs. Shared-Disk 6
MySQL HA Solutions Clustering & Geographical Redundancy Clustering Technologies Replication Technologies MySQL Cluster MySQL + DRBD MySQL + SAN Well-Managed MySQL Replication Unmanaged Small Business ISPs & Mainstream Business Data Centers Banking Medical Telco Military Defense 9 9. 9 9 9 35 days 4 days 8 hours 50 mins 5 mins % 7
HA Requirements and Considerations HA Solutions from MySQL MySQL Replication MySQL Replication and Heartbeat MySQL, DRBD and Heartbeat MySQL Cluster MySQL w/ Shared-Storage & Clustering Agents Third-Party HA Solutions for MySQL Additional HA Resources plus Q & A 8
MySQL Replication Asynchronous Statement Row-based (5.1) Mixed (5.1) Read vs. Write Response times Load balancing Scale Out Administration Backups Upgrades Testing Geographic Redundancy Requirements Availability Automated IP Fail Over Automated DB Fail Over Typical Fail Over Time Resynchronization of Data Geographic Redundancy Scalability Built-in Load Balancing Read Intensive Write Intensive # of Nodes Support No No Varies No Yes Scale-Out Yes No Dozens for Reads 9
MySQL Replication Topologies Master > Slave Master > Slaves Masters > Slave (Multi-Source) Master > Slave > Slaves Master < > Master (Multi-Master) Circular (Multi-Master) 10
Writes & Reads Inside MySQL Replication Web/App Server Writes relay binlog mysqld I/O Thread SQL Thread index & binlogs data Replication mysqld binlog data MySQL Master MySQL Slave 11
MySQL Replication plus Linux Heartbeat Heart-Beat Protocol Messages sent between nodes Failed heart-beat starts fail over process Manages a Virtual IP IP Address Takeover Characteristics Easy to configure with low complexity (however ) Open source Low cost Virtual IP management is automatic Asynchronous data replication Manual or automated fail over Recovery and multiple failures can get complicated Requirements Availability Automated IP Fail Over Automated DB Fail Over Typical Fail Over Time Resynchronization of Data Geographic Redundancy Scalability Built-in Load Balancing Read Intensive Write Intensive # of Nodes Support Yes No Varies No Yes Scale-Out Yes No Master/Slaves 12
MySQL Replication & Linux Heartbeat IP Management Active MySQL Server Passive MySQL Server Asynchronous MySQL Replication Heartbeat implements serial, UDP, and PPP/UDP heartbeats together with IP address takeover, this solves MySQL Replication s fail over problem, but, data resynchronization is not automatic and can be complex. 13
MySQL, Heartbeat & DRBD Cluster Distributed storage Similar to network RAID 1 Synchronous replication No special networking components Excellent performance Blocks vs. rows of data Manages inconsistencies of data during a failure Hides the complexity of many recovery actions Heartbeat manages fail over and virtual IPs Requirements Availability Automated IP Fail Over Automated DB Fail Over Typical Fail Over Time Resynchronization of Data Geographic Redundancy Scalability Built-in Load Balancing Read Intensive Write Intensive # of Nodes Support Yes Yes < 30 secs Yes MySQL Replication No Yes Possible 1 Active 14
MySQL, Heartbeat & DRBD Cluster IP Management Active DRBD Server Passive DRBD Server Synchronous Block Replication 15
Managing Failures Passive DRBD Server X IP Management Active DRBD Server X X X 16
Data Resynchronization after Host Failure IP Management Passive DRBD Server Active DRBD Server Resynchronization Active Server is still able to process transactions during a resynchronization operation 17
System Requirements System Requirement Hosts Support Minimum 2 (More for scale-out or partitioning) Hardware Memory Storage Network Operating System COTS or SMP - 32 & 64-bit Multiple NICs for redundancy Varies (More is generally better) (32 KB per 1 GB block) SCSI, RAID 0+1 for I/O Performance 100 Mbps or better (Gigabit recommended, others supported) Linux (Redhat, Suse, CentOS, Fedora, Debian) 18
MySQL Cluster In-Memory Database Disk Data (6.2) Shared-nothing storage Synchronous replication Fast-automatic data fail over High performance NDB API No special hardware Best for primary key lookups Transactional Online backup No automated IP failover Requirements Availability Automated IP Fail Over Automated DB Fail Over Typical Fail Over Time Resynchronization of Data Geographic Redundancy Scalability Load Balancing Read Intensive Write Intensive # of Nodes Support No Yes < 3 Secs Yes Yes (w/ Replication) Scale-Out Yes (Look ups) Yes 5.0: 64 5.1: 255 19
System Requirements System Component Requirement Hosts Maximum of 255 total nodes (48 Data Nodes) Hardware COTS Advanced TCA 32 & 64-bit x86, PowerPC & SPARC Memory Storage Network Operating System Varies on size of database, # of hosts, # of replicas Shared-Nothing - Memory & Disk Data SCSI or RAID for I/O performance 100 Mbps or better (Gigabit recommended, SCI supported) Linux (Red Hat, SuSE), Solaris, HP-UX, Mac OSX, others 20
Cluster Components SQL Node (MySQL) NDB API (Application) Data Node (NDB Storage Engine) Management Node Standard SQL Interface Scale-out for Performance Enables Replication High Performance C++ API Developer s Guide Data Storage (Memory/Disk) Automatic & User-Defined Partitioning Local & Global Checkpoints Scale-out for Capacity & Redundancy Configuration Arbitration Use Two for Redundancy 21
4 Data Node MySQL Cluster Clients Application/SQL Nodes (NDB API and/or MySQL Server) Management Nodes Data Nodes NDB Storage Engine 22
Geographic Redundancy As of 6.2 support for asynchronous replication between Clusters is possible Enables a higher degree of availability and continuity Asynchronous MySQL Replication NDB API NDB API NDB Storage Engine Cluster 1 NDB Storage Engine Cluster 2 23
Data Distribution & Partitioning authid (PK) 1 fname Albert lname Camus country France Base Table 2 Ernest Hemingway USA 3 Johan Goethe Germany 4 Junichir o Tanizaki Japan Partition Partition authid (PK) fname lname country authid (PK) fname lname country 1 Albert Camus France 2 Ernest Hemingway USA 3 Johan Goethe Germany 4 Junichir o Tanizaki Japan 24
Distribution & Partitioning: 2 Data Nodes Two Data Nodes (Physical) * Complete copy of database on each node Two Replicas (Logical) * Two copies of each partition One Node Group * One set of identical Data Nodes Node Group authid (PK) fname lname country authid (PK) fname lname country 1 Albert Camus France 1 Albert Camus France 2 Ernest Hemingway USA 2 Ernest Hemingway USA 3 Johan Goethe Germany 3 Johan Goethe Germany 4 Junichir o Tanizaki Japan 4 Junichir o Tanizaki Japan 25
Distribution & Partitioning: 2 Data Nodes Query: SELECT fname, lname FROM author WHERE authid BETWEEN 1 AND 3; Result: Albert Camus Ernest Hemingway Johan Goethe In a two node cluster, even if we loose a Data Node, we can still satisfy any query. X Node Group authid (PK) fname lname country authid (PK) fname lname country 1 Albert Camus France 1 Albert Camus France 2 Ernest Hemingway USA 2 Ernest Hemingway USA 3 Johan Goethe Germany 3 Johan Goethe Germany 4 Junichir o Tanizaki Japan 4 Junichir o Tanizaki Japan 26
Distribution & Partitioning: 4 Data Nodes Four Data Nodes (Physical) * Half of database on each node Two Replicas (Logical) * Two copies of each partition Two Node Groups * Two sets of identical Data Nodes Node Group Node Group authid (PK) fname lname country authid (PK) fname lname country 1 Albert Camus France 2 Ernest Hemingway USA 3 Johan Goethe Germany 4 Junichir o Tanizaki Japan 27
Distribution & Partitioning: 4 Data Nodes Query: SELECT fname, lname FROM author WHERE authid BETWEEN 1 AND 3; Result: Albert Camus Ernest Hemingway Johan Goethe In a four node cluster, even if we loose a Data Node in each Node Group, we can still satisfy any query Node Group 1 Node Group 2 X X authid (PK) fname lname country authid (PK) fname lname country 1 Albert Camus France 2 Ernest Hemingway USA 3 Johan Goethe Germany 4 Junichir o Tanizaki Japan 28
Detecting Lost Connections - Heartbeats 29
Data Node Failure & Recovery Server 3 failure is detected Server 4 handles Server 3 requests 1 2 3 4 Node Group 1 Node Group 2 30
Data Node Failure & Recovery Server 3 begins copying data from Server 4 Complete or just changes since failure 1 2 3 4 Node Group 1 Node Group 2 31
Data Node Failure & Recovery Once data is fully restored, Server 3 rejoins the cluster 1 2 3 4 Node Group 1 Node Group 2 32
MySQL w/ Shared Storage & Clustering Agents Active/Passive likely configuration Multiple instances not allowed concurrent access to same data files Automated management Virtual IPs Fail over Data synchronization Mounting file systems Characteristics High cost (storage, hardware, software) Idle resources Longer fail over times High initial complexity Many options and proven vendors 33
Additional HA Solutions for MySQL HA/Performance Networking for MySQL Cluster Dolphin SCI Interconnect Operating System Clustering Sun Solaris Sun Cluster Microsoft Windows Clustering Services Red Hat Cluster Suite High Availability Middleware Continuent m/cluster Backup Software Zmanda http://solutions.mysql.com/ 34
MySQL Professional Services/Support MySQL Cluster Jumpstart MySQL Scale-Out, HA and Replication Jumpstart MySQL Architecture & Design 24 x 7 Production Support http://www.mysql.com/support http://www.mysql.com/products/enterprise/drbd.html Learn more about MySQL Professional Services and these bundles at http://www.mysql.com/consulting/ 35
Questions? best OpenSystems Day Fall 2008 Ralf Gebhardt Senior Systems Engineer MySQL Global Software Practice ralf.gebhardt@sun.com 36