Database Resilience at ISPs. High-Availability. White Paper

Transcription

1 Database Resilience at ISPs High-Availability White Paper

2 Internet Service Providers (ISPs) generally do their job very well. The commercial hosting market is segmented in a number of different ways but one view is on how sites are hosted: Virtual hosting, Dedicated hosting and Client equipment. The latter tends to be used by clients with significant in-house expertise and the two former represent the majority of web presence. ISPs offer a variety of service options and configuration interfaces and hardware options but typically a standard build is offered but real high-availability is rarely an option. ISPs are very good at building highly available solutions from a networking point of view but two problems are normally left to users to solve: Global DNS - which protects against catastrophic failure at an ISP, Database Resilience - whether local or across multiple sites. Global DNS In August 2003 large segments of the Internet went down for days when Eastern USA suffered a catastrophic power melt-down. The loss of power over such a wide area and extended time meant that even those companies with generators could not stay online as the upstream facilities had failed. While this is an unusual occurrence it graphically demonstrates how even the best ISP facility can fall victim to service outages. Figure 1 - Global DNS Views of Eastern USA at night, from space before and after the power cuts of August The only way to cater for this kind of outage is to have hosting in disparate ISP locations and to run a dynamic DNS service, often referred to as Global DNS. Software solutions and hardware appliances are available to offer this service off-the-shelf and again this requires a disparate and integrated deployment to be effective. Web Site Structure Many if not most web sites are now built using a back-end database. It may be used for storing content and layout information but often contains live information about customers, stock, transactions and logistics. If the database fails then the web site can no longer function properly and business transactions can be lost: Customers buying online will simply switch to another supplier and they will do so if there are just a few seconds of delay. Best Buy TM for instance went down on the run up to Christmas and instead of their advertising

3 improving their sales, the servers crashed and Circuit City TM, a close competitor, picked up an unexpected boost to sales. Many customers who switch in these circumstances switch permanently. Database Resilience At the high-end, where clients supply their own equipment, database servers are often configured in a cluster and may also use off-site replication. The clustering protects against server or database application faults and the replication protects against the catastrophic failures. Dynamic DNS (Global DNS) Figure 2 - Replication Example of clustered, replicated databases Standard Switch Web & Database s Oracle replication (Data Gaurd) Clustered Remote Nodes Mirror/RAID Disks Affordable Database Resilience The majority of web sites are built on ISP owned equipment and use MySQL rather than Oracle or other expensive commercial database applications. MySQL is fast, free and provides all the key features required for even a large online presence. However, ISPs don t typically offer an off-the-shelf HA solution. So in the event of a server crash or database freeze, the client is typically required to restart the application or rebuild the content. Some ISPs offer backups but what about all the transactions since the backup was taken? And how long does it take to get back online? MySQL Changes Figure 3 - MySQL HA MySQL Remote Replication MySQL s Clustered Usually RAID Disks Primary Warm Standby (Replicated Data)

4 MySQL does have a replication feature set and this is quite effective but, significantly, it does not provide any automated failover capability. The replication ensures that the data is secure and can be used both locally and with disparate systems. To ensure web site availability, clustering needs to be added to this solution. Figure 3 shows a configuration with replication to the remote node. This is a solution that can work within a Global DNS configured environment. Sorry - My Load Balancers Do All That - Don t They? This is a common misconception. Load balancing switches are excellent at recognising that a web server is no longer functioning correctly and they may even be able to recognise that a database is no longer serving content but they aren t able to actually control the database. Even if they are global load balancers. The MySQL replication model is master to slave(s) and to promote a slave into a master requires administrator interaction. RSF-1 is used to check if a fault exists and to attempt to restart the master before promoting the slave and it does this automatically. The following diagram gives an overview of what technologies can improve reliability at each layer - without administrator intervention - i.e. with automatic recovery. Load Balancing MySQL Dynamic DNS (Global DNS) RSF-1 Load ( Balancing + MySQL ) ISP/Telco Feed Switches/Telco In Load Balancer Switches 1 Figure 4 - System View Overview of which technology can address single points of failure (SPOF) Web s Database Disk - RAID With NetMon to yet further improve availability 2 - Standard on most OS s or provided by hardware Using both load balancing Dynamic DNS switches (Global DNS) and traditional load balancing switches in the ISP, a reliable solution can be constructed right down to the web server. MySQL provides the capability to replicate the data but no automatic detection and fail over. RSF-1 added to these technologies provides the complete solution.

5 The web servers connect to the database via a virtual IP address (VIP) and this is managed by RSF-1, making the failover likely to be transparent to end users in most circumstances. An Ready ISP Solution The traditional way to solve the database availability solution at a local level is to use shared disk: Figure 5 - MySQL HA Traditional Cluster MySQL s Clustered Mirror/RAID Disks { In most environments locally shared disk is an excellent and cost effective solution but in an ISP environment this very much a bespoke solution and would increase costs, so a software only solution is typically preferred: Primary Hot Standby MySQL Changes Figure 6 - MySQL HA MySQL Local Replication MySQL s Clustered Primary Mirror/RAID Disks Hot Standby (Replicated Data) Under normal operation, MySQL on the primary server logs any database updates to it s binary log files, periodically a SQL thread on the secondary server connects to the primary server and retrieves it s latest binary log files. Once received from the primary server, the latest binary log files are saved locally as relay log files by the thread before being applied to the MySQL database by a second MySQL thread running on the secondary (standby) server. If the primary server fails, RSF-1 on the secondary server will detect the failure and immediately initiate a failover of the MySQL service exactly as if it were in a shared storage environment.

6 However, if MySQL should fail on the primary server instead of the server itself, mysqlmon on the secondary server will only promote it s MySQL database to replication master (from replication slave) provided it s relay log files are up-to-date with the binary log files on the replication master before it crashed. This ensures a replication slave will only become a replication master if it has an up-to-date copy of the MySQL database locally, guaranteeing data consistency. This feature is only available with RSF-1, not with vanilla MySQL replication. How MySQL Replication Works MySQL s replication works by having a dedicated process to monitor the transaction log for write operations and send the changes to another instance, which will roll back the transactions on another copy of the database. The transactions are normally rolled back in an asynchronous mode; that is the master/active database does not wait for confirmation from the slave/remote node that the transaction has been rolled back. This is for performance reasons but is at the potential cost of integrity; some transactions may possibly be lost during a failover. This type of replication provides a very basic disaster recovery and is adequate for very small companies where substantial data loss will not cause any impact on business operations or financial operations. It also requires manual intervention to intiate a switch-over. Slave Figure 7 - Basic Type Basic Replicated Configuration In a commercial environment this sort of basic and unverifiable replication is just unacceptable and an integrated solution that monitors the replication and verifies that it is up to date and is able to automatically failover is a prerequisite to delivering commercial service. RSF-1 has been closely integrated with MySQL and a monitoring and management component called MySQLmon provides a number of advanced features for a two node replicated HA system: Both nodes are configured as MySQL replication masters, which in theory would allow both nodes to accept read and write requests. However, connections from clients are managed by RSF-1 and a primary is designated. The secondary node is closely monitored to ensure that transactions continue to be received from the primary. In the event that the secondary node stops receiving updates then it will be locked to prevent failover until such time as the transactions catch up. Should the primary node fail, then the secondary node will be reconfigured to become the new primary node and the client connections will be moved over. When the original primary node recovers MySQLmon will reconfigure it s role, so it becomes the secondary node and prompt it to catch up, ready to take over should the new primary fail. When the catch up is complete it will be placed into automatic failover mode again.

7 Application read/write Reporting read only MySQLmon is an agent for RSF-1 which is closely integrated with MySQL and RSF-1 specifically for replicated environments, it is available with RSF-1 from High-Availability.Com or authorised partners. Figure 8 - Managed RSF-1 Managed Replicated MySQL Database Public Locked The application server connects to the primary database and read and write operations are performed. The write transactions are logged and those logs are transmitted to the secondary database which applies the same changes to the database. Optionally a read only application server may, for load balancing purposes, be connected to the secondary database. Index & Logs Figure 8 shows an RSF-1 managed MySQL cluster with the replication logic expanded, although the interaction with RSF-1 is not shown. Application read/write Reporting read only MySQLmon closely monitors the state of both instances of the database and ensures that updates are being applied and where required changes the roles of the databases. If the secondary server should fail, as and when it recovers, MySQLmon will place it into recovery mode, while the outstanding changes are applied or if that is not possible then a full synchronisation will be initiated. MySQLmon will place the node into automatic failover mode when the databases are synchronised. The secondary server is promoted to become the new primary server. Logging of new write transactions is started and client connections are moved across. Figure 9 - Failure Primary Failure Dead Application users will probably not be aware of the failure. Depending on how the administrator configures the setup the, optional, read only client connections can remain active or if performance issues are a concern they can be dropped until the secondary server is restored. Index & Logs Figure 9 shows an example where the primary server has died, which may be the result of an application fault, network interface failure or perhaps a hardware fault. RSF-1 will not allow the former primary node to re-enter the cluster as the primary until RSF-1 has established the state of the other node. This prevents a potential conflict situation that could arise if two nodes accepted write requests. The risk is that a write operation on a row could be completed in two places at once, which RSF-1 with MySQLmon prevents despite the cluster including multiple MySQL master nodes.

8 Application read/write Reporting read only Once the old primary server is restored it is demoted to be the secondary server and begins accepting the changes from the new primary server. This is shown in Figure 10. The read only clients connections will be moved over to the restored (standby) server. Restored Public Locked Index & Logs This solution provides MySQL customers with a fully automated high-availability solution which can be implemented on low cost hardware and operating systems over large distances. The solution only requires network connections between the nodes which makes deployment much cheaper than fibre channel or other dedicated links. The bandwidth required depends on the write transaction rate and transaction size but high bandwidth links will reduce latency of sychronisation between nodes not just allow greater throughput. Figure 10 - Recovery Recovery. customer facing operations include; VoIP telcos, hosted CRM solutions and media streaming content providers. Clients that use RSF-1 and MySQLmon for MySQL Changes MySQL s Clustered Figure 11 Ultimate Availability Mirror/RAID Disks { Primary Hot Standby Replication{ Warm Standby Figure 11 shows a combination of the two solutions which provides for rapid failover. In the event of most faults to a local node takes over but in the case of site failure then a remote node will take over. This solution is implemented by customers looking for the ultimate in availability and disaster recovery in a MySQL environment.