L.A.M.P.* - Shaman-X High Availability and Disaster Tolerance. Design proposition

Transcription

1 Request For Comments L.A.M.P.* - Shaman-X High Availability and Disaster Tolerance Design proposition *L.A.M.P.: Linux + Apache + MySQL + Php applications Version 0.1 draft, 27 Aug 2003 by Dominique Chabord dominique.chabord@bluedjinn.com For Shaman-X project Up to date version of this document at History draft 27-Aug-2003 by Dominique Chabord Creation LAMP-Shaman-X High Availability and Disaster Tolerance (HA & DT) project proposition is to set up a framework regarding High Availability and Disaster Tolerance Expected LAMP-Shaman-X HA & DT project outcome is a guide-line document and a set of prototyping scripts to illustrate design principles and concepts for system integrators who build high availability systems and disaster tolerant solutions based on LAMP and Shaman-X products (WDX, MDX, HDX,and CDX). Contents Introduction 2. Architecture and design principles 3. Todo list and sandbox 4 More information This document can be used under the GNU Free documentation license conditions. LAMP-Shaman-X HA & DT project edited on 8/27/03 page 1/8

2 2. Introduction Linux is very successful in entering the enterprise information systems, in particular because excellent and proven products exist and support critical and scalable infrastructures. Among them, Linux, Apache, MySQL and Php are already leading the industry. Users, who decide in favour to these free products, get the immediat benefits of open source and free software and will improve control on their infrastructure for a long time. Nevertheless, free software will not just do as well as proprietary products. It allows also outstanding convergence of people of all sorts who bring experience, know how and new ideas. Shaman-X project proposes high availability and disaster tolerance products and promotes system integration and consulting in these domains. This document is a request for comments addressed to the freesoftware users and developers community to elaborate best in class solutions for outstanding information systems resiliency. LAMP-Shaman-X HA & DT project edited on 8/27/03 page 2/8

3 2. Architecture and design principles Interdependancy: distributed and layered software Distributed applications require the availability of several computers to deliver functionality to users. Multi-tiered applications and nowdays web services increase computer interdependancy. Unlike a cluster in which several nodes are interconnected by design, distributed application interdependancy involves indifferently stand-alone computers or clustered computers, regardless to their original design or their position on the network. This happens when these computers are found to provide a service which is necessary to the user. One resource is unavailable and the service to user will be partial (best case). Layered products are the basement of distributed applications. In this project, we consider Apache Webserver, MySQL server and storage management software (RAID 1 and DRBD). Every layered software implements duplication, replication, failover and remote administration tools which depend on separate resources, and, as a direct consequence, implies new dependancies between primary and shadow resources. Manage collaborative groups of resources High Availability and Disaster Tolerance for these applications must therefore consider groups of heterogeneous computers which LAMP-Shaman-X HA & DT project edited on 8/27/03 page 3/8

4 collaborate for a given application. A watchdog function must ensure that all pieces of service necessary for this application are correctly performed. Groups differ from an application to the other. All groups share the same hardware infrastructure, network computing power and storage resources. Best performances require harmonious repartition of processing over the infrastructure, to avoid stressing too much some computers while other stay idle. LAMP-Shaman-X HA & DT project edited on 8/27/03 page 4/8

5 2.2. Oversized hardware, data redundancy and geographical dispersion When hardware is oversized for the applications, administration and maintenance are made easier, because some resources can be put off line while updated, upgraded or repaired. Another possibility offered when hardware is oversized is to define some levels of redundancy, for the cases when a resource fails, and to implement automated failover. In the case of WDX, processing can be rearranged on remaining computers when one of them fails. If the application is highly critical, hardware can be made fully redundant and will often be dispersed geographically to ensure operation continuity even if several resources fail or a computer room is down (disaster tolerance). Failover can restart applications from fresh data if they are available, either by sharing access to centralised storage, or by replication mechanisms as DRBD (network disk replicator) and MySQL replication function (shadow databases) Intelligent failover must chose between different failover policies and differentiate failures which require limited actions to restart from a twin resource, from bad cases, generally when several failures happen, where more global rearrangements of processing and data copies are necessary to restore service to users. Several intricated redundancy/failover capabilities coexist in large architectures LAMP-Shaman-X HA & DT project edited on 8/27/03 page 5/8

6 2.3. Shaman-X design for LAMP applications Shaman-X today delivers a versatile distributed lock manager called WDX which aims at implementing intricated failover policies over distributed environments. WDX implements mutual watch and failover capability in arbitrary defined groups of computers (2 to 16 in each group). A computer which runs several layers of software will belong to several groups. For example, a server replicates disks, replicates a MySQL database, runs PHP based applications and publishes Webpages from Apache across redundant networks. To conform Shaman-X recommendation, this server belongs to several groups to fulfil all these missions, as shown in the following illustration. WDX missions Network Failover Apache Web server Web server Restart HDX PHP Applications Applications Restart CDX MDX WDX Failover MySQL Database Database Restart MySQL Replication Database Failover Shaman-X Data RAID 1 DRBD Replication Storage Failover WDX layered missions for layered products LAMP-Shaman-X HA & DT project edited on 8/27/03 page 6/8

7 In practice, a mission corresponds to an instance of WDX, which describes the group and triggers recovery scripts upon failure or state transition WDX is available today. Other components of Shaman-X reference implementation are under specification. They are: MathsDoctor-MDX. MDX is a PHP application which is in charge of complex situation modelling, to manage intelligent failover in interdependant environments. MathsDoctor is able to determine situations based on operators like QUORUM, INHIBITION, STABILITY, PERSISTANCE, and DELAY. HealingDoctor-HDX. HDX is a PHP administration interface which generates html pages and publishes them to user s web browser. HDX provides a graphical interface for all Shaman-X products. HDX also implements administration tools for every mission (ie every group), so missions are coordinated and groups are synchronized. For example, swapping MySQL master and replications must be synchronized with layered missions, stopping applications here, and restarting them there... ConfigurationDoctor-CDX. CDX is not yet specified. The first approach is to build a Php application which interfaces with MySQL and stores there all configuration data for WDX (mission configuration data and scripts), for MDX (solution models based on logical equations) and HDX (resource technical descriptors and mission descriptors). 2.4 Shaman-X is itself a LAMP application From this design, Shaman-X products, WDX,MDX,HDX take advantage of the reliability of the infrastructure and benefit directly from data replication, hardware redundancy, and standardized system administration. LAMP-Shaman-X HA & DT project edited on 8/27/03 page 7/8

8 3. Todo list and sandbox Implementation of WDX missions falls in different categories: 1) Application must be stopped somewhere and started somewhere else 2) Storage is shared between several computers and one of them will get mastership to mount it in read/write mode (entry level of high availability clusters) 3) Data are replicated across a network from a primary to a secondary, and failover is a combination of a closing activity on the primary and opening activity on the secondary (ex: MySQL, DRBD) 4) Networks failover depends on network topology. For stateless servers, IP aliasing can be used. For stateful servers, deconnection and reconnection of clients can be automated with WDX on the client. Last case is network to network failover, when all servers and all clients have network redundant interfaces and swapp simultaneously from one network to the other one. This section is to be updated with some progress on different mission kinds. General Shaman-X to-do list is available at 4. More information Pointers: Project web site: Services: Project manager: dominique.chabord@bluedjinn.com This document: LAMP-Shaman-X HA & DT project edited on 8/27/03 page 8/8