Building scalable and reliable systems

Transcription

1 Lectures on distributed systems Building scalable and reliable systems Paul Krzyzanowski Background The traditional approach to designing highly available systems was to incorporate elements of fault-tolerant design into the computer system. This ranges from replication to TMR (triple modular redundancy). A fully fault-tolerant system offers non-stop availability. The problem with these designs is their high cost. To deal with the issue of scalability, the traditional approach favored a SMP architecture (symmetric multiprocessors). Problems with SMP designs are that performance gain as a function of the number of processors is sublinear, particularly with more than 6-8 processors, primarily because of contention for resources such as busses, memory, and devices. The goal of clustering is to achieve reliability and scalability by interconnecting multiple independent systems. A cluster is a collection of standard, autonomous, machines, configured so that they appear on the network as a single machine. The application and its data can reside anywhere but this is transparent to the application. This may still fall short of the true single-system image that we long for, since the differences between machines may be visible to the administrator or to internal users of the cluster. However, the single system image should be present for external users. A cluster is a collection of standard computing and storage systems interconnected using a high speed network. By using standard components, we can take advantage of the low price of mass produced PCs, workstations, disk systems, and network cards. A cluster is a distributed, not a parallel, system. Each machine runs a separate copy of the operating system on each node. A management subsystem creates the abstraction of an integrated entity (rather than a rack of PCs) and a cluster API (application programmer interface) provides a collection of system interfaces to perform operations such as determining the set of nodes on a cluster, monitor all state, launch applications, etc. Clustering for reliability Companies such as Stratus and Tandem have built fault-tolerant systems through hardware replication. These are proprietary hardware designs. Systems such as Novell s SFT-III (Software Fault Tolerance) allow two systems to be configured identically with the state of the memory (program execution) and the disk mirrored from one machine to an active backup. If one system fails, the other takes over immediately. There still is a problem. Software faults can crash both systems. It s also a relatively expensive solution since a dedicated backup is required. In the general case, if one server fails, the fault is isolated to that node. The workload is automatically spread over surviving nodes. This allows a machine to be taken down for maintenance without disrupting the system as a whole. A node that picks up work does not have to be sitting idle waiting for a failure. This overhead is that k systems are now doing the work that was done by k+1 systems. Important issues in cluster reliability are how application failover takes place and how long it takes the cluster to realize that one of its members is dead. Design options for failover are: cold failover When a cluster detects that an application is dead on one machine, it Rutgers University CS417: Distributed Systems 1

2 restarts it on another machine. warm failover A running application checkpoints itself periodically. When it dies, the cluster restarts the last checkpointed image. It may also maintain a log of inputs to perform a roll-forward to bring the application to the point where the original system died. hot failover The application s state is lockstep synchronized on a backup system. When the application dies, the backup takes over. Clearly, hot and warm failovers have the advantage of bringing the application to the state last presented to the user. The issue with a checkpointed restart is to ensure that all activity is logged if rollforward is used to ensure that it has all the state necessary to execute properly and not will not generate redundant disk activity (for example). Issues for both warm and hot failovers are that system specific state is not present in the devices (e.g. a connection over a particular serial port) and that relevant operating system state is also reestablished (e.g. open files, network connections). Another issue in failover is whether the cluster can support application failover to different machines or only to a single machine. This is known as multi-directional failover. Once failover takes place, an extra level of fault tolerance is achieved if the application can be failover to yet another system if the backup system dies. This is known as cascading failover. Clustering for scalability The appeal of scalability via a clustered architecture is that the level of scalability can be greatly enhanced from the typical 2-8 CPUs possible with an SMP system. Memory contention problems are reduced, workload can be balanced across multiple servers, and nodes can be added incrementally, reducing a need for an initial expensive outlay. Most clustered systems today do not realize significant scalability as they cannot support more than a few servers (IBM is a notable exception). Large server farms tend to use IP-based load balancing rather than operate as a true cluster. To a large extent, this is because there is not much of a business need for systems capable of highly-parallel operations. In areas where this need exists, there are many custom efforts. One area where scalability is highly desirable, for example, is frame rendering for computer graphics (animation). The typical solution is to maintain a queue of frames to be rendered and have a dispatcher that remotely executes the process on an available server. One popular effort for supercomputing clustering arose from the Center of Excellence in Space Data and Information Sciences (CESDIS), a division of University Space Research Association at the Goddard Space Flight Center. This is a collection of software and configuration known as Beowulf. It was initially built to address problems associated with large data sets in Earth and Space Science applications. What made it possible to build highly scalable supercomputing systems is: - commodity off-the-shelf (COTS) computers have become cost effective - low cost, high speed, switchable networking is available (switches allow network bandwidth to scale with the number of hosts) - publicly available software (Linux, GNU compilers and tools, MPI (message passing interface) and PVM (parallel virtual machine) libraries). These allow software to be written that is largely hardware and platform independent In addition to this, many more people have accumulated experience with parallel software: generally concluding that it is difficult to write and solutions tend to be custom. Programs that do not require fine-grain computation and communication can usually be ported to run effectively on Beowulf clusters. Under Beowulf, the machines (nodes) are generally dedicated to the cluster (to the application that will be run). This means that the performance of the nodes is not subject to external factors, making load balancing easier. The interconnect network among the cluster elements is generally a separate network that is isolated from the external network, making the network load be determined only by the application run on the cluster. A global process ID is provided, enabling processes on one node to be able to send signals to a process on another node. Rutgers University CS417: Distributed Systems 2

3 Beowulf is not a single product but rather a collection of publicly available software that helps build a cluster. It generally runs on Linux systems (and also on the variants of BSD). The system includes: - BPROC: Beowulf Distributed Process Space. This allows process Ids to span multiple nodes in a cluster and provides a mechanism for starting processes on other nodes - Network device drivers. In additional to supporting conventional drivers, Beowulf contributed channel-bonded ethernet drivers, enabling network traffic to be striped across multiple ethernet cards. - Hardware monitor driver provides a /proc interface for an LM78 hardware monitor. - Programming tools that include: - PVM (parallel virtual machine) and MPI (message passing interface) libraries - Distributed shared memory (page based with software-enforced page ownership and consistency policy) - Other tools, such as a cluster monitor and global ps/top/uptime tools. Depending on the application needs, different configurations of Beowulf are possible: Networking: - single entry point to cluster: one monitor and one keyboard and a single external IP address. The rest of the cluster hides behind this with IP masquerading. Users log onto the main node and spawn remote jobs with ssh. - multiple nodes: each has its own external address. Each system may double as a desktop machine. Disks/file systems: - local disks synchronized nightly (except for /var, /tmp, /etc/sysconfig) with a utility such as rsync - local disks not synchronized: useful for applications that do only number crunching - NFS root (or some other distributed file system): useful for programs that need disk synchronization but are not disk bound Process management: - batch system: job scheduling is left to the programmer (job runner). Generally a queue is used to ssh remote jobs. This is good for highly parallel applications, such as supercomputing or rendering - preemptive scheduling/migration: this is not a part of the Beowulf solution but may be incorporated into the system. Processes can be automatically migrated based on cluster status. This is geared to an environment that is not dedicated to the cluster or one where different types of jobs are run. Two popular packages that provide this capability are Condor (not open source) and Mosix - fine-grained control: programs control their own synchronization and load balancing using MPI and/or PVM libraries and bproc for process dispatch. Architecture Shared resources Two cluster architectural models prevail: shared-disk and shared-nothing. A shared-disk cluster allows multiple systems to share access to disk drives while a shared-nothing cluster has no shared devices. With a shared-disk cluster, each node has its own memory but storage resources are shared. This works well if applications do not generate much disk I/O since a shared device can be a point of contention. Disk access must be synchronized to maintain data integrity. Synchronization is achieved by using a distributed lock manager (DLM) to serialize requests. The benefit of a shared-disk cluster is that everyone Rutgers University CS417: Distributed Systems 3

4 sees the same storage system. If an application moves to another machine, it can continue making the same disk requests (but cache coherency remains an issue). The detriment, as just mentioned, is disk contention. Under a shared-nothing cluster, each system has its own storage resources (cache and disk). The benefit is that there is no need to deal with DLMs and there are no problems with cache coherency. If a machine A needs data that resides on a disk connected to machine B, A sends a message to machine B with the request. If there are many such requests, performance is still an issue. Moreover, if machine B fails, storage resources have to be switched over to a live node or be completely inaccessible. In general, shared nothing is better at providing linear scalability because of lower contention. A hybrid architecture is also possible, where shared-nothing can be used for scalable, easilypartitioned applications and shared-disk be used for difficult-to-partition applications or those that mostly do disk reads rather than disk writes. Cluster interconnects Traditional LANs and WANs are often too slow to serve as a cluster interconnect (connecting server nodes, storage nodes, I/O channels, and perhaps memory pages). This led to the emergence of the System Area Network (SAN). A SAN is a switched interconnect that can switch any cluster resources together. It provides low latency, I/O without processor intervention, and a scalable switching fabric (it is easy to add more nodes). A key point in a SAN is the low processing overhead since processing overhead affects scalability. Since it is a dedicated network there is no need no manage a complex protocol stack such as TCP/IP. An example of a system area network is Tandem s ServerNet. Heartbeat network To detect system faults and distinguish them from network faults, it is useful for clustered systems to maintain redundant networks. One network can be used to send periodic heartbeat messages to test a machine s liveness. Ideally, this network will be a reliable network with a bounded response time. Lucent RCC uses a serial line connection for sending heartbeats. Microsoft s Cluster Server uses a SCSI bus for this, with one machine performing a reset and waiting to see if another reestablishes a connection within 10 seconds (a rather long time). Software The goal of software is to hide the complexity of clustering from applications, application programmers, and end users. The first is difficult, the next easier, and the last much easier to accomplish. For example, a cluster-aware database can partition databases and tables across nodes transparently and will know how to divide queries across nodes. Cluster-aware applications can also fix deficiencies in the underlying system s ability to provide failover by managing write-ahead logs and detecting restart. Most vendors of clustering software provide an API to enable developers to design such cluster-aware applications. At the operating system level, clustering software aids in providing a single-system image. Applications should be allowed to run in an environment were resources can be accessed in a deviceindependent manner and the application is not aware of what machine it is using or whether resources are local or remote. To an administrator, the cluster should be managed as one logical entity. For example, a ps command on a Data General cluster shows all processes throughout the cluster. An exec system call runs a process on a machine selected (by certain parameters) by the cluster manager. Rutgers University CS417: Distributed Systems 4

5 Cluster server architectures Sample cluster systems are Stratus RADIO, Compaq s Proliant cluster, DECsage Avaliable Server Environment, Sun Microsystem s Solaris MC system, the Microsoft NT cluster, and others from companies such as HP, IBM (SP line), and Tandem. Stratus RADIO As an example of a high-end cluster architecture, we will look at the Stratus RADIO (Reliable Architecture for Distributed I/O) system. It is a rack mounted collection of 6 24 compute and storage nodes. Each node rack has two communication nodes and redundant power supplies. A compute node is a dual processor (PC architecture) with on-board memory, small swap disks, and a network interface. It runs Unix or NT and communicates with others via TCP/IP. Each compute node has its own IP address. Disk nodes use standard disk technology but run a non-standard operating system that acts as a front-end to the network. It allows RADIO nodes to share disks, simulate n-way multiported disks, and provides software functionality for disk mirroring. Internal communication amongst compute and storage nodes is via fast ethernet or ATM. Network adapter nodes allow the interconnection of multiple RADIO systems as well as interconnection to other systems. An internal management network monitors the status of nodes and provides the system administrator with control of the operator interface to PCs comprising the cluster. Nodes going off-line and on-line are detected within milliseconds via hardware support. Every component in the system can be hotswapped. Both software and hardware can be upgraded without disrupting continuous service. The system has no single point of failure. Microsoft NT The Microsoft NT Cluster Server architecture is an extension of NT. The system is structured into a three-tiered architecture: Top tier Cluster abstractions Middle tier Provides distributed operations Bottom tier Windows NT and drivers Failover manager Resource monitor Cluster registry Global update Quorum Membership Windows NT server Cluster disk driver Cluster network drivers The Windows NT Cluster runs on a network of conventional PCs running NT cluster software. The cluster software aims for a reasonable level of fault tolerance but does not support lockstep fault tolerance running applications cannot be migrated from failed machines to working machines. A key aspect in the architecture is that of name abstraction. An application can be shut down on one machine and restarted on another there is no physical dependency on the name or IP address of any machine. A quorum device keeps track of who s in charge. A SCSI disk is typically designated to act as the quorum device. It provides several functions: - provides arbitration and knowledge of who s in charge at any time - arbitrates and provides a place for doing checkpoints - stores configuration information (via logs). Rutgers University CS417: Distributed Systems 5

6 The SCSI-based quorum device is also used to support the heartbeat of the cluster. Suppose the network connection between two machines, A and B, goes down. B can use the quorum device to determine if A is still alive by doing a low-level SCSI reset, waiting for A to re-establish its disk connection and, if that fails, timing out and taking charge. The global update mechanism allows for the propagation of global updates in the system to all nodes. All nodes must commit to the update and a rollback takes place if any node fails. Every operation in the system is tightly synchronized with atomic broadcasts. Applications and outside entities do not use any aspects of the internal namespace (names, IP addresses). Load balancing and fault tolerance A very common need for scalability is not running parallel applications but rather running multiple instances of a specific server, for instance a web server. The need here is for load balancing. There are several reasons for doing this: - load balancing - failover (requests will go to surviving servers) - planned outage management (maintenance, hardware/software upgrades) Load balancing with REDIRECT A very simple system can be assembled to provide load balancing and/or fault tolerance for a set of web servers. The HTTP protocol provides for a REDIRECT error code, where a client is told to access a web page from somewhere else. We can take advantage of this part of the protocol and have the client make requests to a server proxy. This server proxy can then serve a page if it has it cached or else select one Back-end server Local area network Local proxy firewall Internet firewall Server proxy Back-end server users Back-end server Figure 1 Web cluster with REDIRECT of the available servers and send the requesting client a REDIRECT message to that particular server (Figure 1). Load balancing with software load balancers Software on a server can perform the task of load balancing without sending messages back to the client to issue another request. IBM s Interactive Network Dispatcher Software is an example of this type of software. One load balancer receives all incoming requests. It balances the load by using a number of weights and measures and supports load balancing among specific services (HTTP, FTP, SSL, NNTP, POP3, SMTP, telnet). Upon determining the destination system, it forwards the request to that machine, but rewrites the source address to be that of the original machine so that replies do not go back through the load balancer. This works in environments where client to server communication has a lower bandwidth than server to client communication (the web is a good example of this). A similar product is also available from Microsoft as part of the Windows 2000 clustering solution. Rutgers University CS417: Distributed Systems 6

7 Load balancing with routers Routers have been getting smarter in recent years, providing features such as increased sophistication in firewalling and packet filtering. A particularly interesting router in the domain of fault tolerance and load balancing is the Cisco LocalDirector series. This family of routers load balances traffic across multiple servers, allowing one to mix operating systems, hardware, and applications. The LocalDirector is (currently) a 200MHz Pentium processor with 32 MB RAM and can support over 800K simultaneous TCP connections. Its configuration allows one to assign one or more virtual addresses to one or more real (internal) addresses. When an outside request comes in, the address is mapped to the physical address. Special assignments can be made per port (e.g. telnet destination). A sample configuration is shown in Figure 2. Since the LocalDirector is a router and is operating system independent, its capabilities for making decisions on directing traffic are limited to the data (rather than requesting the load on a particular machine). Several choices are available for load balancing: - pick the machine with the least number of TCP connections - factor in weights when selecting machines (this allows one to make one machine preferable to another, all other things being equal) - pick machines round-robin - pick the fastest machine (where speed is measured by response times to SYN packets in requesting TCP connections). Values can also be set to determine if a server failed and should be reassigned. The LocalDirector supports failover. Two units can be cabled together such that one will take over in case the other fails. Web servers Virtual address Physical address Nttp servers switch (netnews) Figure 2 Local Director sample configuration LDAP servers (directory) An installation that uses load balancing with this product is the Excite web search engine. Their system contains over 16 SGI Challenger computers acting as web servers. They receive over two million hits each hour. The system configuration comprises a Cisco 7000 high-end core router connected to the Internet, filtering and sending packets to a LocalDirector, which distributes the traffic amongst the SGI web servers. A router such as the LocalDirector can also aid in providing somewhat automated control of address mapping to achieve fault tolerance. For instance, a simple fault-tolerant system might contain redundant Internet Service Providers (ISPs) connected to a local area network that feeds into two LocalDirectors (for fault tolerance). These feed their packets onto another local area network that connects to a server machine and a backup. Both the server and a backup share RAID (Redundant Array of Inexpensive Disks) disks at the backend and all devices are connected to uninterruptible power supplies. Rutgers University CS417: Distributed Systems 7

8 References LocalDirector Student Guide version 1.5.6, Cisco Systems, available from Building Secure and Reliable Network Applications, Kenneth P. Birman, 1996 Manning Publications Co. Microsoft Exchange 5.5: Windows NT Clusters, White Paper, 1997 Microsoft Corporation, available from Network and Internetwork Security, William Stallings, 1995 Prentice Hall. Distributed Operating Systems, Andrew Tanenbaum, 1995 Prentice Hall, pp Rutgers University CS417: Distributed Systems 8