White Paper. Storage Area Networks; Unclogging LAN s and Improving Data Accessibility

Transcription

1 White Paper Storage Area Networks; Unclogging LAN s and Improving Data Accessibility This important technology is moving into the mainstream in distributed networking and will be the normal, adopted way of attaching and sharing storage in a few short years. Michael Peterson, Strategic Research Corporation Mylex Corporation Ardenwood Blvd. Fremont, CA Tel: (510) Sales Fax: (510) International Sales Fax: (510)

2 Table of Contents Executive Summary... 3 Server-Dependent or Server-Independent Storage... 4 SAN Taxonomy... 5 Fibre Channel SAN s... 7 SAN s and Clusters SAN s and NAS SAN-Attached RAID Array Requirements Mylex External RAID Controllers for NT Clusters Product Line SAN Features References Copies of this and other white papers may be obtained from the Mylex web site ( RAID Controllers Are Not Created Equal; Many Will Not Survive on Wolfpack-II Clusters DAC960PJ/PR Two Node Clustering Legal Notice Compaq is a registered trademark of Compaq Computer Corporation. ServerNet is a registered trademark of Tandem Computer Incorporated. Pentium Is a registered trademark on Intel Corporation. VAXClusters is a registered trademark of Digital Equipment Corporation. NetWare is a registered Trademark of Novell Corporation Other product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. MYLEX 2

3 Executive Summary The most significant challenge facing IS executives is satisfying increasingly diverse network requirements and managing network complexity to reduce total cost of ownership (TCO). Success depends on delivering the: Bandwidth required by web-based, i.e. multimedia, and warehouse applications, Latency and predictability required by time-sensitive applications, e.g. video conferencing, Performance and resiliency required by mission critical applications, e.g. OLTP. When computing resources were used only for internal operations, the cost of information bottlenecks and network failures was limited to lost productivity. As computing resources are used to engage customers, as well as manage operations, bottlenecks and failures translate into lost business in addition to lost productivity. The first design objectives of Storage Area Networks (SAN s) is to mitigate network congestion by moving high volume storage-related data movement off the client network and onto specialized sub-networks, often referred to as the networks behind the servers. LAN SAN Figure 1. LAN s and SAN s SAN-based storage (and storage traffic) is removed from the LAN, externalized and shared by UNIX and PC servers; sharing storage devices is the second benefit of a SAN topology. In addition to de-congesting client networks and amortizing storage costs across servers, other benefits of the SAN architecture include: Enabling flexible, modular expansion by de-coupling server and storage investments, Improving bandwidth and capacity scaling by eliminating SCSI and PCI bottlenecks, Increasing application performance with multiple gigabit SAN links, Simplifying systems integration and enriching storage management, Improving fault tolerance and availability with multiple paths to data, and Lowering total cost-of-ownership (TCO). MYLEX 3

4 Server-Dependent or Server-Independent Storage Server-Dependent Storage A paradigm shift from centralized to distributed storage began in the 1980 s. Peer-to-peer networks, UNIX and the advent of PC servers lead to the notion that moving computing resources closer to workgroups would increase productivity. This lead to islands of computing with centralized mainframes and distributed UNIX, NetWare and NT servers. A collection of disparate networks and gateways attempted to tie it all together. IS managers were faced with multiple copies of inconsistent data, networks that were expensive to manage and corporate assets (data) that were vulnerable to intrusion and network bottlenecks The AberdeenGroup, a respected market research firm, refers to this environment as server-dependent storage. LAN LAN UNIX UNIX NT NT NT Figure 2. Server-Dependent Storage Server-Independent Storage With Gigabaud shared and dedicated bandwidth, emerging SAN technologies mirror LAN technologies. The AberdeenGroup advises: Unless enterprises view and implement storage as if it were part of a giant network across the enterprise, they will pay too much for their storage and will face extreme, labor-intensive difficulties in performing vital storage-related functions, such as managing the storage and backing up and moving critical data. LAN LAN UNIX UNIX NT NT NT SAN Figure 3. Server-Independent Storage MYLEX 4

5 SAN Taxonomy Storage Area Network or System Area Network SAN is one of the more overloaded acronyms in computer jargon. Its meaning is speaker- (or writer) sensitive. To systems people, SAN means Systems Area Network, and to storage people, it means Storage Area Network. Some people consider both definitions of the acronym synonymous. However, while System Area Network and Storage Area Network topologies can be similar or even identical, there is an important distinction between the two technologies. System Area Network A System Area Network is a specialized network used in clusters or other multi-host configurations for node-to-node and node-to-device (primarily disk), and device-to-device communications that provides both high bandwidth and low latency. Low latency is the distinguishing characteristic of a System Area Network. Short message latency across a System Area Network is generally less than 10 micro seconds; an order of magnitude less than Fibre Channel or Gigabit Ethernet. Short message low latency is a prerequisite to high performance for applications that are distributed across cluster nodes, e.g. parallel database management system. Instances of a distributed application in a cluster environment need to frequently exchange messages to synchronize program execution or synchronize access to shared resources. Most System Area Networks use proprietary protocols, however, this is expected to change when the VI Architecture is introduced later this year or next year. The VI Architecture is an interconnect-independent set of protocols and API s that standardize the interface between OS s and cluster interconnects. ServerNet, developed by Tandem Corporation, and SCI, an ANSI standard implemented by Dolphin, are examples of a System Area Networks. Note: A System Area Network technology can also be used to implement a Storage Area Network. Storage Area Network A Storage Area Network can use a specialized or a standard network technology (e.g. Fibre Channel) and its purpose is to provide direct connections between servers and storage devices, and between storage devices and other storage devices, e.g. a storage array and a tape library. The design objective of Storage Area Networks is high bandwidth for moving large amounts of data but not necessarily low latency. Storage Area Networks have been implemented for years with ESCON and HIPPI interfaces, and more recently with SSA and Fibre Channel. Storage Area Networks can be used in homogeneous environments, e.g. all UNIX servers, or heterogeneous environments, e.g. a mix of UNIX and NT servers. A SAN can be local to the servers or remote from the servers, and can be dedicated to a single server are shared by multiple servers. Storage Area Networks generally use a standard networking protocol such as SCSI or IP. System Area Networks use a fabric topology with low latency switches to interconnect servers and devices. Storage Area Networks use switches or hubs, or a combination of switches and hubs, to interconnect devices, and can be implemented in fabrics, bus or loop topologies. SAN will be used in the Storage Area Network context in the balance of this paper. MYLEX 5

6 Related Terminology SAN-Attached Storage (SAS) Shared storage devices that are connected to servers, and possibly each other, via a Storage Area Network typically implemented with Fibre Channel or SSA in open system environments. Network-Attached Storage (NAS) Network-Attached Storage is implemented with devices that directly attach to client networks, typically an Ethernet LAN, and provide optimized file services to clients and servers on the network using standard file I/O protocols such as NFS and SMB. The Network-Attached Storage device is essentially a specialized server that functions as a server in a client-server relationship with other networked computers requesting file access. Mylex s AutoNet and Auxpex NAS systems are examples of leading-edge Network-Attached Storage devices. Note: Storage attached to an application server (but not directly attached to a LAN) is sometimes referred as network-attached storage. In a sense, this is true since clients can access server-dependent storage over the network. However, this is stretching the NAS definition since general purpose application servers are not optimized for file serving. SAN Topologies SAN s can be designed with switched fabrics or arbitrated loops using Fibre Channel or SSA, or in bus topologies using SCSI. Fibre Channel and SSA are scalable technologies that can be used to build extended SAN configurations. SCSI is bandwidth, distance and device attachment limited, and hence, useful only for small SAN configurations. SAN Interconnect Devices In addition to switches, hubs and routers are interconnect devices that can be employed to construct SAN networks. Hubs are an interconnect mechanism used in loop configurations to simplify cable management and enhance fault tolerance. Routers are useful for interconnecting complex SAN s particularly over long distances for data vaulting or disaster protection applications. SAN Interfaces HIPPI, ESCON, SCSI, SSA and Fibre Channel are candidate SAN interfaces. HIPPI has been around for years and its use has been primarily limited to technical applications. ESCON is the dominate interconnect in mainframe environments. Ultra-2 LVD can be used as a low cost SAN interface but speed is limited to 80 MB/s and distance to 12 meters. SSA is an IBM technology with similar (workload dependent) performance characteristics to Fibre Channel. SSA has become a popular SAN interconnect. Fibre Channel appears to be emerging as the standard interface for SAN s based on its industry-wide support, and its performance, distance and flexibility characteristics. MYLEX 6

7 Fibre Channel SAN s Industry Standard Fibre Channel is an industry standard that defines a set of high performance serial I/O protocols and interconnects for flexible data transfer developed by an ANSI committee and support by over 70 companies. The Fibre Channel standard was designed to: Provide a common interface for transferring large amounts of data at high transmission rates and with low error rates, Enable the simultaneous use of different transport protocols, such as, SCSI and IP, over a common interface, and Support multiple physical interfaces and transmission media with varying distance and cost characteristics. Networks and Channels Fibre Channel was designed to provide seamless integration between client networks and channels that connect storage and other devices to servers. A network connects heterogeneous computers located anywhere in the world and enables them to communicate with one another at any point in time on a peer-to-peer basis. Consequently, networks use complex protocols to authenticate users, set up sessions, route data and correct errors over potentially low quality transmission lines. Complex protocols generally impose high overheads on networked computers. Conversely, channels are employed in structured, predictable environments to connect storage and other devices to serves over distance limited, low error rate transmission paths. Fibre Channel has not been widely deployed in networks; the cost of Fibre Channel is high relative to 10/100 and Gigabit Ethernet, and network infrastructures are in place to support Ethernet. However, Fibre Channel is rapidly becoming a storage interconnect standard. Fibre Channel provides the bandwidth, distance and flexibility required for Storage Area Networks: Full duplex transmission at 25 MB/s and 100 MB/s today at distances up to: 25 meters between devices using video or mini-coax copper cable, 500 meters between devices using multi-mode fibre cables, and 10,000 meters using single mode fibre cables, Full duplex 200 MB/s and 400 MB/s data rates in the not too distant future, High addressibility: 126 devices in an arbitrated loop, and 1,000 s of devices in switched fabrics, Low error rates and end-to-end error checking for high data integrity, Optional redundancy for high availability, Low cost in an arbitrated loop topology, and Scalable bandwidth in switched fabric topologies. MYLEX 7

8 Loops or Fabrics Fibre Channel devices, frequently called nodes, have a single port or multiple ports to communicate with other nodes. Today, most Fibre Channel Host Bus Adapters (HBA s) provide a single port but HBA s with dual ports are expected in the future. Each port has a pair of electrical (for copper cables) or optical transceivers (for fibre cables); one for transmitting data and the other for receiving data. The pair of conductors is referred to as a link. Fibre Channel nodes implement a single link or a dual link for fault tolerance. Fibre Channel Storage Area Networks can be configured as an arbitrated loop with bandwidth shared by the devices on the loop or as a switched fabric that provides scalable bandwidth. Arbitrated Loops Nodes arbitrate to gain access to the loop, and then pairs of nodes establish a logical point-to-point connection to exchange data with the other nodes on the loop acting as repeaters. Loop bandwidth is shared by the nodes; only one pair of nodes can be communicating at any point in time. Fibre Channel Arbitrated Loops are similar in operation to other shared media networks, such as Ethernet or Token Ring. Figure 4. Fibre Channel Arbitrated Loop and Fabric Topologies Switched Fabrics In a switched topology, the full bandwidth of a link is available to pairs of communicating nodes and multiple pairs of nodes can be simultaneously transmitting and receiving data. As Fibre Channel nodes are added to a switched configuration, the aggregate bandwidth of a Storage Area Network increases. The fabric switch is an intelligent device that provides a crossbar switching function enabling multiple pairs of nodes to simultaneously communicate. MYLEX 8

9 Hubs or Switches Hubs and switches are interconnect devices that can be used in Storage Area Networks implemented with Fibre Channel links. Both are available with advanced management capabilities similar to LAN management techniques; SNMP is commonly offered, and some devices conform to the new Web-Based Enterprise Management (WBEM) standard. Both devices can also be cascaded to increase node connectivity; hubs up to arbitrated loop limit of 126 nodes, and switches to thousands of nodes. Hubs Except for two node configurations, hubs are generally used in arbitrated loop topologies, and can be used in complementary roles to switches in fabric topologies. Hubs have relatively low costs and simplify cable management. In addition, hubs contain bypass control switches that enable failed nodes on a loop to be bypassed without compromising the integrity of the loop. The hub s bypass control switches also enable nodes to be hot plugged into a Storage Area Network or removed with affecting loop operation. Hub Hub Loop A Loop B Figure 5. SAN With Dual Arbitrated Loops and Hubs Switches Switches are used to create Fibre Channel fabrics. They provide the same resiliency features as hubs and enable the fabric s aggregate bandwidth to scale as nodes are added. With hub- connected Storage Area Networks, aggregate bandwidth remains constant as nodes are added but bandwidth per node decreases; more nodes sharing a fixed amount of bandwidth. With switched fabrics, bandwidth per node remains constant as nodes are added but the aggregate bandwidth of the fabric increases proportionally to the number of nodes. Fabric Loop Switch Hub Figure 6. SAN With Switched and Shared (Loop) Interconnects MYLEX 9

10 SAN s and Clusters Clusters Defined A cluster is a group of autonomous servers, called nodes, that work together to enhance reliability and scalability, and can be managed as a single entity to reduce management costs. Clusters always share storage devices and can share data. Cluster Models Clusters have been implemented using the Shared Disk and Shared Nothing models: Shared Disk Model -- Data is simultaneously shared by cluster nodes. An access control mechanism, generally referred to as a distributed lock manager, synchronizes access from multiple nodes to shared data. Example: Digital s VAXClusters. Shared Nothing Model -- Access to data is shared but at any point in time, disk volumes are exclusively owned by one of the nodes. Nodes access data that they do not own through the nodes that own the data. Example: Microsoft NT Clusters. Advocates of the Shared Nothing model argue that Shared Nothing clusters are more scalable because the overhead of a distributed lock manager increases as nodes are added and eventually bottlenecks the cluster. NT SMP Server NT SMP Server NT SMP Server NT SMP Server P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P Shared Memory Shared I/O Shared Memory Shared I/O Shared Memory Shared I/O Shared Memory Shared I/O External Controller External Controller External Controller External Controller Figure 7. Four Node Cluster With Shared Access to RAID Arrays Clusters Use SAN Technologies Data is removed from the network and stored on the network behind the servers, Storage model is server-independent, not server-dependent, Access to storage devices are shared, generally across high performance links, I/O bandwidth and capacity can be scaled to meet application requirements, Redundant links or fabrics are for fault tolerance and increase application availability, Data can be centralized for increased security and enhanced storage management, Modular, flexible expansion; server and storage investments are de-coupled, Storage nodes can communicate and exchange data (in advanced cluster designs), Hubs or switches are for cable management, resiliency and bandwidth scaling, and Total cost-of-ownership (TCO) decreases. MYLEX 10

11 SAN s and NAS Network-Attached Storage Defined SAN s and Network-Attached Storage (NAS) are complementary technologies and will likely evolve in the similar directions. Network-Attached Storage is implemented with devices directly attach to client networks, typically Ethernet LAN s, and provide optimized file services to clients and servers on the network using standard file I/O protocols such as NFS and SMB. Network-Attached Storage is essentially a specialized server that functions as a server in a client-server relationship with other networked computers requesting file access. NAS NAS SAN Figure 8. NAS on LAN Segments and Connected to SAN With NAS devices, storage is moved from PC s, workstations and workgroup servers to file access optimized NAS devices where it can be secured and managed. NAS design objectives are similar to SAN objectives but at much lower price points. Workstation-dependent storage is moved to workstation-independent storage, Data is centralized at the workgroup level where it can be more easily secured, shared, RAID-protected, backed-up and accessed in an optimal fashion, LAN s can be more easily designed to avoid bottlenecks with the centralized NAS storage on shared or dedicated 10 Mb/s, 100 Mb/s or Gbit segments, Modular expansion is enabled; workstation and storage investments are de-coupled, Redundant links can be used to increase data accessibility, and Total cost-of-ownership (TCO) decreases. MYLEX 11

12 SAN-Attached RAID Array Requirements SAN Side Connectivity Fibre Channel Interface -- Fibre Channel is the interface standard for SAN s. It provides the performance, distance and scalability (in switched fabrics) that is required for SAN environments. The Fibre Channel standard is supported by over 70 leading companies and a broad range of Fibre Channel interconnect devices (hubs and switches) and storage products (RAID arrays, tape and optical libraries, and disk, tape and optical drives) will be available in a performance, feature and cost competitive market environment. Dual Channels High performance SAN-attached storage devices will require multiple front-end SAN channels for high performance and fault tolerance. Dual channel SAN devices offer twice the potential performance of single channel devices at marginally higher costs and continue to provide data access if a SAN interconnect device or link fails. Heterogeneous Host Support Most F1000 enterprises have heterogeneous environments with mainframes in the data centers, and UNIX, NT and NetWare servers distributed across the network. The investments in these systems (acquisition costs and more importantly, the application/database development, infrastructure and training costs) are substantial. Migration to a more uniform environment is a multi-year proposition. SAN-attached RAID arrays should be storage volumes formatted with different file systems and allow heterogeneous hosts to access these volumes. Data Availability Features Duplex RAID Controllers SAN s should be designed without any single point of failure that can cause a storage device on the SAN to become inaccessible. SAN-attached arrays should be configured with duplex controllers and with the disks connected to both controllers. Multiple SAN interfaces (on each controller) and duplex controllers with shared disks provide the level of fault tolerance required in SAN configurations. Redundant paths to data is a necessary but not a sufficient condition for high availability. Transparent Host Independent Failover In addition to redundancy, controllers should implement a transparent failover/failback scheme such that logical disks, i.e. logical arrays, are continuously accessible. For low-end SAN s implemented with a SCSI bus, this requires each controller to be able to assume the SCSI ID of its partner in addition to its own. With Fibre Channel, each controller requires a port held in reserve to accommodate a controller failover, i.e. assume a failed controller s port ID; although the Fibre Channel standard allows ports to have multiple addresses, this feature has not been implemented in Fibre Channel chips. Failover and failback should be implemented such that these transitions occur without any required host intervention. Alternate Path Support -- Following the no single point of failure principle, SAN-attached servers should have redundant Host Bus Adapters (HBA s) with alternate path software support. Alternate path is generally implemented as a software driver and it provides a level of indirection between the OS and HBA s. If one HBA fails, the driver redirects I/O s intended for the failed HBA to the alternate HBA so that I/O requests can be satisfied without NT knowing that an alternate path to storage was used. MYLEX 12

13 Data Integrity Features Mirrored Write Caching -- Most external controllers implement write-back caching to enhance performance but few implement a mirrored write-back caching scheme to protect data. Data in a controller s cache is vulnerable to power loss or controller failures until it is written to disk. To make matters far worse, if data in a write-back cache is lost, the application is oblivious to the loss since the controller has acknowledged the write as complete. Controller battery back-up units can hold-up cache contents during power outages but cannot move cached data to an operational controller which can write it to disk. Mirroring write-back caches across controllers solves this problem and can be designed with minimal effect on cost or performance. In controllers designed with a mirrored cache architecture, I/O s are written to cache memories in multiple controllers before the write is acknowledged as complete. If one controller subsequently fails, the surviving controller flushes the contents of the failed controller s cache (which are stored in its cache) safely to disk. Mirrored caches protect data like mirrored disks. Data Management Features SAN-attached RAID arrays need to support data management software typically implemented on one or more of the servers connected to the SAN. This includes data back-up and archiving, hierarchical storage management (HSM), disk mirroring and all the widely used RAID levels, on-line expansion of logical drives and on-line addition of logical drives. Bandwidth Scaling Capabilities Scalable I/O Performance SAN s require a balanced design. The number (and compute power) of servers attached to a SAN can vary by orders of magnitudes and SAN s will evolve over time. Expanding a SAN with additional nodes will result in marginal performance gains if the SAN-attached RAID subsystems do not have horsepower to feed the additional nodes. Scalable compute power requires scalable I/O power. Storage subsystems with external controllers inherently provide scalable I/O performance since additional units can be incrementally added without disrupting cluster operation. However, a large number of under-powered controllers are less cost-effective and more difficult to manage than fewer controllers with performance more closely matched to the cluster s I/O requirements. Active-Active Operation -- Duplex controllers can operate in active-passive or activeactive mode just like cluster nodes. In this context, active-active implies a duplex controller configuration with both controllers simultaneously servicing SAN I/O requests. This is analogous to the active-active operation of NT cluster nodes. To realize the full performance potential and total cost-of-ownership (TCO) effectiveness of a SAN, all storage resources, must contribute to performance. Controllers that operate in activepassive mode with one controller idle until the other fails is a waste of a valuable cluster resource. MYLEX 13

14 Capacity Scaling Capabilities Scalable I/O Capacity -- System storage has been increasing 50% per year since the first disk was invented. SAN s require controllers with surplus back-end channel capacity to accommodate expanding storage needs. Controllers that only support a few disk channels are marginal for SAN applications. Controllers with four to six back-end channels can obviously accommodate more storage but their arrays can also be configured to provide performance and data availability benefits. Logical arrays on controllers with two or three disk channels are striped vertically down a channel. Since applications direct most I/O to a single logical array, vertically stripping arrays can cause a single I/O processor (IOP) to become a bottleneck (the hot disk phenomena moved into the controller). Horizontal striping balances the I/O load across IOP s. External RAID Controller IOP IOP IOP IOP IOP Figure 9. Vertically and horizontally stripped arrays. If an IOP fails, vertically striped arrays on the failed channel become unavailable to the controller with the failed IOP. However, if a RAID 5 array is horizontally striped across channels, then an IOP failure causes the loss of a single disk which RAID 5 algorithms can repair. This failure mode is identical to a single disk failure in a RAID 5 set MYLEX 14

15 Mylex External RAID Controllers for NT Clusters Mylex Product Line Mylex offers a seamless product line of external RAID controllers designed to meet the performance, connectivity, cost, interface, topology, data integrity, data availability and management requirements of SAN environments. A common RAID code base delivers a uniform set of data integrity, data availability and performance optimization features across hardware platforms: At varying levels of performance and storage connectivity, With Ultra, Ultra-2 LVD and Fibre Channel interfaces, and At price points for entry level clusters and with performance for enterprise clusters. Mylex external RAID controllers are available in simplex configurations for network servers and duplex configurations for clusters. In duplex mode, cluster-focused features are implemented to accelerate performance, expand connectivity, protect data and guarantee data accessibility. Active-active controller operation, Dual cluster host ports, Surplus storage channels, Transparent failover / failback, and Cache mirroring. SF FL SX SX SF SF FL FL SX SX SF FL Figure 10. Relative Performance Figure 11. Data Integrity and Availability The relative performance of simplex (light gray) and duplex (dark gray) configurations of Mylex external controllers is illustrated in Figure 9. Data protection and accessibility is important in any cluster environment; hence, robust data integrity and availability features are uniformly implemented across the product line. MYLEX 15

16 Product Line SAN Features Heterogeneous Server Support Mylex external RAID controllers can accommodate a SAN that is configured with multiple operating systems such as UNIX and NT servers. Disk volumes can be configured into logical arrays and then formatted with NTFS file systems or any UNIX file systems. Up to eight logical arrays accessed as LUN s (Logical Unit Numbers), can be configured on each controller. A logical array is the unit of space allocation and RAID level protection. Each logical array can have a different RAID level to provide the level of performance and fault tolerance required by the application(s) using the array. NT and UNIX servers can simultaneously access a Mylex external controller formatted with NT and UNIX volumes which provides the level of flexibility required in the enterprise with mixed server types or those in the process of migrating from UNIX to NT servers. NT Server NT Server SAN Port 0 Active Port 1 Reserved Port 0 Active Port 1 Reserved Port 0 Active Port 1 Reserved Port 0 Active Port 1 Reserved FL Controller FL Controller FL Controller FL Controller Figure 12. DAC FL Attahed to a Fibre Channel SAN In Figure 12, two pairs of duplex DAC FL controllers are SAN-attached. Each pair has redundant paths to two disks. The dark shaded disk on each controller has been formatted with a UNIX file system and the lightly shaded disks have been formatted with NTFS files systems. The two SAN-attached servers have paths through the SAN interconnect to both pairs of redundant controllers, and redundant paths from the controllers to the disks. Figure 12 shows an example of a UNIX server accessing data through the controller pair on the left and the NT server accessing data through the controller pair on the right. Mylex SAN-attached controllers can provide reliable simultaneous access to heterogeneous servers in SAN topologies. Each controller can be accessed by all the servers on the LAN simultaneously. MYLEX 16

17 Active-Active with Transparent Failover / Failback One of the key benefits of SAN s is high data availability; SAN devices can fail without impacting server access to data (aside from a momentary cluster transition delay). Active-active server operation Server heartbeat monitoring Transparent server failover and failback Data Availability A key benefit of Mylex controllers is high data accessibility; controllers can fail without impacting server access to SAN-wide storage (aside from a momentary controller transition). Active-active controller operation Controller heartbeat monitoring Transparent controller failover and failback Mylex external controllers provide analogous and complementary behavior for cluster nodes. For cost-performance effectiveness, Mylex controllers support active-active operation (both controllers simultaneously satisfy I/O requests). Mylex controllers provide high availability by heartbeat monitoring and transparent controller failover / failback. Like clustered servers, Mylex controllers are linked by a private network used to transmit I m alive heartbeat messages (Figure 14) between the controllers. Other controllers operate in active-passive mode similar in concept to a hot spare disk. The passive controller waits for the active controller to fail and then springs into action. Active-active controller implementations, such as Mylex, deliver performance approaching double that of active-passive controllers Heartbeats Controller 0 Controller 1 Figure 13. Controller Heartbeat Messages As shown in Figure 13, all Mylex external RAID controllers have dual SAN ports for high performance and fault tolerance. Mylex controllers can be attached to servers, NT clusters and low-end SAN s via SCSI busses, to mid-range and high-end SAN s via Fibre Channel loops or fabrics. MYLEX 17

18 SCSI Bus Attachment During normal operations, each controller has a unique SCSI ID. The controllers disk channels are connected to the disks and each other; each controller has equal access to all drives. If a controller fails, the surviving controller senses the absence of heartbeats, assumes the ID of the failed controller in addition to its own (Figure 15), and updates its data structures as necessary with configuration information stored on disk (COD). The failover process is transparent to the cluster nodes since the nodes still see the same two SCSI ID s on the storage interconnect.. A A B Heartbeats Controller 0 Controller 1 Figure 14. Controller Failover When the failed controller is replaced, it is detected by the surviving controller which allows it to restart and returns the failed controller s SCSI ID, and then it starts processing I/O (Figure 14). Controller failover / failback operations are transparent to cluster nodes. During the failover / failback process, cluster nodes simply continue sending I/O requests to the same SCSI ID s across the shared storage interconnect. As far as the nodes are concerned, these commands are processed identically whether both controllers are functional or one has failed. A Heartbeats B Controller 0 Controller 1 Figure 15. Controller Failback Controller failover typically takes three to five seconds from failure until the surviving controller responds to commands sent to the failed controller s ID. During this period, commands sent to the failed controller cause a Selection Timeout which cause I/O commands to be resent according to the recovery algorithm implemented in HBA firmware, the driver or OS. The recovery algorithm reissues the selection and command sequence to recover from transitory Time-out conditions. Failover occurs fast enough that retries result in successful selection and outstanding tasks queued to the failed controller are cleared by a Flush Queue command and re-issued. The surviving controller receives the re-issued commands, places them in its execution queue and the cluster resumes normal operation. MYLEX 18

19 Fibre Channel Loops and Fabrics The DAC SF and DAC FL external RAID controllers are ideally suited for Fibre Channel SAN configurations. Each controller has dual SAN ports for accelerated performance and resiliency in the face of interconnect failures. The ports can be directly attached to SAN servers or attached to Fibre Channel hub or switch. In high availability SAN topologies, DAC SF and DAC FL controllers are configured in pairs (duplex controller configurations). One fibre port on each controller is reserved (attached to the SAN but inactive) to accommodate failover of its partner. [Note: Current Fibre Channel chips do support multiple addresses for each port.] If a controller fails, its active port address is taken over by the reserved port of its partner, and its partner services I/O requests addressed to itself and to the failed partner. When the failed controller is replaced, it is detected by the surviving controller which allows it to restart and returns the failed controller s port ID, and which then it starts processing I/O. FC HBA FC HBA FC HBA FC HBA SAN Port 0 Active Port 1 Reserved FL Controller Port 0 Active Port 1 Reserved FL Controller Port 0 Active Port 1 Reserved FL Controller Port 0 Active Port 1 Reserved FL Controller Figure 16. DAC FL Attahed to a Fibre Channel SAN Figure 16 shows a pair of duplex DAC FL controllers attached to a SAN and servicing I/O requests from four SAN-attached servers. An arbitrary number of DAC SF and DAC FL controllers can be attached to a SAN; I/O bandwidth and storage capacity scales in direct proportion to the number of attached controllers. Failover and failback operations are host independent and transparent to server operation. Aside from a momentary delay during the controller transition, the SAN-attached servers are unaware that a controller has failed and its address has failed over to a reserved port on another controller, or that a controller has been replaced and a failback operation has occurred. MYLEX 19

20 Mirrored Write Caching To accelerate application performance, Mylex external RAID controllers use sophisticated caching algorithms for both read and write operations. The write cache can be optionally set to Write-Through or Write-Back mode. In Write-Through mode, the Command Complete message is not returned to the cluster node that issued the Write until the data has been written to disk. Write-Through caching ensures that data is written to disk before acknowledging I/O completion to the system and improves performance in applications that frequently read data that has recently been written by retaining a copy of the data in the cache. In Write-Back mode, higher performance is achieved by returning the Command Complete message when data has been written to cache but prior to writing it to disk. This permits the node to clear its write command queue an order of magnitude faster than if it waited for the data to be written to disk; hence, more commands can be processed in a given amount of time. This also allows multiple write operations to be coalesced by the controller into larger contiguous blocks before being written to disk in bursts without missing revolutions of the disk and enduring long latency times. Cache Structure Controller 0 Read Ahead Write Back Figure 17. Conventional Controller Cache Without protection provided by Mylex controllers, Write-Back caching may subject data to certain risks. After a node has been told that a Write command was completed, it is unable to re-issue the command if a failure prevents the cached data from actually being written to disk. To ensure data integrity in the event of power failures, Mylex controllers are protected by battery back-up and / or active UPS monitoring. If AC power is lost, the write caches are held up by the battery backup unit or UPS and flushed to disk when AC power is restored before new commands are accepted from a node. If AC power is lost to the UPS, write caching automatically switches to Write-Though. When AC power is restored to the UPS and it has recharged, Write-Back caching is turned back on. MYLEX 20

21 To protect the write cache data in the event of controller failure, Mylex external controllers implement a cache mirroring technique. In duplex configurations, data written to the Write- Back cache in one controller is immediately copied to the write buffer in the second controller (Figure 18). Read Ahead Write Back Write Buffer Read Ahead Write Back Write Buffer Controller 0 Controller 1 Figure 18. Mirrored Cache Writes Disk Bus If the controller servicing the I/O Write commands fails, the surviving controller has a copy of the writes and will immediately complete the Write operations for its failed partner (Figure 19). The cache mirroring process is accomplished by a Write operation from one controller to the other using one of the disk channels. To minimize bus contention with disk I/O operations, the least busy channel is selected. Read Ahead Write Back Write Buffer Read Ahead Write Back Write Buffer Controller 0 Controller 1 Disk Bus Figure 19. Surviving Controller Flushed Failed Controllers Writes to Disk MYLEX 21