Evaluation Report. PACS Education. An Introduction to Storage Area Networks (SAN) Crown Copyright June MHRA Educational Report MHRA 03055

Transcription

1 June 2003 Evaluation Report NUMBER MHRA PACS Education An Introduction to Storage Area Networks (SAN) MHRA Educational Report MHRA Crown Copyright 2002 Crown Copyright 50

2 Crown Copyright 2003 Apart from any fair dealing for the purposes of research or private study, or criticism, or review, as permitted under the Copyright, Designs & Patents Act, 1988, this publication may only be reproduced, stored or transmitted in any form or by any means with the prior permission, in writing, of the Controller of Her Majesty's Stationery Office (HMSO). Enquiries concerning reproduction outside those terms should be sent to HMSO at the undermentioned address: The Copyright Unit, The Stationery Office, St Clements House, 2-16 Colegate, NORWICH, NR3 1BQ

3 Lead Author Alan R. McBride (PACSnet Senior Evaluator) Contributing Authors Dewinder S. Bhachu (Head of PACSnet) Jonathan E. Turner (PACSnet Principal Evaluator) Annelise Wilken (PACSnet Principal Evaluator) Centre Director Dr. Alan Britten PACSnet BENCE-JONES OFFICES (PERIMETER RD.) ST. GEORGES S HOSPITAL LONDON SW17 0QT TELE: FAX: WEBSITE:

4 Summary This report gives a brief introduction to Storage Area Networks (SAN) within the context of a PACS solution. This can be a highly technical subject however this report aims to cover the basic concepts of a SAN and compares this to Network Attached Storage (NAS). The shared storage model is a concept devised by the Storage Network Industry Association (SNIA), see section 2, to aid in the design of networks by taking the complexity and abstracting it to the basic components. The report covers the main technical areas: i) SAN Equipment This will cover such areas such as Host Bus Adapters (HBA), the different types of transfer media by stating the benefits of each type; the multiple types of cable and the different connectors associated with each type. Finally the section ends with a discussion on fibre channel switches. ii) iii) iv) Major benefits of a SAN are outlined, to cover areas such as storage consolidation and methods are described on the use of backup and restore across a SAN. Outlined are the three types of SAN Topology Point-to-Point, Arbitrated Loop and Switched Fabric. The report examines the concept used to route the data through the switched topology. Fibre Channel Features: This details the protocol used to transfer the data across the SAN. It examines why this method is a faster and more reliable than Ethernet and looks at specific details of fibre channel protocol. v) The use of a SAN in a PACS solution is examined through two case studies; University Hospital Leicester (UHL) and Bristol Royal Infirmary (BRI). This discusses the structure of the SAN environment at each site, the benefits and the pros and cons of each site encountered. 4 PACSnet PACS Education: Introduction to Storage Area Networks

5 Acknowledgements PACSnet would like to thank Mr Bill Remmer (University Hospital Leicester, UHL), Prof N. Shastry (Bristol Royal Infirmary, BRI) for their kind assistance discussing their respective implementations of SAN to PACS. PACSnet would also like to thank Jon Hall of Insignia Medical Systems and Stewart Buchanan of Saggitta Performance Systems Ltd. for their invaluable advice on Storage Area Networks. Finally PACSnet would like to thank the Storage Networking Industry Association (SNIA) for the use of their images and their advice on the Shared Storage Model PACSnet PACS Education: Introduction to Storage Area Networks 5

6 6 PACSnet PACS Education: Introduction to Storage Area Networks

7 Contents Page 1. Introduction to Storage Area Networks (SAN) 1.1 Introduction Storage Area Networks (SAN) Fibre Channel Protocol Fabric Fibre Channel Protocol Media SAN vs Other Storage Models Network Attached Storage Advantages and Disadvantages of NAS implementation Shared Storage Model 2.1 Shared Storage Model SAN vs NAS SAN Equipment 3.1 Host Bus Adapter (HBA) Fibre Channel Switch Fibre Channel Protocol Transfer Media Transmission distances Connectors Gigabit Interface Converter (GBIC) Benefits of Fibre Channel Protocol and Storage Area Networks 4.1 Fibre Channel Protocol Benefits SAN Benefits SAN Topology 5.1 Point-to-Point Arbitrated Loop Switched Fabric Fabric Shortest Path First (FSPF) Summary of FCP characteristics compared with SCSI Fibre Channel Protocol Features 6.1 Fibre Channel Protocol Layer Architecture Port Types Data Encoding Ordered Sets Fibre Channel Protocol Frame Format Class of Service Flow Control 42 PACSnet PACS Education: Introduction to Storage Area Networks 7

8 Contents (contd.) Page 6.8 Fibre Channel Protocol Addressing Scheme Application of SAN to PACS: Case Studies 7.1 University Hospital Leicester (UHL) Bristol Royal Infirmary (BRI) Conclusion 47 8 PACSnet PACS Education: Introduction to Storage Area Networks

9 1. Introduction to Storage Area Networks 1.1 Introduction Introduction to Storage Area Networks The implementation of a PACS, either radiology or hospital wide has a significant impact on the hospital Local Area Network (LAN). The size of the image files can cause a significant deterioration in the transfer of data, which has a subsequent effect on the operation or workflow of the radiology department. The concept of a Storage Area Network (SAN) is not new and has indeed been implemented in the commercial world for many years, however the introduction of SAN in the healthcare environment is a recent event. These were initially implemented by pharmaceutical companies and research establishments involved in gene sequencing. With the implementation of PACS, the data required for the storage of medical images is increasing rapidly and trusts are now investing in SAN to reduce the load on the LAN and increase the performance and reliability of PACS operation. The integration of PACS with the information systems of a trust (HIS & RIS) allows the large image data files to be transferred on a dedicated network; the result is the freeing up the hospital wide LAN bandwidth for the transmission of patient related information. The SAN allows the hospital to implement the PACS whilst retaining its legacy systems reducing the cost of implementation. The ease of implementation and the seamless scalability of a SAN makes this technology an excellent solution to the storage and backup for a PACS. A Storage Area Network (SAN) is a network that sits behind the local area network (LAN) connecting the servers into centralised disk storage and backup libraries, fig 1. This allows for a scalable storage solution where each server has access to the storage pool creating an optimal and cost effective storage environment. The SAN uses a different protocol to transfer the data, whereas a LAN uses Transfer Control Protocol/Internet Protocol (TCP/IP), a SAN is implemented using Fibre Channel Protocol (FCP). The report will discuss the theory behind FCP and storage area networks and will go into detail on the hardware used to implement such a storage solution. The Storage Network Industry Association (SNIA) have created and outlined a shared storage model. The report will cover the concepts behind this model and its significance in outlining and design of network topologies. The report will conclude with examples of the application of SAN to Picture Archive and Communication Systems (PACS) which will follow case studies. 1.2 Storage Area Networks (SAN) Storage Area Networks (SAN) is a network structure for storage which is one of the methods to replace the traditional direct attached storage (DAS) model. This model has a server connected to the disk array or tape through a Small Computer Systems Interface (SCSI) architecture. This model had problems whereby problems existed for PACSnet PACS Education: Introduction to Storage Area Networks 9

10 Introduction to Storage Area Networks access to data if either the server or the SCSI connection failed, hence in a mission critical environment such as a PACS where data is required continuously; this possible failure to gain access to data is unacceptable. Workstation LAN PACS Server SAN Web Server Tape Library FC Disk Array Fig 1: Storage Area Network SAN, fig 1, is a networked environment based on FCP. This is entirely separate from the LAN which usually relies on the TCP/IP protocol. The networked environment and the use of FCP provide the SAN with many benefits, making it suitable for storing large amounts of data in a critical environment: i. Minimal Error Rates: The SAN environment allows for large data transfer with very little data corruption hence there is less retransmitted data, speeding up data flow. ii. iii. iv. Flow Control: This involves the signalling between the receiving and transmitting nodes controlling the data flow using a buffering method. This is discussed in detail in section 6.7 Full duplex transmissions: This separates the send and receive signal between communicating nodes In-Order Delivery: FCP sequentially transfers data across a SAN. Data is received by a node in the order it was sent, hence at the receiving node less time is spent reconstructing the data from the information contained in the frames, section 6.5, and as a result there is faster transfer of data from node to node. 10 PACSnet PACS Education: Introduction to Storage Area Networks

11 Introduction to Storage Area Networks 1.3 Fibre Channel Protocol Fabric A fibre channel switch is a sophisticated piece of hardware that allows data to flow across a network, which is discussed in detail in section 3.2. There are two main topologies used to create a SAN, these being ring topology (similar to the token ring in Ethernet) and the switched topology however they are referred to as either as an arbitrated loop or switched fabric. The use of the term fabric is part of the SAN vocabulary and is often used with little or no explanation. A fibre channel fabric is a single or multiple number of switches which form a network known as a switched or fabric topology, discussed in section 5.3. This is a highly manageable and scalable topology which provides a high performance connection to nodes. 1.4 Fibre Channel Protocol Media A point often overlooked when discussing SAN and Fibre channel technology is the media by which the data travels. A SAN can be implemented using copper or optical fibre; the majority use optical fibre to gain the full benefits of SAN implementation which are discussed below. As Ethernet uses TCP/IP, SAN uses FCP, however confusion often occurs as it is assumed that FCP and fibre optical cable are automatically linked and can only function with optical media. This is not the case as FCP can be used on copper cabling. 1.5 SAN vs Other Storage Models Direct Attached Storage (DAS) Direct-attached storage (DAS) is the most common method for saving and retrieving data today. DAS is defined as one or more spinning or streaming devices that are connected to a single server via a physical cable, fig 2. DAS was initially implemented as a series of servers and storage devices independent of each other. With the increase in computer networks, this allowed for the connection of heterogeneous systems. Over time pitfalls were revealed in the DAS model, with single-points of failure within the storage causing the main problems: server downtime is the critical component in the DAS model, and systems can take hours to weeks to fix with subsequent loss of data accessibility over this period. PACSnet PACS Education: Introduction to Storage Area Networks 11

12 Introduction to Storage Area Networks HIS Server PACS Server LAN Web Server Fig 2: Direct Attached Storage (DAS) 1.6 Network Attached Storage (NAS) Network Attached Storage (NAS) filers or appliances are defined as specialized file servers that reside on a local area network (LAN), fig 3. NAS appliances operate in a client/server environment and provide a file sharing capability over the network. The NAS appliances allow access and share the data in a heterogeneous environment. NAS appliances support file sharing using several protocols, including Common Internet File System (CIFS), Network File System (NFS) or AppleShare (EtherTalk). 1.7 Advantages and Disadvantages of NAS implementation NAS offers a simplification of implementation and management, higher availability, greater flexibility in data and lower cost, when compared to DAS. The benefits to be gained by the implementation of a NAS can be significant. NAS solves cross-platform and direct user access problems preventing data being exclusively owned by one server. The majority of NAS solutions still continue to use SCSI bus architecture for disk arrays, however others use Fibre channel arrays which are used in the environments that have large disk populations and where redundancy is critical to maintain system operation. Some of the advantages are: i. File sharing by hosts running different operating systems ii. iii. Cost efficiencies of using the existing network infrastructure Increased file access performance 12 PACSnet PACS Education: Introduction to Storage Area Networks

13 Introduction to Storage Area Networks iv. Support for large amounts of consolidated storage v. Simplification of configuration complexity vi. vii. viii. Faster installation Ease of integration into the existing infrastructure Improved automation of backup and recovery Although NAS is very cost effective and efficient, to consolidate storage and share data from heterogeneous systems on a network is not without problems and disadvantages: i. Latency caused by the added overhead and increased time to process the NFS and CIFS protocols can reach unacceptable levels. ii. iii. iv. The speed at which data is stored and retrieved may be too slow to satisfy application requirements. The bandwidth has to be carefully managed because increased data transfers can cause network bottlenecks to occur. The inability of NAS appliances to scale capacity without degrading performance v. Speed of data capture and retrieval from hard disk media. vi. Lack of direct wide-area connectivity options. Tape Library NAS Disk Array PACS Server Web Server LAN Workstation Workstation Fig 3: Networked Attached Storage PACSnet PACS Education: Introduction to Storage Area Networks 13

14 Shared Storage Model 2. Shared Storage Model 2.1 Shared Storage Model The Storage Network Industry Association (SNIA) has created an architectural model which allows for complex technical systems to be abstracted to their basic components, fig 4. The model is separated into the applications layer and the storage domain. The storage domain is separated into two layers: the file/record system, which organizes the data, and the block subsystem, which includes block aggregation for organising the data in bulk. The physical systems are included in the block system where the data is stored. The services subsystem provides a series of management and organizational functions for the system. The model demonstrates the similarities of both the NAS and SAN infrastructures occupying the same space in the storage domain. Copyright 2001 Storage Networking Industry Association. Used by kind permission. Fig 4: Shared Storage Model The difference between file and block access is a practical one. The file - however it is accessed - exists on the disk, written in the standard block addressing method. In a file system access, the call for the file by its file access name requires low-level processing of the blocks. In storage networking terms, the access to a file can come from two routes: file transfer or block transfer. The block access of files is achieved at the low-level areas and provides high performance access to data; however the file access requires processing of blocks at an upper level prior to the file transmission across a network. Although file transfer has a poorer performance due to the reconstruction of the blocks it does provide a higher level of functionality to the user allowing access from heterogeneous operating systems. 14 PACSnet PACS Education: Introduction to Storage Area Networks

15 Shared Storage Model In fig 5, the SNIA shared storage model describes the functionality of a block orientated SAN. The SAN resides between the servers and the disk systems using FCP or gigabit Ethernet to connect between the hosts and the storage systems and offers high bandwidth, long distances and more advanced features such as resource sharing, consolidation and high availability, each of which will be discussed in detail in this document. Copyright 2001 Storage Networking Industry Association. Used by kind permission. Fig 5: Shared Storage Model (SAN Structures) In fig 6, the model architecture describes the NAS structure. This links from the file/record subsystem across the block storage to the physical data system. NAS serves up constructed files whereas the SAN offers blocks of data. NAS systems are optimized for heterogeneous file access. The access to the application in a NAS is through NFS or CIFS over IP which are available on a number of operating systems allowing for the seamless integration of differing operating systems. The major disadvantage of the NAS can be seen in fig 6 where the application s access to the NAS is over LAN, this being the major bottleneck to the network in fast Ethernet but less of a problem in gigabit Ethernet. PACSnet PACS Education: Introduction to Storage Area Networks 15

16 Shared Storage Model Copyright 2001 Storage Networking Industry Association Fig 6: Shared Storage Model (NAS Structures) The NAS and SAN models are not mutually exclusive. It is entirely possible to have a requirement for both models; fig 7, below. The NAS head is organised to use the SAN network using Fibre channel, this gives the NAS the benefits of the SAN, i.e. high bandwidth, flexible networking and ability to grow storage capacity during system operation. Copyright 2001 Storage Networking Industry Association Fig 7: Shared Storage Model (SAN & NAS Structures) 16 PACSnet PACS Education: Introduction to Storage Area Networks

17 Shared Storage Model 2.2 SAN vs. NAS There are several preconceptions surrounding both the SAN and NAS models, with the main myth being that they are not compatible. The discussions on the shared storage model show that they are not mutually exclusive in their implementation and although the SAN and NAS perform separate functional tasks they can be partnered in a complimentary manner. The SNIA shared storage model makes a definite point of not defining SAN or NAS models but instead uses the term storage network (SN). A SN is a network dedicated to storage traffic whatever the hardware. The SAN and NAS models are described in separate areas of the shared storage model, therefore the question should not be Should SAN or NAS be implemented as part of the storage policy? but How can the design and implementation of the storage network provide the functionality, the storage capacity and the necessary expansion for the environment? PACSnet PACS Education: Introduction to Storage Area Networks 17

18 SAN Equipment 3. SAN Equipment 3.1 Host Bus Adapter (HBA) A Host Bus Adapter (HBA) is installed into a server; this provides a gateway to the SAN by connecting the server to the storage. The HBA is different from a network interface card (NIC) which relies on server CPU to perform the majority of processing tasks on the data. The HBA, however, has several features which allow it to control and process the data which frees up the server CPU for the operating system and applications software. The HBA architecture as part of the unit has an on-board processor, a protocol controller and buffer memory to control the data flow. This allows the HBA to organize and manage the I/O transactions with no input from the server CPU. The HBA provides a critical link between the SAN and the operating system and application software. In this role, the HBA enables a range of high-availability and storage management capabilities, including load balancing, fail-over, SAN administration and storage management. 3.2 Fibre Channel Switch To enable communication across the SAN, a network of fibre channel switches i.e. a fabric is created, which allows the nodes to communicate. The switch unlike the hub, which broadcasts to all ports, connects two nodes through a routing scheme, (discussed in section 5.4). This one to one connection allows the maximum bandwidth of 100 Mbps to be achieved. In a hub when more nodes are added the bandwidth is shared, as all data is transferred to all the ports; however because of the switch configuration as nodes are added the bandwidth remains at 100 Mbps. There are two switching mechanisms employed on fabrics. One mechanism is store and forward, where the entire frame, (the information sent across the network) is buffered by the switch. However the most common algorithm used is called cutthrough which is also used on Ethernet switches to route the data. When a frame enters the fabric, the cut-through mechanism reads the destination ID. The destination ID is the 24 bit port address of the target node, and the internal routing mechanism of the switch directs the frame to the correct port. The target port address is in the first four bytes of the frame header, hence the frame destination is read immediately; this reduces the time to make the routing decision, thus increasing the performance. Section 6.5 discusses the frame construction in detail. In the construction of a switch there are several major components: Switch Ports: Each switch has between 8 to 128 ports with the latter being known as directors. The ports on a switch can have different functions and are defined as F-ports, FL-ports, E-ports or G-ports; a further discussion on port functionality is in section PACSnet PACS Education: Introduction to Storage Area Networks

19 SAN Equipment Fabric Controller: This is a function which initialises, configures and sends information to other switches in the fabric Internal switching function: This performs the path selection based on the information retrieved from the inter-switch links, a database is created and this information is passed to the route selector. Switch Communications: Routing Path selector: There is an inter-switch link (ISL) through the E_port which stores the information on the status of all the switches in the fabric This controls the route the frame travels through the fabric, and the information to construct this table is taken from the communication with the internal switching function. The routing of the frame through the fabric is achieved using the algorithm Fabric Shortest Path First (FSPF). A discussion of FSPF can be found in section Fibre Channel Protocol Transfer Media FCP does not operate exclusively on optical fibre cable, but is a protocol used to send data over networks and can be used over copper or fibre cable. The two technologies are both able to cope with gigabit transmission speeds, however the optical fibre can be used over longer distances (up to 10 km), whereas the copper cabling can only cope with a maximum distance of 25m. Optical cables on a SAN are connected with industry-standard SC connectors. On the optical side there are two modes of operation. The difference between them is in the laser wavelength. The two modes are: Short Wave mode - SW Long Wave mode LW The distances that can be achieved, as described in designing a Storage Area Network are shown in fig 8. Diameter (micron) Cladding (micron) Mode Laser type Distance Single mode Longwave =< 10 km Multi mode Shortwave <= 500 m Multi mode Shortwave <= 175 m Fig 8: FCP distances at 100 MB/second PACSnet PACS Education: Introduction to Storage Area Networks 19

20 SAN Equipment 3.4 Transmission distances The transmission of a signal through any medium will gradually degrade. This is due to the noise inherent in the physical properties of the medium and external effects on the medium. Copper media, due to its structure degrades the signal over shorter distances than fibre, and is also affected by external magnetic fields. Optical fibre is not affected by any external sources in a normal operating environment. However there are two main features shown in fig 8 that affect the distance the signal can be transmitted: i. diameter of the fibre ii. Laser frequency. The high frequency short wavelengths travel shorter distances than the low frequency long wavelength lasers. Another major influence on the distance the signal travels is the diameter of the fibre: losses due to internal reflections are greater in the larger diameter fibres causing the signal to degrade faster. 3.5 Connectors The different type of cabling used to operate the SAN require different connectors. The connector for copper cables, which are used for short distances, is the 9-pin DB9 connector. This is usually a 9 pin connector this can be seen in fig 9 (b) the female connector, however only four of the pins are required for fibre channel, fig 9 (a). DB9 Male DB9 Female Fig 9 (a): DB9 Male connector Fig 9 (b): DB9 Female connector Fibre optic cables use the Stick and Click (SC) connector, fig 10. The name comes from the insertion of the connector into the port and it clicking into place. These are known as SC connectors. This is a side by side connector: one side for transmitting and the other for receiving. The transmitting side connects to the receive at the other end of the cable and vice versa. 20 PACSnet PACS Education: Introduction to Storage Area Networks

21 SAN Equipment Fig 10: SC Connector Another type of fibre optic connector is the screw and turn (ST) connector, fig 11. This connector is inserted into a port and when twisted is locked into place. Fig 11: ST Connector As two types of cable can be used to operate a SAN, a device exits which can covert an optical interface into a copper signal and vice versa. This is known as a Media Interface Adapter (MIA), fig 12. MIAs are most commonly attached to host bus adaptors; however they can also be used with switches and hubs. This is used when a hub or switch only supports the connection of one type of media, MIAs can be used to convert the signal to the appropriate media type (copper or optical). PACSnet PACS Education: Introduction to Storage Area Networks 21

22 SAN Equipment Fig 12: Media Interface Connector (MIA) 3.6 Gigabit Interface Converter (GBIC) In a network, the active component for optical/electrical components that transmit and receive signals are called transceivers with the most common type of transceiver being the gigabit interface converter (GBIC), fig 13. Fig 13: Gigabit Interface Converter (GBIC) A GBIC is a data communication transceiver which converts data coming in from the device on one side to an optical signal on the other side. The devices usually supply a differential serial data signal as input and the GBIC converts this signal into an optical or electrical signal. 22 PACSnet PACS Education: Introduction to Storage Area Networks

23 SAN Equipment If the GBIC is receiving an optical signal, this signal is then converted to an electrical signal which is delivered to the host device as a differential serial data signal. The data format on the device side is standardized to 8b/10b encoding, discussed in section 6.3. It is important to note that because of different cabling media used in FCP, there are different GBICs for each cable type, and for the different lasers wavelengths used in optical fibre cabling. PACSnet PACS Education: Introduction to Storage Area Networks 23

24 Benefits of Fibre Channel Protocol & Storage Area Networks 4. Benefits of Fibre Channel Protocol & Storage Area Networks 4.1 Fibre Channel Protocol Benefits The use of FCP provides a highly reliable and high performance networking environment specifically designed for the requirements of storage. Fibre channel was specifically designed for server to server and server to storage device; this provides the storage devices with a high performance and controlled flow of reliable data. FCP utilises the network to its full capabilities which can give up to 2 Gbit/sec (200 MB) on a full-duplex dedicated connection. One of the strengths of FCP is that it is not just another protocol but will adapt to a number of protocols at the upper layer, see section 6.1. FCP allows protocols such as SCSI and IP to be transferred across as FCP frames. In other network protocols, such as TCP/IP, the transmission data arrives out of sequence, causing pressure on the server CPU to rearrange the packets to reconstruct the data. FCP provides an advanced methodology to control the flow of frames across the network which guarantees in-order delivery. Therefore the HBA has few problems rebuilding the data blocks from the FCP frames. A detailed discussion on flow control is in section 6.7. The adoption of FCP provides the user with the ability to operate in a heterogeneous environment. The highest performance and throughput of data is achieved by the optimization of data movement. The main problem often encountered when adopting new technology is the use of legacy equipment providing access to information and applications. This is particularly important in a hospital environment and FCP has the ability to interface to these systems. The support for IP and SCSI protocols at the upper levels reduces the burden on the user when implementing a SAN. 4.2 SAN Benefits Prior to the implementation of network storage any workplace would operate a series of stand alone servers attached to the network each with their own attached storage. In the poor performing cases up to 50% of the storage could be unused, and there is significant cost incurred on the administration of each of the servers and storage devices, fig PACSnet PACS Education: Introduction to Storage Area Networks

25 Benefits of Fibre Channel Protocol & Storage Area Networks 50% Used Space HIS Server 50% Used Space PACS Server 50% Used Space LAN Web Server Fig 14: Utilisation of storage in direct attached storage In a SAN environment the storage can be consolidated, fig 15. The storage capability of the RAID array is shared between the servers in a SAN environment. In fig 15 the data from the three DAS servers can be consolidated onto two devices leaving the third free for expansion hence the actual storage utilization can rise to 80%. This consolidation can be carried out in a heterogeneous environment; the amount of storage capacity can be reduced or the number of physical devices can be retained and with the growth in storage, the spare device can be assigned, partly or wholly, to individual servers. The overall management of the storage can therefore be simplified. The administration of multiple systems is reduced by the implementation of a SAN, and the storage is now managed as a single resource. A benefit of consolidation is the ability to virtualize the storage. In a SAN, storage virtualization is achieved by mapping all the storage devices into one logical storage volume. In heterogeneous environment each of the servers or applications can be assigned a specific portion of storage, the storage can be easily managed and allocation of new storage capacity to any server is a simple task to carry out. PACSnet PACS Education: Introduction to Storage Area Networks 25

26 Benefits of Fibre Channel Protocol & Storage Area Networks Fig 15: Utilisation of storage in SAN topology With any storage system it is necessary to preserve a copy of all the data and this is achieved with a backup process. The backup is run across the local area network (LAN), fig 16, and depending on the time of day and the amount of data to backup this can cause problems with some applications and information not being available during any backup or restore times, this in some environments can lead to a lax backup process. Backup Server Tape Library PACS Server LAN Web Server Fig 16: DAS Backup When the system is setup to carry out SAN backup the process is carried out quicker. The competition that the DAS or NAS backup processes have from the traffic on the 26 PACSnet PACS Education: Introduction to Storage Area Networks

27 Benefits of Fibre Channel Protocol & Storage Area Networks LAN is avoided, as a resulting reduction in backup time allows the system to have a greater time in service also reducing unscheduled down time that occurs during the restoration of corrupted data. In a SAN topology the backup process is carried out faster with little or no effect on the LAN or application servers. There are two strategies used to implement the back and restore process: i) LAN-free backup and restore ii) Server-free backup and restore Fig 16 above, illustrates the problems associated with DAS backup. The traffic on the LAN is in competition with the backup process, this can lead to problems if the data to backup increases to a point where the allocated time for backup cannot complete the task. In LAN-free backup and restore, a SAN attached disk and tape library can solve this problem by allowing the server to send the data across the SAN from the disk array to the tape library, fig 17 below. The backup is faster, more scalable and more reliable obtaining improved utilization of all the resources. The improvement in backup performance can be dramatic with the only effect being a reduction in application server performance as this requires processing time to organise the backup. Tape Library FC Disk FC Switch PACS Server WEB Server LAN Fig 17: LAN-free Backup and recovery The Server-free backup and recovery model, fig 18, involves the transfer of data from the disk to the tape library without using the host servers. Backups can be performed at any time provided that a FCP backup device is fitted to buffer the data from disk to tape. PACSnet PACS Education: Introduction to Storage Area Networks 27

28 Benefits of Fibre Channel Protocol & Storage Area Networks The backup process is enabled by technology called third-party copy which has two protocols, both allowing the backup to take place without affecting the application servers. SAN devices such as host systems, storage devices and switches can be used to implement third-party copy. The most significant benefit is the increase in operating efficiency of the application server. Storage Router Data Flow Command FC Disk FC Switch Tape Library PACS Server WEB Server LAN Fig 18: Server-free Backup and recovery 28 PACSnet PACS Education: Introduction to Storage Area Networks

29 SAN Topology 5. SAN Topology 5.1 Point-to-Point This is a simple topology that allows bi-directional communication between two nodes, fig 19. This topology is very similar with SCSI direct attached storage (DAS) model, however the connection between the server and the disk array is faster and the distance supported by the technology greater than the SCSI model. It is clear that a point-to-point topology has its limitations, the main limitation arising from the inability to add new devices to an existing point to point topology. 100 MB/s 100 MB/s Full Duplex 200 MB/s (100 x2) Disk Server Fig 19: Point to Point Topology 5.2 Arbitrated Loop The arbitrated loop, fig 20, is a ring topology, which was initially popular due to its flexibility and its low cost per port. In an Arbitrated Loop configuration, the transmitter of each node is connected to the receiver of the next node. In order to send data from one node to another, devices must arbitrate for access to the loop. The bandwidth between the nodes attached to the loop is shared with the initiating device arbitrating for control of the loop. Once the device wins arbitration, it then opens a communication session with the target and sends the data. Only one connection can be established at a time. When the data transfer is completed, the initiator closes the session and releases control of the loop, allowing other devices to arbitrate for the loop. Currently, the maximum bandwidth is 100 MB/sec. PACSnet PACS Education: Introduction to Storage Area Networks 29

30 SAN Topology NL-Port NL-Port NL-Port NL-Port NL-Port NL-Port NL-Port NL-Port Fig 20: Arbitrated Loop Topology 5.3 Switched Fabric A switched fabric, fig 21, is an extensive storage network in which large numbers of servers and storage systems are connected using fibre channel switches. Switches can be cascaded and combined with loops to create highly interwoven networks known as fabrics. Fortunately, these complex solutions can be kept under control by software that takes advantage of SAN management capabilities built directly into the fabric. Tape Library FC Disk FC Switch PACS Server WEB Server LAN Fig 21: Switched Fabric Topology 30 PACSnet PACS Education: Introduction to Storage Area Networks

31 SAN Topology A fabric topology is the most efficient and highest performing fibre channel topology. A fabric is constructed from one or more fibre channel switches connected by a series of ports, which all perform separate functions. In a fabric there are two main types of ports: F_ports which are connected to nodes and E_ports which are connected to other switches forming the fabric. A discussion on ports is found in section 6.2. The switched fabric topology supports a 24-bit addressing scheme, fig 22. The address is separated into three 8 bit fields, one for domain, one for area, and one for port, in order of significance where each fibre channel switch is a fabric domain. Fig 22: Switched Fabric addressing scheme A switched fabric is a highly scalable topology which is well suited for high performance applications such as PACS. The performance is achievable due to the high bandwidth; the switch is fully interconnected hence there is no competition for the resources. Scalability is achieved by the ability to add switches which increases the size of the fabric; with a subsequent increase in the number of ports. 5.4 Fabric Shortest Path First (FSPF) In a switched topology the route the frame travels from the transmitting node to the receiving node is determined by the fabric using an intelligent path selection and routing standard called Fibre Shortest Path First (FPSF), fig 23. This is defined as part of the Fibre Channel Protocol. This is an important process and it is crucial that this is taken into account in the design of any fabric as this will minimize the bottlenecks and maximize the bandwidth to gain full functionality from any implemented SAN. The standard does not state the algorithm to be used to calculate the best path; therefore several approaches have been examined to determine the most efficient route through any fabric. There are two classical techniques: vector weighted path and metric weighted path. The vector weighted path algorithm calculates the number of interconnections between each of the points; however this does not take into account the bandwidth of each of the links. The metric weighted algorithm calculates the PACSnet PACS Education: Introduction to Storage Area Networks 31

32 SAN Topology cost to travel down each of the links between two points in the fabric, where in fibre channel the cost is calculated from the speed of the link; the metric weighted algorithm is the most commonly used in network technology. Fig 23: Fabric Shortest Path First (FSPF) All the switches in the fabric determine its topology from the adjacent switch through the ISL connection. The database which defines the fabric topology is replicated throughout the Fabric. Each switch then calculates the path costs using the link state database, this information is then used to create the routing table for each switch. 32 PACSnet PACS Education: Introduction to Storage Area Networks

33 Summary of FCP characteristics compared with SCSI 5.5 Summary of FCP characteristics compared with SCSI SCSI -command set -Physical transport Fibre Channel 1 st Generation Arbritrated Loop Fibre Channel 2 nd Generation Switched Fabric Node Connectivity (2 8 ) (FC Hub) In theory 16 Million (2 24 ) (FC Switch) Distance Limit 25 meters 10 kilometers 10 kilometers (Plus extensions) Maximum Bandwidth Multi-protocol Support Management Support 160 MB/sec (half duplex with shared bandwidth) 100 MB/sec (Full duplex/shared bandwidth) No Yes Yes No (no hot Plugging) Yes Fault Isolation No Minimum Yes 200 MB/sec (Full duplex with dedicated bandwidth) Advanced (telnet, SES, SNMP, GUI, API) PACSnet PACS Education: Introduction to Storage Area Networks 33

34 Fibre Channel Protocol Features 6. Fibre Channel Protocol Features 6.1 Fibre Channel Protocol Layer Architecture As with the architecture of the OSI model, the FCP is also a layered architecture as shown in fig 24. FCP has layer FC-0 to FC-4 and each carries out the following functions to move data across the network. FC-0: Physical Interface This is the lowest link and similar to the OSI model where the lowest level specifies the physical link to carry the data. The protocol at this level defines the physical media this includes the lengths and the data rates. The protocol allows existing cable media, such as copper or fibre optic, and a number of other different technologies to be used in order to meet the system requirements. FC-1: Byte Encoding This layer describes the encoding and decoding scheme used and transmits the bit stream at gigabit speeds. The coding scheme also provides a mechanism for the detection of transmission and reception errors, the coding scheme is discussed in detail in section 6.3. This layer also provides the command structure necessary for accessing the physical media. FC-2: Data Delivery This layer contains all the controls and processes to send the data across the network. The information required to construct the frames ready to send across the network. The control of when to send the data across the network is also organized from this layer, and is discussed in section 6.7. The destination port of the frame is also organized from this layer. The Class of Service defines the different implementations that can be selected depending on the application this is set-up at this layer and discussed in section 6.6. FC-3: Common Services This layer is currently not fully part of the standard, and at present is not available to SAN. At present FC-2 allows the sending of data to single ports, however this layer will implement features such as striping (to transmit one data unit across multiple links) and multicast (to transmit a single transmission to multiple destinations) and hunt group (mapping multiple ports to a single node). FC-4: Upper-Layer Protocol Mapping This layer provides mapping of FCP capabilities to pre-existing protocols, such as IP or SCSI. An example is the replacement of upper layer protocols error recovery by FCP. 34 PACSnet PACS Education: Introduction to Storage Area Networks

35 Fibre Channel Protocol Features Fig 24: OSI and fibre channel models 6.2 Port Types At the beginning of FCP technology development a great deal of emphasis was placed on the technology; the protocols covering the physical hardware were built into the FC-2 layer of the FCP. As part of this layer the definition and characteristics of the various port types are defined. The basic source and destination of communications under FCP would be a computer, a controller for a disk drive or array of disk drives, a bridge, a terminal, or any other equipment engaged in communications. These sources and destinations of transmitted data are called nodes. Each node has the ability to receive and transmit data under the FCP. Each of the ports contains a transmitter and receiver and is given a unique name or identifier. There are several port types, fig 25, defined in the FCP, and the function of each port determines how it initiates and responds to network transmissions. The common port definitions used in FCP are: N-port: This is found in systems and devices which connect to fabrics NL-port: This is found in systems and devices which connect to fabrics in loops F-port: Switch ports to connect to N-ports FL-ports: Switch ports to connect to NL-ports PACSnet PACS Education: Introduction to Storage Area Networks 35

36 Fibre Channel Protocol Features E-ports: Expansion ports, inter-switch connection G-ports: Generic ports on a fabric that can be F, FL or E ports Fig 25: Fibre Channel Protocol Port Types N-ports: FCP was originally devised to have communications between N-ports and F-ports. N-ports are the logical network function implemented in HBA and storage subsystem controllers that access the Fibre channel network. N-ports act as storage initiators and targets communicating over the Fibre Channel SAN, having the responsibility to manage the exchange of frames. N-port is strictly a wiring element; the HBA will perform the storing or filing functions, with the N-port performing a wiring operation. F-ports: An F-port is provided in fibre channel switches to provide a point for the connectivity of N-ports. Requests by one N-port to connect to another are serviced by the switch when it makes a connection between corresponding F-ports in the switch. N-ports and F-ports are constantly communicating with each other; when there is no transmission of frames between the N-port and F-port an IDLE sequence is sent from the N-port to the F-port creating a heartbeat which allows any problems in the link to be identified immediately. L-ports: Nodes in a loop share a common cabling structure, where the L-port is used. This was designed to control the communication on loops. Fibre channel loops are ring networks similar to the token ring, hence the L-port has some networking functionality built-in. 36 PACSnet PACS Education: Introduction to Storage Area Networks

37 Fibre Channel Protocol Features FL-ports: To enable a loop to communicate with a fabric through N-nodes; two new ports were devised: the FL-port and the NL-port. FL-ports are found on fabrics and allow the fabric to act as a node onto the loop. A loop can only address one FL-port, therefore two active switches cannot be connected to the loop at the same time, however more than one switch can be connected, but only when acting as a redundant system. The switch would stay in standby mode until the active switch lost the ability to connect to the loop. NL-ports: These ports have both N and L functionality, therefore they can manage the networking requirements of a loop but also have the capability of acting as link to a fabric. E-port: These are ports that are used to communicate with other switches to create a fabric of multiple switches. G-port: The G-port is a generic port which can auto detect the port on the other end of the connection and on that initial communication will automatically configure its function, i.e. if an E-port is detected the G-port will configure to an E-port, likewise the G-port will configure to a F-port if the port on the other end is a N-port. 6.3 Data Encoding Fibre channel uses an advanced technique for encoding the data prior to transmission this is known as 8b/10b encoding algorithm, fig 26. Prior to being sent each 8 bit byte is transformed into two 10 bit characters. No character can contain more than 6 bits of the same type and no more than 4 of the same bit type in a sequence. The number of 1 s or 0 s in a 10 bit character determines if it has a positive, negative or neutral disparity. Running Disparity Fig 26: 8b/10b Encoding Algorithm When the 10 bit characters are created one has a positive and the other a negative disparity. When the 10 bit characters are transmitted it is essential to the coding scheme that the above rules apply, hence the bits are in balance, therefore if a negative disparity character has been sent the next character will have a positive PACSnet PACS Education: Introduction to Storage Area Networks 37

38 Fibre Channel Protocol Features disparity, hence keeping the correct balance of bits. This is used for error detection in FCP along with verifying received characters during decoding, and the computation of a Cyclic Redundancy Code (CRC). In the 10 bit character sets there are also a series of control characters that are used to define the beginning and the end of frames and to send control commands known as ordered sets, discussed below. There is a special command character called K28.5 which again has a positive and negative disparity, fig 27, this is a special character by having 2 bits of one type followed by 5 of the opposite type. Positive Negative Fig 27: 10 bit characters produced for command character K28.5 The 8b/10b character is described in the format Kx.x or Dx.x, where the K signifies command character however a data character is defined by the letter D. The original 8 bit data byte is used to calculate the character descriptor, an example data byte is This is calculated into the DX.X format from the first 3 bits 110 giving the value 6, and the final 5 bits giving the value 10, therefore this is shown as D Ordered Sets As in all protocols the transfer of data across the network requires a series of command sequences which are referred to as ordered sets or words. The sequence of bytes indicates an action or an event that is carried out during the transport of the data. The encoding scheme as described above encodes the data from 8 bit to 10 bit bytes. An ordered set contains 4 bytes or 40 bits with the first byte of each of each set being the command character K28.5 an example of an ordered set is K28.5 D21.4 D21.6 D21.6 which is an end of frame (EOF) ordered set to inform that the transmission of the frame is complete. In fibre channel protocol there are 11 types of start of frame (SOF) and eight end of frame (EOF) ordered sets, each of the ordered sets refers to the actions to be undertaken prior to or after receiving the frame. There a special set of ordered sets known as primitive signals which indicate actions or events on the transport. The type of commands include the IDLE primitive: K28.5 D21.4 D21.5 D PACSnet PACS Education: Introduction to Storage Area Networks