STORAGE AREA NETWORKING PROTOCOLS AND ARCHITECTURE

Transcription

1 STORAGE AREA NETWORKING PROTOCOLS AND ARCHITECTURE SESSION 1 Morning Schedule 9:00am 10:30am Introduction to Storage Area Networking 10:30am 10:45am 10:50am 12:30pm 12:30pm 1:30pm Storage Terms and Acronyms Storage Networking Devices (Switches, HBAs, Disk) Storage Networking Applications Storage Networking Topologies Intro to Storage Protocols (SCSI, FC, FCIP, iscsi) Break Storage Protocols in-depth Introduction to the Standards SCSI Fibre Channel Lunch 2 Printed in USA.

2 Afternoon Schedule 1:45pm 3:30pm 3:30pm 3:45pm 3:50pm 6:00pm Storage Protocols In-Depth (Cont.) Fibre Channel Services iscsi FCIP ifcp isns and SLP Break Storage Network Troubleshooting Required Tools Required Technical Skill Sets Storage Network Architecture Design Practices FC Network Designs IP SANs SAN Extension Implementation and Management 3 Associated Sessions OPT-1051 Introduction to Storage Topologies and Applications OPT-2051 Fibre Channel Storage Area Network Design OPT-2052 FCIP Design and Implementation OPT-2053 iscsi Design and Implementation OPT-2054 Storage Networking Security OPT-3051 Troubleshooting MDS9000 Fibre Channel SAN OPT-3052 Troubleshooting MDS9000 IP Storage Area SAN OPT-4051 Design and Architecture of Storage Networking Platforms OPT-4052 Case Study: Cisco IT Storage Strategy 4 Printed in USA.

3 Reference Materials Cisco Storage Networking Cisco AVVID Storage Networking Partner Program Cisco Metro Optical Product Information Storage Network Industry Association (SNIA) IETF IP Storage ANSI T11 Fibre Channel 5 INTRODUCTION TO STORAGE AREA NETWORKING 6 Printed in USA.

4 Section Agenda Storage Terms and Acronyms Storage Networking Devices Storage Networking Applications Storage Networking Topologies Introduction to Storage Protocols 7 STORAGE TERMS AND ACRONYMS 8 Printed in USA.

5 Technologies Overview (or Storage in a Nutshell ) Storage Area Network (SAN) Technologies SAN Protocols Databases Servers and Mainframes Backup Apps IP CLOUD Call Home Support Center SAN Applications iscsi RAID & VirtualRAID JBODs and NAS Storage Virtualization Mirroring iscsi Drivers IP CLOUD iscsi TAPE FSPF Enhanced Fibre Channel Embedded Management FC HA Virtual SAN Generic Fibre Channel FCIP SAN FC Switch IP CLOUD FC Switch 9 Introduction to SAN Terminology Block Level I/O File Level I/O SCSI Small Computer Systems Interface FC Fibre Channel RAID Redundant Array of Inexpensive Disks iscsi Internet SCSI FCIP Fibre Channel over TCP/IP ifcp Internet Fibre Channel Protocol isns Internet Storage Name Service 10 Printed in USA.

6 RAID Levels RAID Level Description Min Disks 0 Striping/Concatenation 2 1 Mirror Striping/Concatenation then Mirror Mirror then Striping/Concatenation 4 2 Hamming Code N/A 3 Fix parity with concert I/O N/A 4 Fix parity with Random I/O N/A 5 Stripe with distributed parity with Random I/O 3 without log 4 with log 11 Terminology Direct Attached Storage (DAS) Block level I/O Can be internal or external Typically SCSI or FC Limited scalability High cost due to management 12 Printed in USA.

7 Terminology Network Attached Storage (NAS) File level I/O Used for file sharing applications IP-based Deployed over existing low-cost Ethernet networks Redundant links Scalable Multiple servers can share same file system IP NAS NAS NAS 13 Terminology Storage Area Network (SAN) Block level I/O Deployed as separate network Servers share storage subsystem Scalable Multiple paths for high availability 14 Printed in USA.

8 STORAGE NETWORKING DEVICES 15 SAN Components Host Bus Adapter (HBA) Interface between host and storage Supports copper or optical Typically one port; Can be multiple ports 1Gb, 2Gb and 4Gb 16 Printed in USA.

9 SAN Components Fabric Switch 1Gb, 2Gb, and 4Gb 8-40 ports Low latency Can be copper or optical 17 SAN Components Director Class Switch 1Gb, 2Gb, 4Gb and 10Gb FC and FICON 256 ports and growing Low latency Can be copper or optical Multi-service platforms 18 Printed in USA.

10 SAN Components JBOD Just a bunch of disks Limited scalability Typically 2 FC ports SCSI or FC disks Basic controllers No caches 19 SAN Components Storage Arrays 36GB to many TB Typically 2 to many interfaces Subsystems may mix interfaces ESCON/FICON, SCSI, FC, or iscsi SCSI or FC disks Intelligent controllers Large caches iscsi 20 Printed in USA.

11 SAN Components Tape Arrays Tape speed vary 5MBs 30MBs+ Capacity vary 20GB 300GB+ Deployed in servers or external libraries SCSI, FC, Ethernet interface DLT most common; LTO gaining traction 21 STORAGE NETWORKING APPLICATIONS 22 Printed in USA.

12 IT Storage Requirements Scalability Meet high growth demand for storage capacity (>80% per year) Increase capacity utilization rates Availability Share data across distributed data centers via fast speed, long distance connectivity links Provide effective disaster recovery Improve interoperability across heterogeneous equipment Enhance security Manageability Automate storage management functions Provide cross-vendor management tools Managing heterogeneous environments 23 Storage Network Build-Out Application-specific islands of networked storage Homogenous Infrastructure Isolated Islands iscsi Convenient extension of existing FC SAN to IP-attached servers Extensive IP services for NAS environments DAS SAN NAS Starting Point 24 Printed in USA.

13 Storage Network Interconnection SAN interconnection for Business continuance Unified management Remote backup Metro DWDM solutions FCIP Low-latency option for synch replication Lower-cost option for asynch replication and backup consolidation SAN Interconnectivity FCIP FCIP Optical Present Trend 25 Intelligent SAN Intelligent services into the network Common management framework Content, file, and block awareness Transport independent Data Mgmt Services Storage Switching Host Awareness Storage Utility SAN Storage Virtualization Storage Routing Content Delivery Storage Management 26 Printed in USA.

14 STORAGE NETWORKING TOPOLOGIES 27 SCSI I/O Topology SCSI is the protocol used to communicate between servers and storage devices SCSI I/O channel provides a half-duplex pipe for SCSI commands and data Parallel implementation Bus width: 8, 16 bits Bus speed: 5 80 Mhz Throughput: MBps Devices/bus: 2 16 devices Cable length: 1.5m 25m A network approach can scale the I/O channel in many areas (length, devices, speed) Initiator Host System SCSI SCSI Adapter Target 28 Printed in USA.

15 Fibre Channel Topology Very common method for networking SCSI Fibre Channel provides high-speed transport for SCSI payload Fibre Channel overcomes many shortcomings of DAS including: Addressing for up to 16 million nodes (24 bits) Loop (shared) and Fabric (switched) transport Speeds of 100 or 200 Mbps (1 or 2 Gbps) Distance of up to 10km (without extenders) Support for multiple protocols Combines best attributes of a channel and a network Initiator Host System SCSI Fibre Channel HBA Fibre Channel Fabric Target 29 iscsi Storage Topology IP access to open storage sub-systems iscsi-enabled Hosts (Initiators) iscsi iscsi iscsi iscsi driver is loaded onto hosts on ethernet network iscsi Array (Target) iscsi Able to consolidate servers via iscsi onto existing storage arrays Able to build ethernet-based SANs using iscsi arrays Storage assigned by iscsi instance IP Network iscsi Router FC Fabric Storage Pool (Target) FC HBA Attached Host (Initiator) 30 Printed in USA.

16 FCIP SAN Extension Topology FCIP gateways perform Fibre Channel encapsulation process into IP packets and reverse that process at the other end FC Switches connect to the FCIP gateways through an E_Port for SAN fabric extension to remote location A tunnel connection is set up through the existing IP network routers and switches across LAN/WAN/MAN Storage Servers FC SAN FC Switch Production Site Production Site Existing IP FCIP EMC SRDF Gateway Network LAN/WAN/MAN Database Servers FC SAN Backup Server FCIP Gateway FC Switch Storage Backup, R&D, Standby Shared Storage, Data Warehousing, Etc. 31 FCIP and iscsi: Complementary FCIP: SAN-to-SAN over IP iscsi: Host to storage over IP Storage Router iscsi iscsi IP Network iscsi iscsi Storage Router FC SAN FCIP FC SAN FCIP Gateway FCIP Gateway 32 Printed in USA.

17 INTRODUCTION TO STORAGE PROTOCOLS 33 Introducing SCSI SCSI = Small Computer System Interface SCSI is a standard that defines an interface between an initiator (usually a computer) and a target (usually a storage device such as a hard disk) INTERFACE refers to connectors, cables, electrical signals, optical signals and the command protocol that allow initiators and targets to communicate 34 Printed in USA.

18 SCSI Example In this Case, a File is Being Written to the Hard Drive By an Application on the Workstation Initiator Target 1 Target 2 Sun SCSI Connector Tape SCSI Cable Sun Disk The SCSI Command Protocol Is Used to Communicate Between SCSI Devices Opcode (2A = Write 10) Reserved LBA LBA (0010E43) LBA Reserved LBA Len (128) LBA Control SCSI Command 35 Why Is SCSI Important for SANs? SCSI command protocol is the de facto standard that is used extensively in high-performance storage applications The command part of SCSI can be encapsulated in FCP Fibre Channel Protocol or IP and carried across internetworks; This is the core concept behind storage area networking To understand the finer points involved with transporting SCSI across a network with FC or ethernet, the basics of SCSI must be well understood 36 Printed in USA.

19 Standards SCSI has evolved since it was introduced as SASI in 1979 by Shugart Associates it was approved as a standard by ANSI in 1986 and is now referred to as SCSI-1 SCSI-2 was approved by X3 in 1990 and by ANSI in 1994 SCSI-3 refers to a collection of standards, each of which defines a very specific part of SCSI: physical interface, transport interface, command interface, architecture model, programming interface, etc. 37 Sample SCSI Standard Components SCSI Parallel Interface: SPI Initiator Sun Target 1 Sun Target 2 38 Printed in USA.

20 Sample SCSI Standard Components SCSI Primary Commands: SPC SCSI Primary Commands (SPC-2) Initiator Target 1 Target 2 Sun Sun SCSI Block Commands (SBC) SCSI Stream Commands (SSC) 39 SCSI Standards: The Big Picture CAM ASPI Generic SBC SSC SES More SPC-2 / SPC-3 FCP SBP ATAPI SPI-x FC-xx Printed in USA.

21 SCSI Architecture Model This specification describes a reference model for the coordination of standards applicable to SCSI-3 I/O systems and a set of common behavioral requirements which are essential for the development of host software and device firmware that can interoperate with any SCSI-3 interconnect or protocol. SCSI Architecture Model November SCSI Architecture Model The SCSI architecture model defines generic requirements and implementation requirements Each SCSI implementation standard must fulfill the requirements set forth by SAM 42 Printed in USA.

22 SAM Highlights: Client-Server SCSI is a client-server protocol The client is called the initiator (this is usually the OS I/O subsystem) and issues requests to the server The server is called the target (this is usually the SCSI controller that is part of a storage device) and receives, executes and returns initiator requests and their associated responses 43 SAM Highlights: Initiator: Target A single initiator can have multiple application clients Targets have ONE task manager and one OR MORE Logical Units (LU), which are numbered (LUN) The task manager has the authority to modify service requests that have already been received by the target 44 Printed in USA.

23 SAM Highlights: Logical Units Each logical unit within a target is numbered; that number is called a LUN and is the only way to refer to that logical unit The device server is the entity that receives, executes and returns requests that are made to its logical unit The concept of task set is beyond the scope of this presentation 45 SAM Highlights: Command Model SAM defines two categories of protocol services: Execute command/confirmation services; Data transfer services This leads to the three main phases of a data transfer: 1. Execute: Send required command and parameters via CDB; 2. Data: Transfer data in accordance with the command; 3. Confirmation: Receive confirmation of command execution 46 Printed in USA.

24 SAM Highlights: Sample Data READ 1. Send SCSI Cmd issued by initiator the command sent is READ; 2. SCSI command received by target; Data transfers occur during the working phase between initiator and target; 3. Send command complete is returned by the target; 4. Command complete received by target 47 SAM Highlights: Parameters The data transfer model reflects parameters that will be used by SCSI commands This model illustrates that a complete data transfer (right) can be broken up into multiple parts (left) 48 Printed in USA.

25 SAM Highlights: Communication Model Let s Expand on this Portion SAM Defines a Hierarchy of Protocols 49 SCSI Transport Protocol SCSI Protocol FCP Parallel Bus iscsi ifcp TCP IP Ethernet FibreChannel FCIP 50 Printed in USA.

26 SCSI Transport Protocol SCSI Protocol FCP Parallel Bus iscsi ifcp FibreChannel FCIP TCP IP Ethernet Today s In-Depth Protocol Discussions 51 STORAGE PROTOCOLS IN-DEPTH 52 Printed in USA.

27 Section Agenda Introduction to Standards SCSI Protocol Fibre Channel Protocol Internet SCSI (iscsi) Fibre Channel over IP (FCIP) Internet Fibre Channel Protocol (ifcp) isns and SLP 53 INTRODUCTION TO STANDARDS 54 Printed in USA.

28 Standards Groups: Storage ISO / IEC JTC-1 American National Standards Institute (ANSI) InterNational Committee for Information Technology Standards (INCITS) Information Technology Industry Council (ITI) C J11 C++ J16 Techincal Committee on Lower-Level Interfaces (T10) Techincal Committee on Device-Level Interfaces (T11) Techincal Committee on AT Attachment Interfaces (T13) SCSI Fibre Channel HIPPI IPI ATA (IDE) ATAPI 55 Standards Process Technical Committees (T10) write drafts Drafts are sent to INCITS for approval Once approved by INCITS, drafts become standards and are published by ANSI ANSI promotes american national standards to ISO as a Joint Technical Committee member (JTC-1) 56 Printed in USA.

29 Standards Work Group: IP Storage IP Storage Technical Work Group Acts as Primary Technical Focal Point of the Storage Networking Industry Association (SNIA) on IP Storage Issues, Coordinating with the SNIA IP Storage Forum ISOC Internet Society Transport Area Has 23 WGs, One which Is the IP Storage WG IEFT Is the Organization Ratifying the IPS Standards IESG Internet Engineering Steering Group Transport Area IETF Internet Engineering Task Force 57 FIBRE CHANNEL IN-DEPTH 58 Printed in USA.

30 Fibre Channel Protocol Agenda FC Introduction Fibre Channel Communications Model Protocol Constructs FC-PH (Fibre Channel Physical and Signaling Interface) Login Parameters Frame Processing Arbitrated Loop Switch Fabric Operation Switch and Hub Mixed Topology Network Operations FC Error Management 59 Fibre Channel Environment Channel reliability Multiprotocol support Overshared serial media With networking capability and functionality 60 Printed in USA.

31 Fibre Channel Environment High bandwidth High data integrity Highly reliable Destination paced Buffer credits Scalable High availability Shared media Circuit/packet Multiple protocol support Transport flexibility Dedicated conn Class 1 Multiplexed Class 2 Datagram Class 3 Configuration flexibility Switch Loop 61 What Is It? Channels Connection service Physical circuits Reliable transfers High speed Low latency Short distance Hardware intense Networks Connectionless Logical circuits Unreliable transfers High connectivity Higher latency Longer distance Software intense 62 Printed in USA.

32 What Is It? Fibre Channel Channels Connection service Physical circuits Reliable transfers High speed Low latency Short distance Hardware intense Fibre Channel Circuit and packet switched Reliable transfers High data integrity High data rates Low latency High connectivity Long distance Networks Connectionless Logical circuits Unreliable transfers High connectivity Higher latency Longer distance Software intense 63 Fibre Channel Protocol Levels Levels FC-4 s HIPPI Cluster 370 OEM SCSI IP ATM FC-3 FC-2 FC-1 FC-0 Common Services Signaling Protocol Transmission Code Physical Interface FC-PH N_Port F_Port PC-PH = Physical and Signaling Layer 64 Printed in USA.

33 FC Fibre Channel Functions Structure Is Divided into 5 Levels of Functionality FC-0 defines the physical interface characteristics Signaling rates, cables, connectors, distance capabilities, etc. FC-1 defines how characters are encoded/decoded for transmission Transmission characters are given desirable characters FC-2 defines how information is transported Frames, sequences, exchanges, login sessions FC-3 is a place holder for future functions FC-4 defines how different protocols are mapped to use Fibre Channel SCSI, IP, virtual interface architecture, others 65 Fibre Channel Topologies N N Point to point L L L L Arbitrated loop L L Switched fabric N N F F F F N N 66 Printed in USA.

34 Point to Point Dedicated connection between N port Fibre Channel devices All link bandwidth is dedicated to communication between the two nodes Suitable for small scale scenarios when storage devices are dedicated to file servers NN N N 67 Arbitrated Loop (FC-AL) TX of each node is connected to the RX of the next node until a closed loop is formed Maximum bandwidth: 100 MB/sec. (shared amongst all nodes on loop) 126 nodes max on loop Not a token passing scheme no limit on how long a device may retain control Operational sequence: Arbitrate for control of loop Open channel to target Transfer data Close Number of nodes on loop directly affects performance L L L L L L L L L FC Fibre Channel Hub 68 Printed in USA.

35 Data Integrity Upper Level Protocol Signaling Protocol Transmission Code Operation Control and Byte Counts Operation Frame counts CRC (32 bit) Frame delimiters 8b/10b Code Physical Media Fibre Reliability 69 Flow Control Back pressure technique Frame credit Established by receiver during LOGIN Transmitter Must have credit to transmit Receiver Reinstates credit with ACK 70 Printed in USA.

36 FIBRE CHANNEL COMMUNICATIONS MODEL 71 The Model The Fibre Channel communications model is based on the definition of: Physical objects Protocol construct These objects and constructs: Define the behavior of the physical elements Control the transfer on information Provide for link management Provide the basis for: Hardware Firmware Software 72 Printed in USA.

37 Physical The fundamental physical objects in Fibre Channel are: Ports Link Nodes Fabric Some Logical Items Used in These Discussion Are: Addressing Communications Model 73 Fibre Channel: Port Types N port: Node ports used for connecting peripheral storage devices to switch fabric or for point to point configurations; can be considered the end port F port: Fabric ports reside on switches and allow connection of storage peripherals ( N port devices) L port: Loop ports are used in arbitrated loop configurations to build storage peripheral networks without FC switches; these ports often also have N port capabilities and are called NL ports E port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches G port: A generic port capable of operating as either an E or F port; its also capable of acting in an L port capacity; Auto Discovery 74 Printed in USA.

38 N_Port Host / Device Host/ Device Interface N_Port Serial Data Out Serial Data In 75 Link A link consists of 2 unidirectional fibers transmitting in opposite directions May be either: Optical fiber Copper Transmitters may be: Long wave laser Short wave laser LED Electrical Link Transfer Rates Clock Mbaud/sec Mbytes/sec Printed in USA.

39 Link Host / Device Host/ Device interface N_Port Serial Data Out Serial Data In Link 77 Node The equipment which contains one or more N_Port or NL_Port (topology dependent) May be Computer Controller Device Is NOT a switch fabric 78 Printed in USA.

40 Node Controller N_Port N_Port N_Port N_Port Link Link Link Link 79 Communications Model Point to point N_Port to N_Port Flow control Acknowledged N_Port Node Transmitter Node Receiver N_Port Receiver Transmitter Transmitter Link 80 Printed in USA.

41 Fabric Fabric The entity which interconnects N_Ports Provides routing based on destination address Fabric may be: Point to point No routing required Switched Routing provided by switch Arbitrated loop Routing is distributed throughout attached L_Ports 81 Terms Topology The physical structure of the interconnect of ports Defines the logical behavior of transactions Fibre channel has 3 topologies Pt to Pt Switched Arbitrated loop Fabric The fabric is the generic item that interconnects nodes A fabric is made of Fibre Channel topologies like Pt to Pt, switches and loops 82 Printed in USA.

42 Point to Point Node N_Port Transmitter Fabric Node Receiver N_Port Receiver Transmitter Communications Model Source to destination Based on address routing through the fabric 83 Switched Fabric N_Port N_Port N_Port N_Port N_Port N_Port Switch Fabric Communications Model Source to Destination Based on Address Routing through the Fabric A B 84 Printed in USA.

43 Arbitrated Loop NL_Node A Link NL_Node B Communications Model Source to Destination Based on Address Routing Distributed in the NL_Ports on the Loop A B 85 FC PROTOCOL CONSTRUCTS 86 Printed in USA.

44 What Are Protocol Constructs The fundamental protocol structures in the Fibre Channel are called constructs, and they are: Frames Sequences Exchanges Information Units (IU) Procedures Upper Layer Protocols (ULP s) 87 Construct Introduction FC-2 defines these constructs that allow the related information to be: Grouped together Coordinated Handled in an efficient manner To accomplish this we define the notion of: Frames Sequences Exchanges Also defined are means for the Upper Level Protocols ULP s to communicate with FC-2: Information Units (IU) A procedure called the login defines the operating environment between the N_Ports Exchange of the data describing the characteristics of the ports 88 Printed in USA.

45 Chunks The ULP s deal with chunks of data that are moved across the network These chunks of data may be either Control Status Real data 89 Frames Frame FC-2 layer will take this chunk of data and move it from Transmitting node to receiving node In the units of what Fibre Channel calls frames Frame Size FC-2 will determine the size of the frames based on operating environment established between the two communicating nodes FC-3 FC-2 FC-1 FC-0 Common Services Signaling Protocol Transmission Code Physical Interface 90 Printed in USA.

46 Frame Structure General FC-2 Frame Format Frame Format Idles SOF Frame Header Data Field CRC EOF Idles 24* Bytes CRC Calculated on Frame Header and Data Field Only * 6 Idle Words (24 bytes) Requires by TX 2 Idle Words (8 bytes) Guaranteed to RX 91 Frame Header Word R_CTL Routing 2 4 CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits 2 3 DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID D_ID 24bits Destination 8 S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type 7 Frame Control SEQ_CNT 8 bits Sequence Count RX_ID 8 bits Respon Exch ID 0 92 Printed in USA.

47 Data Field Data Field Optional Headers Payload F I L L Typical MTU 93 Sequence Sequences Each chunk of Upper Level Protocol (ULP) data is moved within the envelope of what Fibre Channel calls a Sequence (SEQ) A sequence consists of a set of related frames As expected there are lots of rules governing sequences Information Units (IU) The ULP tells the FC-2 how to transfer theses chunks of data through a structure called a information unit Very few rules for IU s IU is a convention defined outside of FC-PH IU s are unique to each upper level protocol 94 Printed in USA.

48 Sequence Sequence Initiator (SI) The N_Port which is transmitting the data frames Sequence Recipient (SR) The N_Port which is receiving the data frames SI N_Port Data Chunk Fabric Data Frame SR N_Port Data Chunk 95 Sequence Initiator (SI) Read Command (Chunk) Fabric Data Frame Target (SR) Sequence (SR) Sequence Data Frame (SI) Data (Chunk) Sequence Data Frame Status 96 Printed in USA.

49 Sequence Identifier Sequence initiator assigns an identifier to each sequence This identifier is called the Sequence_Identifier or Seq_ID The Seq_ID uniquely identifies a given sequence within the context of the operation Each frame is identified within this operation by Seq_ID and Seq_CNT 97 Sequences: Active and Open Sequence Initiator (SI) A sequence is ACTIVE From the time the first frame of the sequence is transmitted until the frame with the end sequence flag is sent A Sequence is OPEN From the time the first frame is transmitted until the reception of the ACK to the last frame Sequence Recipient (SR) A sequence is ACTIVE and OPEN From the time of the first frame of the sequence is received until the transmission of the ACK to the last frame of that sequence 98 Printed in USA.

50 Sequences: Active and Open Originator (SI) First Data_Frame Responder (SR) SOF Received ACK to first Frame Active Open Active & Open Frame with End_SEQ set ACK to last Frame EOF Transmitted EOT Received 99 Sequence Streaming Sequence streaming is the ability to Begin transmission of the next sequence while one or more previous sequences are OPEN Sequence Recipient (SR) grants permission to have up to n streaming sequences; This is determined at N_Node login time Must Support n=1 sequence status blocks (state info) (This Allows for More Data in the Pipe for Distant Connections) 100 Printed in USA.

51 Exchange Upper level protocols frequently deal with related bits of data as: Request/reply Command/data/status These relationships are called operations Exchanges Operations of data grouped together into what Fibre Channel call exchanges An exchange consists of a set of related sequences Exchanges are bi-directional Sequences are unidirectional and sequential There are other rules that govern exchanges 101 Exchange Initiator (SI) Read Command (Chunk) Fabric Data Frame Target (SR) Sequence (SR) Sequence Data Frame (SI) Data (Chunk) Exchange Data Frame Status Sequence 102 Printed in USA.

52 Exchange Exchange originator The N_Port which transmitted the FIRST data frame for this exchange Exchange responder The N_Port which is the destination of the FIRST data frame of this exchange The designation for the originator and responder are fixed for the duration of the exchange Unlike the SI and SR Which Change Roles Within the Exchange 103 Exchange Identifiers X_ID An exchange has two identifiers associated with it Exchange originator: Assigns an OX_ID which is meaningful to it Exchange responder: Assigns a RX_ID which is meaningful to it In general terms it is called the X_ID Meaningful is that in the exchange there is context with information like state, control, and status with regards to the exchange An N_Port will save, create and update this information throughout the exchange based on the assigned X_ID s 104 Printed in USA.

53 Information Unit Upper Level Protocols (ULP s) know about Information Units (IU s) but know nothing about: Frames Sequences Exchanges A ULP deals with units like: Order of events within the operation Which node will transmit in the next phase (Command phase, data phase, status phase) Is required to have some knowledge about Fibre Channel An information unit is a Fibre Channel sequence 105 Information Unit The IU contains information sets with such items as LUN, task attributes, CDB and the command byte count The IU s are used in protocol mapping from FC-4 to FC-2 and are assigned an identifier that is useful to humans not used by the machine All the information needed to support a ULP is formed into a IU table and is listed as a first, middle or last IU in the exchange We Will See More of these Tables when We Cover SCSI Mapping onto Fibre Channel 106 Printed in USA.

54 FC-2 Hierarchy The Hierarchy of Constructs Construct Exchange Information Unit Sequence Frame Meaning Consists of one or more Sequences for ULP Operation The structure used by the ULP to define a Sequence (not visible over link) Consists of one or more related Frames Contains in its Payload a ULP chunk of data Frame Fields OX_ID / RX_ID SEQ_ID SEQ_CNT 107 FC-2 Hierarchy Frame Fields OX_ID & RX_ID EXCHANGE SEQ_ID SEQUENCE... SEQ_CNT Frame Frame... Frame Information Unit Per ULP Terms 108 Printed in USA.

55 FC-PH (FIBRE CHANNEL: PHYSICAL AND SIGNALING INTERFACE) STRUCTURE, PROCEDURES, AND PROTOCOLS 109 Transmission Code Fibre Channel uses a 8b/10b transmission code Each 8 bit data byte to be transmitted is converted into a 10 bit quantity The 10 bit quantity is then transmitted over the FC media The 10 bit quantity is then converted back to the 8 bit data byte by the receiving node The 10 bit quantities are called transmission characters Transmission characters come in two forms Data charters Special characters 110 Printed in USA.

56 8b/10b Code Why 8b/10b 1. To ensure the sufficient transitions are present in the serial bit stream to make clock recovery possible at the receiver 2. Increase the likelihood of detecting any single or multiple bit errors 3. To provide special characters with distinctive and easily recognizable characters to achieve word alignment on the incoming bit stream 111 8b/10b Code Characteristics of 8b/10b The 10 bit transmission code Supports all 256 values of the 8 bit data byte Contains unused code points Illegal codes(called code violations) Detection of code violations May occur on the transmission character in which the error occurred or may be detected on a subsequent character Contains special characters Running disparity with DC balance (Count of 0 s and 1 s Equal the Same over Time) 112 Printed in USA.

57 8b/10b Code Running Disparity Disparity: The Difference Between the Number of Ones and Zeros in a Transmission Character Running Disparity: A Binary Parameter Indicating the Cumulative Disparity of All Previously Issued Transmission Characters Transmission Characters Always Have Either: 6 Ones and 4 Zeros = Positive Disparity 4 Ones and 6 Zeros = Negative Disparity 5 Ones and 5 Zeros = Neutral Disparity Rules: A Positive Disparity Transmission Character Can Not Be Followed By Another Positive Transmission Character A Negative Disparity Transmission Character Can Not Be Followed By Another Negative Transmission Character At Transmission Character Boundaries the Difference between the Number of Ones and Zeros is + or b/10b Code Code Notation Each valid transmission character has been assigned a name in the form of: Zxx.y Z = K or D D=Data K=Special Character xx = Decimal Value of the 5 LSb bits y = Decimal Value of the 3 MSb bits 114 Printed in USA.

58 Conversion Table MSB j Transmission Order LSB FC-2 Bits H G F i E D C B A FC-1 Code Bit Example D1.0 D or K FC-1 Transmission Character Neg Disp Value j and i are add as part of the 10b conversion process 115 Special Characters K28.5 only special character used in Fibre Channel out of the 12 set aside Has no 8 bit representation The only FC transmission character with 5 consecutive 1 s or 0 s Used to find word boundaries and sync Used in ordered sets Current Running Disparity Current Running Disparity 116 Printed in USA.

59 Transmission Transmission word consists of 4 continuous transmission characters treated as a unit 40 bits long Aligned on a word boundary There is a ordered set and a data word Transmission Order Byte Ordered Set 0 K Encoded Data Byte Encoded Data Byte Encoded Data Byte Data Word Encoded Data Byte Encoded Data Byte Encoded Data Byte Encoded Data Byte 117 Ordered Set Transmission word starting with the K28.5 special character Three classifications of ordered sets are defined Delimiters Primitive signals Primitive sequences MSB LSB K28.5 Dxx.y Dxx.y Dxx.y The Three Data Characters Define the Meaning of the Ordered Set and Are Repeated for the Third and Fourth Character 118 Printed in USA.

60 Primitive Signals Primitive signals are ordered sets Transmission of primitive signals are interrupted occasionally to transmit frames Three basic types Receiver_Ready (R_Rdy) Idle (idle or I) Arbitrate (ARBx) 119 Delimiters Delimiters are ordered sets that delineate a frame Immediately preceding and following the contents of a frame Two basic types Start_of_Frame (SOF) End_of_Frame (EOF) SOF delimiters Identify the start of a frame Identify the transmission class Used to establish a Class_1 connection Identify the beginning and continuation of a sequence EOF delimiters Terminate frames Identify the end of a sequence Terminate connections Indicate known frame errors 120 Printed in USA.

61 FC-1 Synchronization Procedures Sync acquire Initialization Loss of sync procedure Primitive sequences 121 Sync Procedures Bit synchronization The state in which a receiver is delivering retimed serial data at the required bit error rate Transmission word synchronization Achieved when the receiver identifies the same transmission word boundary on the receive bit stream as the established by transmitter at the other end of link Acquired by detection of three consecutive ordered sets without errors Loss of synchronization procedure The receiver shall enter the loss-of-sync state upon detection of the fourth invalid transmission word Synchronization acquired procedure The receiver shall enter the synchronization-acquired state when it has achieved both bit and transmission word sync 122 Printed in USA.

62 Synch Acquired Loss of Sync State Waiting on Bit Synchronization Bit Sync Acquired Data Word Rx Ordered set #1 Data Word Rx Ordered set #2 Data Word Rx Ordered set #3 Sync Acquired 123 Loss-of-Sync Procedure Sync Acquire State One Invalid Word in Next 2 Words One Invalid Word in Next 2 Words No Invalid Words Detected First Invalid Word Second Invalid Word Two Consecutive Valid Words Two Consecutive Valid Words One Invalid Word in Next 2 Words Third Invalid Word Two Consecutive Valid Words One Invalid Word in Next 2 Words Fourth Invalid Word Loss Of Sync 124 Printed in USA.

63 FC-1 Constructs Port states Primitive sequences NOS/OLS/LR/LRR Primitive sequence protocols Sequence flows Relationships Port state transition table 125 Port States Four primary operational states Active state Link recovery state Link failure state Offline state Operational states of a port N_Ports F_Ports Port state changes occur as a result of Conditions detected within the port In response to reception of primitive sequences In response to upper level controlling entity 126 Printed in USA.

64 Primitive Sequences Ordered set that is transmitted continuously to indicate that specific conditions within the port are encountered Transmitted while the condition exist Four primitive sequences Not Operational Sequence (NOS) Offline Sequence (OLS) Link Reset Sequence (LR) Link Reset Response Sequence (LRR) 127 Primitive Sequence NOS Not_Operational Sequence Transmitted by the port to indicate that Link failure had been detected Loss of sync Loss of signal Port is offline K28.5 D21.1 D31.5 D Printed in USA.

65 Primitive Sequence OLS Offline Sequence Transmitted by port to indicate that it is: Initiating the link initialization protocol Receiving NOS Entering the Offline state K28.5 D21.2 D10.4 D Primitive Sequence LR Link Reset Sequence Transmitted by port to indicate that it is: Initiating the link reset protocol To recover from a link timeout To remove a Class_1 connection K28.5 D9.2 D31.5 D Printed in USA.

66 Primitive Sequence LLR Link Reset Response Sequence Transmitted by port to indicate that: Link reset is being received K28.5 D21.1 D31.5 D Primitive Sequence Protocols Link Initialization Protocols Required after Port power-on Port internal reset Port has been in offline state Online to offline protocols Required to enter offline state 132 Printed in USA.

67 Primitive Sequence Protocols Link Failure Protocol Required after Detection of loss of synchronization for a period of time greater than 100ms which is the receiver-transmitter timeout value (R_T_TOV) Loss of signal while not in the offline state Link Reset Protocol Required after Link reset Link timeout 133 Primitive Sequence Flows NOS Link Failure State (LF) Offline State (OL) Link Recovery State (LR) Active State (AC) Link Reset Protocol Link Initialization Protocol Link Failure Protocol Online to Offline Protocol Idle 134 Printed in USA.

68 Primitive Sequence Meanings Currently Transmitting NOS OLS Not Operational Link Failure Meaning Offline State Internal port failure Transmitter power down, perform diags, or perform initialization Receiver shall ignore Link error or Link Failure Transmit in Response OLS LR LR LRR Link Reset Remove class_1 Conn Reset F_Port OLS recognized Link Reset Response Link Reset Recognized LRR Idles IDLE Operational Link Idles and R_RDY recognized Idles or R_RDY 135 Link Failure Port A AC Port B AC Link Failure Condition LF NOS OLS LF LR AC LR Idle LRR Idle OL LR AC AC = Activity State LR = Link Recovery State LF = Link Failure State OL = Offline State 136 Printed in USA.

69 Offline Port A AC Port B AC Request to Go Offline OL OLS LR OL After 5ms Minimum Diags May Be Preformed LR LRR Idle Idle LR AC Request to Go Online AC = Activity State LR = Link Recovery State AC Idle LF = Link Failure State OL = Offline State 137 Frame Header Detail Routing control (R_CTL) Addressing (D_ID) (S_ID) Type (TYPE) Frame control (F_CTL) Sequence identifier (SEQ_ID) Sequence count (SEQ_CNT) Exchange identifiers (OX_ID) (RX_ID) Parameter field 138 Printed in USA.

70 Frame Detail: Routing Control Word R_CTL Routing CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID D_ID 24bits Destination S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type Frame Control SEQ_CNT 16 bits Sequence Count RX_ID 16 bits Respon Exch ID 139 Routing Control The Routing control field is an 8 bit field R_CTL consist of two 4 bit sub-fields Routing Information category Routing Info Category 140 Printed in USA.

71 Routing Control The R_CTL is used to direct the frame to the process the frame is directed to; For example: Frames directed to the fabric for extended link services (0x22) Indication of the function or purpose of the frame payload from the upper level protocol at FC-4 (0x01) 141 Port Addressing Word R_CTL Routing CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID D_ID 24 bits Destination S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type Frame Control SEQ_CNT 16 bits Sequence Count RX_ID 16 bits Respon Exch ID 142 Printed in USA.

72 Port Addressing D_ID and S_ID fields are 24 bits each They provide the address or identifier of the Source and destination port of a frame Although the address map is flat, there are several formats depending on: Topology Location 143 Port Address Identifiers Applicable to all topologies Point to point Switched Loop Dynamically assigned or administratively assigned Used for frame routing Unique within Fibre Channel network Assigned by the fabric Some address reserved for special functions 144 Printed in USA.

73 Port Address Identifiers Topology Point To Point Switched Assignment By N_Port with Higher Worldwide Name (MAC) By Switch During Fabric Logon Bound to Physical Port on Switch Arbitrated Loop Acquired During Loop Initialization 145 Address Identifiers 8 bits 8 bits 8 bits Switch Topology Model Switch Domain Area Device Private Loop (Not Connected to a Switch) Arbitrated Loop Physical Address (AL_PA) Public Loop (Connected to Switch) Domain Area AL_PA 146 Printed in USA.

74 Reserved Addresses FC-PH has defined a block of addresses for special functions: High order 16 addresses in the 24 bit address space Called the well known addresses Main Address Used Today FF FF FC FF FF FD FF FF FE Directory Server Fabric Controller Fabric F_Port which N_Port is attached to 147 Data Structure Type Word R_CTL Routing D_ID 24bits Destination CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type Frame Control SEQ_CNT 16 bits Sequence Count RX_ID 16 bits Respon Exch ID 148 Printed in USA.

75 Type The TYPE is a 8 byte field Indicates the upper level carried in the payload of the frame Examples: SCSI 08h IP 05h SNMP 24h Fibre Channel services 20h 149 Frame Control Word R_CTL Routing CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID D_ID 24bits Destination S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type Frame Control SEQ_CNT 16 bits Sequence Count RX_ID 16 bits Respon Exch ID 150 Printed in USA.

76 Frame_Control The frame control is a 24 bit field It contains a number of flags that are used to control the flow of the sequence The more common flags are exchange and sequence management, acknowledgement control and error conditions Bits deal with the sequence and exchange settings Bits deal with X_ID Bits form the ACK level for class 1 & 2 Bits 5-4 used for aborting the sequence 151 Frame Control Bits Acknowledgment Capability Provide assistance to Sequence Recipient (SR) by translating the ACK capabilities bits in the N_Port class parameters Meaningful only in Class 1 and 2 data frames 0 0 = No ACK 0 1 = ACK level 1 one for every frame 1 0 = ACK level N N = number of frames 1 1 = ACK Level 0 single ACK for complete exchange, used in video streaming 152 Printed in USA.

77 Sequence Identifier Word R_CTL Routing CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits 2 3 DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID D_ID 24bits Destination 8 S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type 7 Frame Control SEQ_CNT 16 bits Sequence Count RX_ID 16 bits Respon Exch ID Sequences Sequences Deal with chunks of upper level protocol Are made up of one or more frames which transport the ULP The data phase may be subdivided into multiple sequences Uniquely identifiable with SEQ_ID The command, data, and status phases of SCSI are examples of sequences 154 Printed in USA.

78 Sequence Identifier The Sequence Identifier (SEQ_ID) is a 8 bit field All Frames of a sequence will carry the same SEQ_ID value Data content of these frames are related in some way by the ULP 155 Sequence Count Word R_CTL Routing CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID D_ID 24bits Destination S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type Frame Control SEQ_CNT 16 bits Sequence Count RX_ID 16 bits Respon Exch ID 156 Printed in USA.

79 Sequence Count Sequence count (SEQ_CNT) is a 16 bit field Identifies the order of the transmission of frames within this sequence Used by Sequence Recipient (SR) to account for all transmitted frames Used by Sequence Initiator (SI) to account for all transmitted acknowledges (ACK s) in Class 1 and Sequence Count Within a Sequence_Initiative The SEQ_CNT of the first data frame will be zero The SEQ_CNT of each subsequent data frame in the sequence will be incremented by 1 The first data frame of the next sequence may be either zero or one more then the last data frame, this is called continuously increasing SEQ_CNT If streamed sequences is used, continuously increasing SEQ_CNT is required 158 Printed in USA.

80 Sequence Count Sequence initiator Assigns SEQ_CNT to data frames Keeps a record of ACK frames received Sequence recipient Records SEQ_CNT of data frames Transmits an ACK frame for each valid data frame when Rx buffer is available Knows that sequence was received without error if all Frames are Rx without errors and are accounted for Sequence initiator Knows the sequence was received without error if it has Rx an ACK frame to all frames within the sequence 159 Exchange Identifiers Word R_CTL Routing CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID D_ID 24bits Destination S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type Frame Control SEQ_CNT 16 bits Sequence Count RX_ID 16 bits Respon Exch ID 160 Printed in USA.

81 OX_ID and RX_ID 2 byte fields each Contain the originator exchange identifier and responder exchange identifier They point to state and context information regarding the exchange in the originator port and responder port OX_ID s are reused after each exchange is over 161 Parameter Field Word R_CTL Routing CS_CTL 8 bits Class Spec TYPE 8 bits Data structure SEQ_ID 8 bits DF_CTL 8 bits Data field OX_ID 16 bits Orig Exch ID D_ID 24bits Destination S_ID 24 bits Source F_CTL 24 bits Parameter Specific to frame type Frame Control SEQ_CNT 16 bits Sequence Count RX_ID 16 bits Respon Exch ID 162 Printed in USA.

82 Parameter Field The parameter is a 4 byte field The content of the parameter field is dependent on the specific frame type as identified in the routing field FC-4 data frames ACK link control Port reject and frame reject frames Port busy and fabric busy frames 163 LOGIN PARAMETERS 164 Printed in USA.

83 Login Procedure to Determine the Operating Environment for Communications between Two Ports Exchange service parameters done with login frame PLOGI or FLOGI Required before communications can be established between the two ports Applies to all topologies Applies to all ports, node and fabric Bi-directional ACCEPT Frame contains service parameters of the port addressed 165 Login Service Parameters Contain the Following Type of Information Version of Fibre Channel support N_Port or F_Port functionality Service classes supported Size of receive buffers Number of sequences supported Support for Intermix ACK capability Error policy supported Others 166 Printed in USA.

84 ACK s Informs Transmitter that: One or more valid data frames were received by the sequence recipient for the corresponding sequence qualifier Interface buffer is available for another data frame, this only applies to class 1 and class 2 Class 3 are not ACK ed Flow control Re-instates end-to-end credit 167 ACK s Frame Header Constructed from the data frame which is being acknowledged S_ID and D_ID are swapped F_CTL with both exchange and sequence context bit inverted SEQ_ID is unchanged SEQ_CNT is set to the sequence count of the highest data frame being replied to by the ACK Parameter Field Bit 16 = History bit Bits 0-15 are ACK type specific 168 Printed in USA.

85 ACK s Again there are three types of ACK s ACK_1 default for class 1 and 2 one ACK sent for each SEQ_CNT ACK_N Class 1 or 2 N=ACK sent by recipient for the support indicated during port login ACK_0 class 1 or 2 single ACK sent at end of sequence We could spend a lot more time discussing ACK s but there is little or no class 1 or 2 used in networks today and doubt if we will see any soon 169 Busy and Reject Port Reject P_RJT Fabric Reject F_RJT Transmitted by destination port or fabric in response to a specific data frame Applicable to only Class 1 and 2 Sent in reply to valid frames Transmitted by the receiver of the data frame with reason code Indicated that the corresponding data frame was NOT delivered to the ULP 170 Printed in USA.

86 Busy and Reject Busy sent by fabric if unable to deliver frame due to busy condition Busy sent by port if temporarily busy and unable to process a frame If F_BSY or P_BSY is sent, fabric or port give reason code Class 1 busy only allowed on the connection request Class 2 any frame may Rx busy Class 3, busy is not sent; If a frame can not be delivered it is discarded without notification 171 Flow Control and Credit Flow model Frames are moved: From one Buffer To another Buffer Frame Flow is: From the Source buffer and To the destination buffer Depending on the class of service Multiple intermediate buffers may be involved Applies to: All topologies 172 Printed in USA.

87 Flow Control and Credit Frame flow is controlled by the receiver Back-pressure mechanism ACK s class 1 and 2, RDY s class 3 Flow control is based on frame flow Which frames are flow controlled is dependent on class of service Receiver defined parameters during the login procedure Maximum frame size Number of buffers 173 Flow Control and Credit Receiver Establishes operating environment through login Size of buffers Number of buffers (credits) allocated to this transmitting port Pumps-up these credits By ACK s when buffer is available A receive Buffer is available after The frame was verified to be valid, no errors And the frame has been moved off the interface buffer 174 Printed in USA.

88 Flow Control and Credit Transmitter Keeps Credit maximum value Credit_Count Consumes one credit for each frame it transmits Credit_CNT = Credit_CNT 1 for each Data_Frame Tx Regenerates credit for each ACK Rx ed Credit_CNT = Credit_CNT + N Stops transmitting when Credit_CNT = Flow Control and Credit FC-2 defines two type of credit Buffer to Buffer (BB) End-to-End (EE) BB credit is the flow of connectionless traffic Over a LINK from Tx to Rx Class 2 and 3 Signal used = R_RDY EE credit is the flow on connection traffic Source to destination node Class 1 and 2 Signal used = ACK Both based on Credit Credit_CNT Differ in Frames controlled and acknowledgement signal 176 Printed in USA.

89 Flow Control and Credit Sequence Initiator Fabric Sequence Recipient TX Buf RX Buf TX Buf RX Buf R_RDY R_RDY ACK TX Buf RX Buf ACK R_RDY TX Buf RX Buf R_RDY BB_C BB_C EE_Credit 177 Class of Service Applicable to all fabric topologies Switched Point to point Arbitrated loop These three classes of service are Class 1 dedicated connection Class 2 connectionless multiplexed Class 3 datagram Delimiters used to set required class for a sequence 178 Printed in USA.

90 Class of Service SOF delimiter The required class of service along with basic sequence management are specified in the SOF delimiter of every frame The SOF delimiter dedicate basic link management functions within the fabric The SOF delimiter identifies basic Sequence management functions within the destination N_Port in the initial frame of the sequence and the last frame of the sequence EOF delimiter Last frame of a sequence is terminated by a special EOF Dedicated connections are removed by a special EOF 179 Class of Service Class 1 Dedicated connection service Connection oriented service between two N_Ports Frames received in order transmitted Guaranteed delivery with notification of non-delivery Guaranteed throughput Optional Intermix Can mix Class 2 and 3 frames if allowed 180 Printed in USA.

91 Class of Service Class 1 Requires explicit connection establishment SOF(C1) delimiter Requires explicit removal of connection ACK with EOF(DT) delimiter Once connection is established BSY and RJT will not occur Flow control Buffer to buffer on SOF(C1) frame: R_RDY End to end for all other data frames: ACK 181 Class of Service: Class 1 Flow Initiator SOF(C1) R_RDY Fabric Recipient Connection Requested R_RDY ACK Connection Established SOF(n1) SOF(n1) SOF(n1) ACK ACK ACK Conn Removed EOF(t) 182 Printed in USA.

92 Class of Service Class 2 Multiplexed connectionless service Connectionless oriented service between two N_Ports Order of frame reception not guaranteed Guaranteed delivery Notification of non-delivery No throughput guarantees Optional intermix 183 Class of Service Class 2 Multiplex on a frame-by-frame basis Between different destination N_Ports Among different sequences BSY and RJT may occur on any frame Flow Control Buffer-to-buffer for all frames: R_RDY End-to-end for all data frames: ACK 184 Printed in USA.

93 Class of Service: Class 2 Flow Initiator SOF(C2) Fabric Recipient R_RDY R_RDY ACK R_RDY SOF(n2) R_RDY SOF(n2) R_RDY R_RDY ACK ACK R_RDY R_RDY R_RDY 185 Class of Service Class 3 Datagram multiplexed connectionless service Connectionless oriented service between two N_Ports Order of frame reception not guaranteed Unacknowledged Delivery NOT guaranteed No throughput guarantees Optional intermix 186 Printed in USA.

94 Class of Service Class 3 Multiplex on a frame-by-frame basis Between different destination N_Ports Among different sequences BSY and RJT will not occur on any frame Flow control Buffer-to-buffer for all data frames: R_RDY 187 Class of Service: Class 3 Flow Initiator Fabric Recipient Data Frame R_RDY R_RDY Data Frame R_RDY R_RDY 188 Printed in USA.

95 EE Credit Switch EE_Credit NL_Node A EE_Credit NL_Node B Applies Only to Class 1 and Class 2 Frames for All Topologies EE_Credit 189 BB Credit Switch BB_Credit NL_Node A BB_Credit NL_Node B For All Class 2 and Class 3 Frames for All Topologies BB_Credit 190 Printed in USA.

96 FRAME PROCESSING 191 Tables The N_Port will keep the following information Available X_ID table Exchange context table Login table 192 Printed in USA.

97 Tables Available X_ID Table This table contains a list of available X_ID s Can be used for OX_IDs or RX_IDs A given implementation may choose to keep two tables one for OX_ID and RX_ID When a device driver sends a request to transmit a frame, a value will be taken for the OX_ID When a port receives a frame for a new exchange, a value will be taken for the RX_ID 193 Tables Exchange Context Table Each exchange ID points to a unique entry in the exchange context table Each entry contains the context and state information for the particular exchange Port_ID involved in exchange X_ID it assigned to exchange ULP and phase within the operation Data source or destination address Data frames transmitted or received (SEQ_CNT) ACK frames transmitted or received (SEQ_CNT) 194 Printed in USA.

98 Tables Login Table This table contains one entry for each port to which this port is logged in with Each entry contains service parameters and working EE_Credit count value 195 Data Frames: Putting It All Together Data Frame Transmission Request for a ULP Initiate some operation with a specific destination port Login process If you are not logged in, initiate login process Build logging table entry for destination port Assign OX_ID if needed Get a value from the available X_ID Table Build the exchange context table 196 Printed in USA.

99 Data Frames Data Frame Transmission (Cont.) Gather information Exchange context table Receive buffer size and destination port Login table Working credit count of destination port Set-up frame header Data frame transmission Segmentation process Credit management 197 Data Frames Transmit Request ULP passes a request to transmit a chunk of data to the N_Port Destination Port D_ID is made The N_Port must access the login table to determine the service parameters on the destination port Number of Rx buffers Value of the working credit count And the rest 198 Printed in USA.

100 Data Frames The Data Transmission ULP data chunk is moved in frames with the use of the sequence All within the context of the exchange A number of processes are involved Initialization of the frame header fields Segmentation and reassemble 199 First and Last Data Frames The first data frame of a sequence is identified by SOF(Ix) Delimiter, where x is the Class of Service The last data frame of a sequence is identified by F_CTL bit 19, End_SEQ= 1 A sequence consists of all data frames Starting at the SEQ_CNT for the first frame through the SEQ_CNT of the last frame 200 Printed in USA.

101 Sequence Processing Sequence Count ULP chunk of data is transmitted IN ORDER All Frames are sent in order Sequence_Count (SEQ_CNT) Frames are assigned sequentially increasing numbers as they are sent The receiving N_Port will use the SEQ_CNT to insure that Frames are reassembled in order and back in its original chunk 201 Sequence Initiator (SI) Sets F_CTL bit 23 0 If it is the exchange originator 1 If it is the exchange responder OX_ID and RX_ID set to assigned values RX_ID = FFFF if first sequence of exchange Routing field (R_CTL) set to 0000 to indicate FC-4 data frame Information category field of R_CTL set according to payload 202 Printed in USA.

102 Sequence Initiator: Frame Header Sequence ID (SEQ_ID) Any value select that is not used Sequence count (SEQ_CNT) Assign sequentially as frames are sent Starts with 0 on first frame of sequence Increments by 1 while sequence initiative is held Parameter Set to offset of the first byte of payload with respects to entire chunk Offset = 0 on first frame and 1 + for second and subsequent frames 203 Sequence Initiator: Frame Header Other important F_CTL bits Bit 23, exchange context Bit 21, first sequence Bit 20, last sequence Bit 19, end sequence Bit 16, sequence initiative Used to pass initiative to other device 204 Printed in USA.

103 Automatic Processes These processes are automatic and are performed by the protocol chip Segmentation and reassembly SEQ_CNT assignment Higher layers are unaware of these processes 205 ULP Processing The Upper Level Protocol (ULP) uses these fields Routing 0000 = FC-4 data frame Type 08 = SCSI/FCP Info category Identifies Specific Function of Payload 01 = Solicited Data 06 = Unsolicited Command 05 = Data Descriptor 07 = Command Status 206 Printed in USA.

104 ARBITRATED LOOP 207 Fibre Channel Arbitrated Loop (FC-AL) Maximum bandwidth: 100 MB/sec. (shared amongst all nodes on loop) 126 nodes max on loop Can be combined with switches L L L L L L Attaches NL_Ports Number of nodes on loop directly affects performance L L L Defined in it s own standard FC Fibre Channel Hub 208 Printed in USA.

105 Loop Advantages Low cost solution with copper transceivers Eliminates the need for a discrete fabric Fabric routing decision distributed around the loop Compatible with all FC- 0 variants Copper within a box Optical between boxes Self discovery procedure Simple additions to FC-PH 209 Loop Advantages Port bypass network High availability configurations possible Supports both public and private loops Provides access fairness 210 Printed in USA.

106 NL_Port N_Port Attaches to the physical transport media Provides the Fibre Channel control and protocol Provides the termination point for Fibre Channel Resides within the node NL_Port Provides all functionality on N_Port with additional function of the loop An NL_Port can function as a N_Port 211 FL_Port F_Port Attaches to the physical transport media at the edge of the switched fabric FL_Port The switched fabric port which attaches to a loop F_Port functionality with additional function of the loop G and GL Ports Will Do Both N and F 212 Printed in USA.

107 Private and Public Private Loop Contains no FL_Port Communications outside the loop via Fibre Channel is not possible Public Loop Contains an FL_Port Communications outside loop via Fibre Channel is possible Private Devices Devices on a public loop may be private, i.e. do not login 213 Addressing Arbitrated Loop Physical Address (AL-PA or PA) Assigned during the loop initialization (soft addressing) A unique 8 bit value 127 valid values Arbitrated Loop Destination Address (AL-PD or PD) The AL_PA used to identify the destination L_Port Target of a primitive signal or D_ID of a frame Arbitrated Loop Source Address (AL_PS or PS) The AL_PA used to identify the source L_Port Source of a primitive signal or S_ID of a frame 214 Printed in USA.

108 The Fabric Definition The entity that interconnects attached N_Ports Provides routing based on destination address Fabric may be: Point to point No routing required Switched Routing provided by the Switch Arbitrated loop Routing is distributed throughout the attached NL_Ports 215 Switched Fabric N_Port N_Port N_Port N_Port N_Port N_Port Switch Fabric 216 Printed in USA.

109 Loop Node Fabric Node = Arbitrated Loop Additional Function Node NL_Port NL_Port NL_Port LOOP NL_Port Node NL_Port Node NL_Port Node 217 Routing Process: Loop The routing function is distributed Each L_Port performs a portion of routing Routing is performed through out-of-band signaling using primitive signals Connection oriented independent of class of service Obtain ownership of the loop (Arbitration) Establish a connection (Open) Transfer frames (Data) Remove the connection (Close) Relinquish the loop 218 Printed in USA.

110 Processes and Procedures Initialization The process by which addresses are assigned and recovery is performed Arbitration The process by which an L_Port acquires ownership of the loop Open The process by which the L_Port which owns the Loop uses to select the L_Port to which it wants to communicate with Close The process by which the L_Port which owns the Loop releases control 219 Fill Words FC-PH defines two signals that may be transmitted between frames (when no other information is being transmitted) Idle R_RDY FC-AL defines several additional signals that may be transmitted between frames FC-AL defines the fill word to be ARB(F0) ARB(x) Idle 220 Printed in USA.

111 Primitive Signals and Sequences FC-AL Defined the Following Unique Signals and Sequences Primitive signals Arbitrate Open Close Mark Primitive sequences Port bypass enable Port bypass disable Loop initialization 221 Credits Buffers Loop Uses Same Credit Method as Previously Discussed But Also Has an Alternate Credit Model Alternate BB_Credit management requested during login When activated service parameter BB_Credit = number of buffers available when circuit is established The receiving L_Port shall transmit R_RDYs for the additional buffers at anytime when opened Used to pump up BB_Credit_CNT Transmitting L_Port Decrements BB_Credit by 1 for each data frame Tx Increments BB_Credit by 1 for each R_RDY Rx Stops transmitting when BB_Credit = Printed in USA.

112 Arbitrated Loop Initialization Procedure Purpose An L_Port will perform the loop initialization procedure to: Determine the Operating environment for the L_Port; Is this a loop? Acquire an address. AL_PA (Physical Address) Report that an error has been detected 223 Loop Commands Loop Initialization Procedure LIP Is an Ordered Set Command Bytes Payload Contents LISM Link Initialization Select Master 12 Command & WWN LIFA Link Initialization Fabric Assigned 20 Command & AL_PA bit map LIPA Link Initialization Previously Assigned 20 Command & AL_PA bit map LIHA Link Initialization Hard Assigned 20 Command & AL_PA bit map LISA Link Initialization Soft Assigned 20 Command & AL_PA bit map LIRP Link Initialization Report Position 132 Command & AL_PA Collect Position map LILP Link Initialization Loop Position 132 Command & AL_PA Distribute Position map 224 Printed in USA.

113 LIP: Initialization Procedure Phase A Start The Initialization Procedure LIP Start The Initialization Procedure Phase B Select Temporary Loop Master LISM FL_Port Wins if Present Otherwise Lowest WWN Wins Phase C AL_PA Mapping Phase LIFA, LIPA, LIHA, LISA Build the AL_PA bit Map in 4 Steps Phase D Reporting Phase LIRP Collect the AL_PA Position Map Phase E Distribute AL_PA Map Phase LILP Distribute the AL_PA Position Map Close 225 LIP: Phase A Loop Initialization Primitive Sequence Transmitted continuously by L_Port until it receives the same LIP configuration LIP (F7F7) the L_Port is attempting to determine if this is a loop and to acquire an AL_PA LIP (F8F7) the L_Port has detected a loop failure at its receiver prior to acquiring an AL_PA LIP (F8) the L_Port (AL_PS) had detected a loop failure at its receiver LIP (F7) the L_Port (AL_PS) has detected a performance degradation 226 Printed in USA.

114 LIP: Phase B Each L_Port will build the LISM with: AL_PA = 00 hex if FL_Port EF hex if NL_Port D_ID S_ID = 0000 hex + AL_PA Example (0000EF) = 0000 hex + AL_PA Payload = Command + WWN Current Fill Word = Idle Each L_Port will continuously transmit a LISM Normal flow control rules are not in effect during initialization 227 LIP: Phase B (Cont.) Each L_Port monitors its receiver Will continue to transmit LISM if Your AL_PA + WWN is less then received AL_PA + WWN Otherwise pass the received LISM You are temporary loop master If the device receives a LISM identical to the one transmitted FL_Ports always win; If two or more FL _Ports; Lowest WWN wins and the others go non-participating If no FL_Port the NL_Port with lowest WWN wins Loop master Current fill word would be ARB(F0) When ARB(F0) s are received, go to phase C 228 Printed in USA.

115 LIP: Phase C Loop Master Will Form the Initial Bit Map as Shown: Lowest AL_PA Word L Bit Position Where L = 1 Requesting F_Login of all NL_Ports Bit Position = 127 vector corresponding to valid AL_PA s Word 0 bit 30 = lowest number 00 hex Word 3 bit 0 = high number AL_PA value EF hex Set the bit = 1 that corresponds to it s Fabric Assigned AL_PA Highest AL_PA 229 LIP: Phase C Loop master will transmit the following three commands allowing an L_Port to choose a desired AL_PA LIFA bit map primed with initial value LIPA bit map primed with results of LIFA LIHA bit map primed with results of LIPA Loop master will then transmit the LISA command LISA bit map primed with results of LIFA allowing L_Ports which were unable to obtain their desired AL_PA to get a soft assigned AL_PA 230 Printed in USA.

116 LIP: Phase C Each NL_Port will Receive, possibly modify and retransmit the four Initialization Command frames Set the Current Fill Word (CFW) = ARB(F0) Modify the AL_PA bit map as follows Set one bit of the initialization command AL_PA bit maps based on history of AL_PA assignment If the bit map corresponding to a desired AL_PA has been set by an up-stream L_Port, this L_Port assumes a soft AL_PA by setting the first 0 bit=1 in the bit map of the LISA frame If no bit positions were available in the LISA bit map, the L_Port will remain in non-participating mode At most the bit map of one command will be modified by each L_port 231 LIP: Phase D The loop master will prime the AL_PA position map to: Byte 0 = 01 hex Byte 1 = it s AL_PA Bytes = FF hex Then transmit the LIRP with this position map Each NL_Port will: Increment the offset by one and store the offset Store its AL_PA at the offset Retransmit the updated LIRP frame The loop master will save the resulting loop position map 232 Printed in USA.

117 LIP: Phase E The loop master will transmit the LILP command with Payload = AL_PA position map Each NL_port will Save the loop position map Retransmit the LILP command When the loop master receives the LIILP command it will Transmit a CLS and go to monitoring state When each NL_Port receives a CLS they will Retransmit the CLS and go to monitoring state Initialization Complete 233 LIP: Summary A. LIP starts the initialization procedure B. Select a temporary loop master Lowest AL_PA WWN wins C. Build a AL_PA bit map Each L_Port indicates the AL_PA it selected in one of 4 requests by the loop master D. Collect a AL_PA position map Each L_Port reports its relative position from master and it s AL_PA E. Distribute the resulting AL_PA position map to each L_Port 234 Printed in USA.

118 Arbitration The process by which L_port request ownership of the loop based on primitive signals Ordered Set MSB LSB ARB(x) K28.5 D20.4 AL_PA AL_PA 235 Arbitration Loop Owner The current loop owner is responsible for Seeds the arbitration process with ARB(F0) Blocks propagation of the received ARB(x) until it relinquishes the loop Initiates a new arbitration window If ARB(F0) is received by setting current fill word = IDLE Fairness variables Access ARB_WON 236 Printed in USA.

119 Arbitration Process When a port is arbitrating it enters the arbitrating state The CFW is updated to the ports ARB(AL_PA) if the CFW is: 1. IDLE 2. ARB(F0) 3. ARB(FF) 4. Lower-priority ARB (higher value AL_PA) Arbitration occurs even if a loop circuit exists between another pair of ports Once a port starts arbitrating it Must continue to arbitrate until it wins Withdraw if it knows that another port is arbitrating 237 Fairness Access Fairness Ports with higher-priority AL_PA values could lock out lower priority ports When they ARB they will always win Lower Priority ports might never win Arbitration Access fairness limits how often a port can arbitrate This is done by not arbitrating the loop until all other ports on the loop that are arbitrating have won; This is called a fair port Access fairness is based on access not duration of usage Does not limit how long a port uses the loop Fairness is recommended by the standard but not mandatory FL_Ports may be unfair but NL_Ports should be fair 238 Printed in USA.

120 Fairness The fairness is controlled by the FC-AL fairness algorithm called a fairness window Window begins when the first port wins arbitration Ends when a port discovers that it was the last arbitrating port IDLE resets the fairness window The variables used are Access = 0 for fairness window open Access = 1 when NL_Port has won arbitration Fair ports can only arbitrate once per window After winning arbitration they wait for the end of the window before arbitrating again Unfair ports can arbitrate at anytime 239 Open If the Port Requires the Loop when It Wins ARB It sends an OPN(yx) or OPN(yy) y=destination port x=source port Full-Duplex establishes a point to point like circuit between the loop ports Half-duplex restricts open recipient to transmit link control frames only Cannot transmit device data frames Used by designs that can not support simultaneous data frames Tx and Rx 240 Printed in USA.

121 Open Selecting the Destination Port Is the intended destination port on same loop or connected via fabric switch? If the upper 16 bits of destination field (D_ID) are all zeros the port is on this private loop If the upper 16 bits of the source(s_id)are all zeros then the source port is a private port and can only talk to ports on same loop If the upper 16 bits of the D_ID are the same as the upper 16 bits of the S_ID then they are both on the same loop or both are public and attached to the same FL_Port If none of these are true, the destination port is not on the same loop and must be accessed via FL_Port 241 Opening a Port on Same Loop Open Originator inserts the destination AL_PD in the OPN The AL_PD is obtained from the low-order 8 bits of the destination address in the frame header This process can be entirely by hardware 242 Printed in USA.

122 Opening a Port Off the Loop Originator inserts AL_PD of the FL_Port 00 in the AL_PD field of the OPN The FL_Port is opened and frames are sent to the FL_Port FL_Port and fabric forwards the frames using the destination address field FL_Port can send to multiple destination ports on the loop during this OPN 243 SWITCH FABRIC OPERATION 244 Printed in USA.

123 Switch Model Port Connection Matrix Port Fabric Controller Port Connectionless Switch Matrix Port 245 Worldwide Names Each switch element is assigned a WWN at time of manufacture Each switch port is assigned a WWN at the time of manufacture During FLOGI the switch identifies the WWN in the service parameters of the accept frame Fabric port and Switch element These address assignments can then correlate each fabric port with the switch element 246 Printed in USA.

124 Switch Ports Four basic types of switch ports F_Port Uses NOS/LOS to attach to single N_Port FL_Port Uses LIP to attach 1 to 126 NL_Ports E_Port Uses NOS/LOS to interconnect switches (inter-link switch ISL) G_Port Uses NOS/LOS can be a F or E port 247 Fabric Addressing The 24 bit address is partitioned into 3 fields Device Area Domain This partitioning helps speed up routing Switch element assigns the address to N_Ports Address portioning is transparent to N_Ports 8 bits 8 bits 8 bits Switch Topology Model Switch Domain Area Device 248 Printed in USA.

125 Directory Server Repository of information regarding the components that make up the Fibre Channel network Located at address FF FF FC (Some readings call this the name server) Components can register their characteristics with the directory server An N_Port can query the directory server for specific information Query can be the address identifier, WWN and volume names for all SCSI targets 249 Directory Server Command Requests These Are Some of the More Used Commands Used to Query the Directory Server Get objects GA_NXT Get all next GFT_ID Get FC-4 types Register objects RFT_ID Register FC-4 types Deregister objects DA_ID Deregister all 250 Printed in USA.

126 Fabric Controller Each switch has a fabric controller Assigned address FF FF FD Every fabric controller in the fabric has the same address It is the N_Port within the switch Responsible for managing fabric, initialization, routing, setup and teardown of Class-1 connections Responsible to receive request and generate responses for the switch fabric Information must be consistent independent of which fabric controller responds to a request 251 Extended Link Services Extended link services provide a set of protocol functions used by the port to specify a function or service at another port Usually sent from N_Port to F_port to perform needed request The R_CTL field of the first word will be set to 0x22 to indicate an extend link service request Many ELS services will return a payload in response some have no reply 252 Printed in USA.

127 Extended Link Services Some of the more important and most used ELS commands are: FLOGI F_Port Login PLOGI N_Port Login FAN Fabric Address Notification PRLI Process Login PRLO Process Logout SCN State Change Notification SCR State Change Registration RSCN Registered State Change Notification 253 ELS: FLOGI FLOGI Fabric login Issued by N_Port to destination FF FF FE to Determine if fabric is present Establish a session with the fabric Exchange service parameters with the fabric FLOGI assigns N_Ports 24 bit address to N_Port or AL_PA to loop ports 254 Printed in USA.

128 ELS: PLOGI PLOGI N_Port login Established sessions between two N-Ports Required before upper level protocol operations can begin N_Port will register to the name server FF FF FC in fabric with all required login parameters N_Port will then query name server for other N_Ports on the fabric 255 ELS: PRLI PRLI Process Login Allows the FC-4 levels to exchange service parameters for communications between each other Process is protocol specific (type field) SCSI-3 FCP mapping requires PRLI 256 Printed in USA.

129 ELS: FAN FAN Fabric Address Notification Used in fabric loop attached topology Provides mechanism for FL_Port to notify NL_Ports of addresses and names of FL_Ports along with fabric name Allows NL_Ports to verify configuration following a loop initialization 257 ELS: SCN SCN State Change Notification Provides notification to ports of events that may effect logins or process logins to ports on the fabric SCN can be sent from N_Port to N_Port N_Port to fabric controller Fabric controller to N_Ports Notification may indicate login session is no longer valid Loss of signal (NOS, LOS, FLOGI) LIP has occurred SCN sent to fabric controller 258 Printed in USA.

130 ELS: RSCN RSCN Registered State Change Notification Similar to SCN but only sends change notice to those ports registered SCN did not define a registration method 259 Class_F Service Communications between switch elements use Class_F Service Unique SOF delimiter and normal EOF delimiter Used to pass control information within the switch Highest priority within switch Connectionless service Has no meaning outside switch, N_Port will discard if received 260 Printed in USA.

131 Inter-Switch Link The interconnection between switches is called the inter-switch link E_Port to E_Port Supports all classes of service Class 1, 2, 3, and switch to switch control traffic, class F FC-PH permits consecutive frames of a sequence to be routed over different ISL links for maximum throughput 261 Interswitch Links (ISLs) Inter-switch link (ISL) connects switches Fabric parameters must match on both switch otherwise link would not come up and fabric will be segmented 262 Printed in USA.

132 Principal Switch Selection Switch 1 Switch 3 Switch 2 Switch 4 Switch 5 Switch 6 Only one switch is designated principal switch in a fabric The switch with the lowest WWN becomes the principal switch originally Principal switch makes sure that no new switch is added to the fabric if it has a domain ID conflict with an existing switch in the fabric 263 Fabric Configuration Process The fabric configuration process enables a switch port to determine its operating mode, exchange operating parameters, and provides for distribution of addresses The process is summarized in the following steps Establish link parameters and switch port operating mode Principal switch selection Domain ID distribution Path selection 264 Printed in USA.

133 Fabric Configuration Stages Operation Establish Link Parameters and Switch Port Operating Mode Select Principle Switch Domain ID Acquisition Starting Condition Switch Port has achieved word synchronization BF or RCF SW_ILS transmitted or received Domain Address Manager had been selected Process The Switch Port attempts to discover whether it is an FL, F, or E port. Switch_Names are exchanged over all ISLs to select a Principle Switch, which becomes the Domain Address Manager Switch requests a Domain_ID from the Domain Address Manager Ending Condition Switch Port mode is known. If a Port is an E port, link parameters have been exchanged & Credit has been initialized. The Principle Switch is selected Switch has a Domain_ID Path Selection Switch has a Domain_ID Path selection (FSPF) is defined in the next section Switch is operations with routes established 265 Fabric Configuration: PS Selection A principal switch shall be selected whenever at least one inter-switch link (A link between two E_Port) is established The selection process chooses a principal switch, which is then designated to assign domain identifier to all the switches in the fabric, and any who join later the fabric later on 266 Printed in USA.

134 Fabric Configuration: PS Selection The principal switch selection can be triggered by anyone of the following events Switch boot and EFP Build Fabric (BF) Reconfigure Fabric (RCF) 267 Fabric Build Process When the switch first boots up and the first E_Port of a switch becomes operational, the switch starts 2xF_S_TOV timer and then sends out an exchange fabric parameters (EFP) from that port containing its own Destination ID (DoID) in the list trying to become Principle Switch (PS) The switch receiving the Exchange Fabric Parameter (EFP) replies with either ACCept or ReJecT after comparing the priority and WWN D (0) (FF, Dd) EFP EFP Domain_id Priority B (0) (FF, Bb) A (0) (FF, Aa) E_Port EFP WWN E_Port EFP EFP E_Port EFP EFP EFP E_Port C (0) (FF, Cc) 0x11 Record Len (0x10) Payload Len Reserved Priority Principal Switch WWN (Word 0) Principal Switch WWN (Word 1) Domain_ID record 0 Domain_ID record M 268 Printed in USA.

135 Fabric Build Process If the received information has a lower value, the switch keeps the received information and considers sending switch as potential principal switch and also consider that link to be potential upstream link At that point switch generates another EFP for all other links with the updated potential principal switch When 2x F_S_TOV expired, all switches in the fabric consider the information collected for the principal switch to be definitive; At that point the principal switch is responsible for assigning the Domain_IDs Domain_id WWN Potential Upstream port A (0) (FF, Aa) Priority E_Port E_Port SW_ACC SW_RJT E_Port E_Port SW_RJT SW_ACC Potential Upstream port EFP EFP EFP D (0) (FF, Bb) B (0) (128, Aa) C (0) (FF, Aa) 269 Fabric Configuration Details After the principal switch selection, the PS will change its priority to 0x02 (PS Priority) and then assign itself a domain ID and then the process of domain distribution starts The principal switch will initialize the process by sending Domain ID Assigned (DIA) SW_REQ out of all its E_Port The intermediate switch is actively involved in this process Each switch will reply back with Request Domain ID (RDI) To allow each switch to request for one or more domain ID The neighboring switch receiving RDI will be able to identify its downstream principal ISL Each switch can send many RDI but once the principal switch has granted the domain ID to the switch, the following RDI from the switch must request the same set of domain_id 270 Printed in USA.

136 Fabric Configuration Flows ID Assignment A B D DIA (SW_REQ) Upstream port E_Port D B A (1) (XX, Aa) E_Port DIA Upstream port E_Port E_Port DIA D (3) (FF, Aa) B (2) (FF, Aa) C (4) (FF, Aa) A SW_ACC SW_RJT RDI (SW_REQ) SW_ACC SW_RJT EFP (SW_REQ) Contains DoID list SW_ACC RDI (SW_REQ) DIA (SW_REQ) SW_ACC SW_RJT RDI (SW_REQ) SW_ACC SW_RJT EFP (SW_REQ) SW_ACC SW_RJT Contains DoID list EFP (SW_REQ) SW_ACC Contains DoID list SW_ACC 271 Fabric Configuration: The PS Battle After the principal switch selection and domain id assignment, all switches in the fabric will start two processes FC_ID assignment FSPF path selection When a new switch is added to the fabric, it will send out an EFP with its local value (I am PS); the fabric rejects that EFP and replies with DIA telling the new switch to send RDI; the RDI is then routed to the current PS If the new switch is part of another fabric (it also has a PS) then both fabric sends out an EFP and after comparing the DoID list the fabric enters one of the following states BF state: If the DoID list does not overlap RCF State: If the DoID list overlap Isolation: No auto-reconfigure state or RCF disabled 272 Printed in USA.

137 Fabric Configuration Disruptive/Non-Disruptive One of the following three conditions can trigger BF (non-disruptive) or RCF (disruptive) Two disjoints fabric are combined together A principal ISL fails (upstream or downstream) A switch with Domain_ID request for another Domain_ID Whenever a switch receives a BF/RCF, the switch starts F_S_TOV timer and enters the BF/RCF state; It forwards BF/RCF out of all E_ports except the incoming port (only once) and wait for the timer to expire When the timer expires, BF/RCF propagation state is left and principal switch selection begins BF is not a disruptive process RCF is a disruptive process 273 Fabric Configuration Distribution Propagation of BF or RCF Requests Switch Starts the Reconfig 274 Printed in USA.

138 Fabric Configuration: Reserve ID s N_ports and E_ports get one port ID; F ports don t get any ID s; FL ports in public AL gets 0x00 port ID Domain_ID Area_ID Port_ID Description Used during FLOGI AL_PA Private Loop NL_Port NonAL_PA Reserved FF 00-FF Reserved 01-EF 00-FF 00-FF N_Port & E_Port. Port ID=00 for FL port for public devices 255 address F0-FE 00-FF 00-FF Reserved FF 00-FA 00-FF Reserved FF FB 00-FF Multicast & Broadcast FF FC 00 Reserved FF FC 01-EF N_Port of domain controller. Port ID is the domain ID FF FC F0-FF Reserved FF FD-FE 00-FF Reserved FF FF 00-EF Reserved FF FF F0-FC,FF Well Known Address FF FF FD N_Port of fabric controller FF FF FE Fabric F_Port, Fabric Login database 275 Fabric Configuration: FSPF FSPF stands for fabric shortest path first Based on link state protocol Begins after domain ID assignment is completed Conceptually based on open shortest path first (OSPF) internet routing protocol Currently a standard defined in FC-SW Printed in USA.

139 Fabric Configuration: FSPF FSPF has four major components Hello protocol Replicated topology database A path computation algorithm Routing table update FSPF discovers the paths to switches using Domain Ids Each switch performs its own shortest path calculations 277 Fabric Configuration: FSPF For FSPF a domain ID identifies a single switch This limits the max number of switches that can support in the Fabric to 239 when FSPF is supported FSPF performs hop-by-hop routing FSPF supports hierarchical path selection Provides the scalable routing tables in large topologies 278 Printed in USA.

140 Fabric Configuration: FSPF Everyone says HELLO to their neighbor, on all initialized ISLs The neighbors say HELLO back, unless they are dead When the HELLO packet is received with both originator and recipient domain id, the two way communication is done and: The ISL is active The ISL may be available as a two-way path for frames 279 Fabric Configuration: Hellos Hello Protocol Point to Point Only Default Hello Int = 20 S Default HelloDead Int = 80 S 280 Printed in USA.

141 Fabric Configuration Link State Update and Ack A LSU(DB-A) B After a 2-way HELLO is established on a Link, each switch exchanges its entire database with its neighbor using the LSU service LSU(DB-B) LSA(DB-B) When the recipient of the LSU has processed the database, it sends back the LSA service LSA(DB-A) 281 Fabric Configuration Link State Record A LSU(LSR-A) B When the databases are in sync, each switch sends its LSR with the new link included using the LSU service LSU(LSR-B) The LSU is flooded to the entire fabric LSA(LSR-B) Each Switch retransmits the LSU by a mechanism called reliable flooding LSA(LSR-A) 282 Printed in USA.

142 Fabric Configuration Link cost is calculated based on baud rate of the link, plus an administratively set factor Link cost = S * (1.0625E12/baud rate) S is administrative factor defaults to 1 Ex: Link cost of 1G port = 1000 Path cost is the sum of the traversed link costs Lower metric more desirable 283 Fabric Configuration Path Selection (FSPF) Operation Summary Operation Perform initial HELLO Exchange Starting Condition The switch sending HELLO has a valid Domain_ID Process HLO SW_ILS frames are exchanged on the link until each switch has received a HELLO with a valid neighbor Domain field Ending Condition Two way communication has been established Perform Initial Database Exchange Running State Two communication has been established Initial Database Exchange has been completed LSU SW_ILS frames are exchanged containing the initial database Routes are calculated and set up within each switch. Links are maintained by sending HELLOs every Hello_Interval. Link databases are maintained by flooding link updates as appropriate Link State Databases have been exchanged FSPF routes are fully functional 284 Printed in USA.

143 FSPF Characteristics Uses FSPF as the routing algorithm FSPF routes traffic based on destination domain ID FSPF uses total cost as the metric to determine most efficient path Static routes can be applied 285 FSPF Characteristics Paths: Finds the shortest path to each domain, then programs the hardware routing tables Routes: Dynamically Round robin Statically Administrator can configure the route Automatically re-routes upon ISL going away and static routing will again take effect upon ISL return Automatic failover Fault detection 150 ms Self heals in 500 ms So, alternate route is live in 650 ms 286 Printed in USA.

144 Routing Software Configurable Parameters Link cost Static routes In Order Delivery (IOD) Timers (be careful) 287 What Is a Route and Path? Route FC ISL FC Path A route is map between the input and output E_port used to reach the next switch A path is a map through the topology between a source and destination 288 Printed in USA.

145 Selecting a Path FC Cost 500 Cost 250 Cost 250 FC Each inter switch link has a cost metric The cost of an ISL is related to the bandwidth The total cost of a path between two switches is the sum of the cost of all the traversed ISLs The path to a destination switch is the one with the minimum total cost More than one path can be selected (with the same cost) 289 ISL Oversubscription Multiple Nodes 1G 1G 1G Switch 1G ISL Switch Oversubscription occurs when more nodes can contend for the use of one ISL Oversubscription ratio is the number of different ports that contend for the use of one ISL This a 3:1 over subscription 290 Printed in USA.

146 FC ERROR MANAGEMENT 291 Timers Four different timers used Receiver-transmitter time-out (R_T_TOV) Error detect time-out (E_D_TOV) Resource allocation time-out (R_A_TOV) Connection request time-out (C_R_TOV) Used in Class 1 You will never see class one 292 Printed in USA.

147 Timers: R_T_TOV Receiver-Transmitter Time-out Used to time events at the link level Loss of synchronization Times Responses for link reset protocol Generally controlled in hardware for all link configurations Default value in FC Standard is 100ms 293 Timers: E_D_TOV Error Detect Time-out Timers for events and responses at the sequence level Missing ACK or R_RDY when buffer credit has reached zero Class 1 or 2 expects response from data frames N_Port logout Timer value is set at fabric login to accommodate the network environment for better scaling according to delivery time of frames Default is 10 sec 294 Printed in USA.

148 Timers R_A_TOV Resource Allocation Time-out Time-out value for how long to hold resources associated with a failed operation Needed to free shared resources for reuse Value to determine how long a port needs to keep responding to a link service request before an error is detected R_A_TOV is 2 times E_D_TOV Default setting in Pt to Pt is 20 sec and fabric is 120 seconds 295 Timers: CR_TOV Connection Request Time-out Determines how long the fabric can hold a class-1 request in the queue during connection establishment Allows for separation of the time in a stacked queue from the E_D_TOV; This separates queuing time from frame transit time Helps in controlling F_BSY issues 296 Printed in USA.

149 Recovery: Class 3 Errors in class 3 sequence can only be detected by the Sequence recipient because there are no ACKs or rejects in class 3 Class 3 SR will discard single or multiple frames until the exchange is terminated The upper level recovery may retransmit the entire Sequence or at least the sequence following the error detection 297 Recovery: Class 3 Errors a class 3 operation can detect: Out of order delivery and potential missing frame based on SEQ_CNT If missing frame is not Rx ed within E_D_TOV Indication of a new sequence when last frame of previous Sequence has not been received (in-order delivery set) Relative offset not in order with an order delivery set 298 Printed in USA.

150 Abort Sequence: ABTS ABTS protocol Used to terminate sequence or exchange Transmitted by the sequence initiator Can be requested by the sequence recipient by setting bits within the F_CTL of the ACK frame Same class of service delimiter as the sequence being aborted is used for ABTS frame 299 Timers Four different timers used Receiver-transmitter time-out (R_T_TOV) Error detect time-out (E_D_TOV) Resource allocation time-out (R_A_TOV) Connection request time-out (C_R_TOV) Used in Class 1 You will never see class one 300 Printed in USA.

151 Timers: R_T_TOV Receiver-Transmitter Time-out Used to time events at the link level Loss of synchronization Times Responses for link reset protocol Generally controlled in hardware for all link configurations Default value in FC Standard is 100ms 301 Timers: E_D_TOV Error Detect Time-out Timers for events and responses at the sequence level Missing ACK or R_RDY when buffer credit has reached zero Class 1 or 2 expects response from data frames N_Port logout Timer value is set at fabric login to accommodate the network environment for better scaling according to delivery time of frames Default is 10 sec 302 Printed in USA.

152 Timers R_A_TOV Resource Allocation Time-out Time-out value for how long to hold resources associated with a failed operation Needed to free shared resources for reuse Value to determine how long a port needs to keep responding to a link service request before an error is detected R_A_TOV is 2 times E_D_TOV Default setting in Pt to Pt is 20 sec and fabric is 120 seconds 303 Timers: CR_TOV Connection Request Time-out Determines how long the fabric can hold a class-1 request in the queue during connection establishment Allows for separation of the time in a stacked queue from the E_D_TOV; This separates queuing time from frame transit time Helps in controlling F_BSY issues 304 Printed in USA.

153 Recovery: Class 3 Errors in class 3 sequence can only be detected by the Sequence recipient because there are no ACKs or rejects in class 3 Class 3 SR will discard single or multiple frames until the exchange is terminated The upper level recovery may retransmit the entire Sequence or at least the sequence following the error detection 305 Recovery: Class 3 Errors a class 3 operation can detect: Out of order delivery and potential missing frame based on SEQ_CNT If missing frame is not Rx ed within E_D_TOV Indication of a new sequence when last frame of previous Sequence has not been received (in-order delivery set) Relative offset not in order with an order delivery set 306 Printed in USA.

154 Abort Sequence: ABTS ABTS can be sent under abnormal conditions End-to-end credits not required Sequence initiative not required Open sequence not required Maximum number of concurrent sequences allowed Unidirectional for class 1 connection The reply to an ABTS is a Basic_Accept 307 iscsi RFC Printed in USA.

155 Session Modules What is iscsi and what is the big picture? iscsi protocol Introduction The iscsi connection Security, data integrity and error recovery iscsi protocol details in-depth Simple iscsi connection flows Service location protocol for IP storage 309 What Is iscsi? A SCSI transport protocol that operates on top of TCP Encapsulates SCSI-3 CDBs (Control Descriptor Blocks) and Data into TCP/IP byte-streams (defined by SAM-2 SCSI Architecture Model 2) Allows IP hosts to access IP or Fibre Channel-connected SCSI targets Allows Fibre Channel hosts to access IP SCSI targets Standards status RFC 3720 (assigned May 2004) Major industry support (Cisco, IBM, EMC, HP, Microsoft) 310 Printed in USA.

156 Storage Technology SCSI Domain SCSI Device Initiator Service Delivery Subsystem SCSI Device Target Application Client Port Device Service Request Device Service Response Task Request Port Logical Unit 1 Device Server Task Response Task Manager To be functional, a SCSI Domain needs to contain a SCSI device that contains a target and a SCSI device that contains an Initiator 311 SAN, NAS, iscsi Comparison DAS SAN iscsi Appliance iscsi Gateway NAS Computer System Application Application Application Application Application File System File System File System File System File System Volume Manager SCSI Device Driver SCSI Bus Adapter Volume Manager SCSI Device Driver Fibre Channel HBA Volume Manager SCSI Device Driver iscsi Driver TCP/IP stack NIC Volume Manager SCSI Device Driver iscsi Driver TCP/IP stack NIC I/O Redirector NFS/CIFS TCP/IP stack NIC Block I/O File I/O SAN IP IP IP SCSI FC NIC TCP/IP stack iscsi layer Bus Adapter NIC TCP/IP stack iscsi layer Bus Adapter FC switch NIC TCP/IP stack File System Device driver Block I/O Adopted from IBM Redbook IP Storage Networking: IBM NAS & iscsi Solutions 312 Printed in USA.

157 IP Storage Networking IP storage networking provides solution to carry storage traffic within IP Uses TCP: A reliable transport for delivery Can be used for local data center and long haul applications Two primary protocols: iscsi IP-SCSI Used to Transport SCSI CDBs and Data within TCP/IP Connections IP TCP iscsi SCSI Data FCIP IP-SCSI Used to Transport SCSI CDBs and Data within TCP/IP Connections IP TCP FCIP FC SCSI Data 313 Initiator and Target Model for iscsi Initiator SCSI device which is capable of originating SCSI commands and task management requests Target SCSI device which is capable of executing SCSI commands and task management requests iscsi Gateway iscsi Gateway FC Target FC iscsi Initiator Target iscsi Initiator iscsi Target iscsi FC Initiator Target FC Initiator iscsi Target Mode iscsi Initiator Mode 314 Printed in USA.

158 iscsi Components iscsi is an end-to-end protocol iscsi has human readable SCSI device (node) naming iscsi includes the following base components: IPSEC connectivity security Authentication for access configuration Discovery of iscsi nodes Process for remote boot iscsi MIB standards 315 iscsi: Internet SCSI PDU The iscsi layer encapsulates the SCSI CDB into a iscsi Protocol Data Unit (PDU) and forwards it to the Transmission Control Protocol (TCP) layer It also extracts the CDB from an iscsi PDU received from the TCP layer, and forwards the CDB to the SCSI layer iscsi mapping provides the SCSI-3 command layer with a reliable transport The communications between the Initiator and target will occur over one or more TCP connections The TCP connections form a session and will carry the iscsi PDU s; the sessions are given an ID called a connection ID (CID); there are two parts of the ID, Initiator Session ID (ISID) and Target ID (TSID) and together make up an I_T nexus 316 Printed in USA.

159 iscsi Model Data Server Host Initiator SCSI CDB s carried by Fibre Channel Exchange and Sequences iscsi Target FC Storage Device FC Target Application Client requests data from LUN 1 Port Target Device Service Mapping Request Device Service Response LUN 1 = LUN 2 ge2 fc1 Port Logical Unit 1 Device Server Logical Unit 2 Device Server SCSI CDB s Carried in iscsi PDU s 317 iscsi Stack SCSI Applications (File Systems, Databases) SCSI Device-Type Commands SCSI Block Commands SCSI Stream Commands Other SCSI Commands SCSI Generic Commands SCSI Commands, Data, and Status SCSI Transport Protocols Parallel SCSI Transport FCP SCSI Over FC iscsi SCSI Over TCP/IP Layer 3 Network Transport TCP IP Layer 2 Network Parallel SCSI Interfaces Fibre Channel Ethernet 318 Printed in USA.

160 iscsi iscsi Packet Preamble Destination Address Sourced Port Destination Port Sequence Number Acknowledgment Number Offset Reserved U A P R S F Window Checksum Urgent Pointer Options and padding Source Address Well-known Ports: 21 FTP 23 Telnet 25 SMTP 80 http 3260 iscsi bytes Type IP TCP Data Opcode iscsi encapsulated FCS 4 Octet Opcode Specific Fields Length of Data (after 40Byte header) LUN or Opcode-specific fields Initiator Task Tag Opcode Specific Fields Data Field TCP Header 319 iscsi Naming and Discovery RFC 3721 Initiator and target require iscsi names Name is location independent iscsi node name = SCSI device name of iscsi device Associated with iscsi nodes, not adapters Up to 255 byte displayable/human readable string (UTF-8 encoding) Use SLP, or isns, or query target for names (sendtargets) Two iscsi name types: iqn iscsi qualified name eui Extended Unique Identifier (IEEE EUI-64) 320 Printed in USA.

161 iscsi Name Structure iscsi Name Structure Type. Unique String iqn Organization Subgroup Naming Authority or Type Date.. Naming Authority. String Defined by Organization Naming Authority iqn com.cisco.1234abcdef da232.betty iqn com.acme.storage.tape.sys1.xyz Date = yyyy-mm when Domain Acquired Reversed Domain Name Host Name eui Type. EUI-64 Identifier (ASCII Encoded Decimal) eui a425678d 321 iscsi Naming and Addressing Terms iscsi host name Name of computer iscsi initiator name (iscsi Node) Name created at iscsi driver load time on host system Initiator Target Session ID (SSID) One or more TCP connections between Initiator and target; This session ID is derived from iscsi host name, iscsi target name and TSID, ISID 322 Printed in USA.

162 iscsi Naming and Addressing Terms iscsi initiator address IP address on Initiator interface; Initiator can have multiple addresses Initiator port Also known as network portal IP address on initiator no port number assigned, again Initiator can have several network portals Target port Also known as network portal IP address + TCP port number on target interface There can be more then one target interface 323 iscsi Naming and Addressing Terms iscsi target name Used to identify multiple SCSI targets behind a single IP address+port; This name is globally unique Initiator session ID This is an initiator-defined session identifier; It will be the same for all connections within a session; An iscsi initiator port is uniquely identified by the value pair (iscsi Initiator Name, ISID) Target session ID Target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator 324 Printed in USA.

163 iscsi Naming and Addressing Terms iscsi network entity Client It is a combination of the following: iscsi initiator iscsi host iscsi initiator address Initiator port (network portal) iscsi network entity Server Is a combination of the following: iscsi target name Target port (network portal) Initiator target session (SSID) 325 iscsi Naming and Addressing Terms iscsi Node iscsi Initiator or iscsi Target; There can be one or more iscsi nodes in a network entity iscsi node will equal iscsi initiator name iscsi target name 326 Printed in USA.

164 iscsi Naming and Addressing Terms Portal Group Groups multiple TCP connections across the same session that is is sent across multiple portals The portal groups are identified by a portal group tag ( ) One or more portal groups can provide a path to the same iscsi node (target node or initiator node) SendTargets requires portal group tag 327 iscsi Discovery Methods Small networks Static configuration, initiators and targets SendTargets command makes configuration easier Medium-sized networks Service Location Protocol (SLP multicast discovery) Large-sized networks isns (internet storage name service) Includes soft zone domains Includes database for ongoing management 328 Printed in USA.

165 iscsi Architecture Network Entity (iscsi Client) iscsi Node (Initiator) Network Portal Network Portal Network Portal Network Portal iscsi Node (Target) iscsi Node (Target) Network Entity (iscsi Server) 329 iscsi Architecture IP Network Network Portal Network Portal Network Portal Portal Group 1 Portal Group 2 iscsi Session (Target Side) iscsi Name + TSID=2 iscsi Session (Target Side) iscsi Name + TSID=1 iscsi Target NodeNode (within Network Entity) 330 Printed in USA.

166 iscsi Session Model An iscsi session exists between a single iscsi initiator (host) and a single iscsi target (iscsi router) An iscsi session consists of one or more iscsi (TCP) connections Login phase begins each connection Deliver SCSI commands in order iscsi Session iscsi (TCP) Connection TCP/3260 TCP/3260 TCP/3260 iscsi Routing Instance iscsi Storage Router 331 iscsi Session Images Across all connections within a session, an initiator sees one target image The target image would represent all identifying elements such as LUN s A target also sees one initiator image across all connections within a session 332 Printed in USA.

167 Put It All Together for iscsi iscsi Driver, Storag e NIC iscsi Node Configuration s will make the connections between storage and iscsi Initiator iscsi Target name configured on iscsi Device mike.cisco.com disk.cisco.com.stor ISID SSID TSID IP iscsi Host Name iscsi Initiator address Initiator Port TCP Connection Target-1 Target-2 Target-3 iscsi Initiator Name Target Port These Network Portals listens for iscsi connections on WKP 3260 iscsi Network Entity-Server Target-4 Storage Systems Target iscsi Connections and SCSI Phases A SCSI command and its associated data and status phase exchanges must traverse the same TCP connection Linked SCSI commands can traverse separate TCP connections for scalability iscsi Session iscsi (TCP) Connection 1 SCSI Command (1) (Read) SCSI Data (1) SCSI Status (1) SCSI Command (1) (Write) SCSI Data (1) SCSI Status (1) iscsi (TCP) Connection 2 Linked SCSI Commands iscsi Routing Instance iscsi Storage Router 334 Printed in USA.

168 iscsi Connection Session Session Can Process SCSI Commands and Data after Login Is Complete iscsi Session has four phases Initial login phase Security authentication phase Operational negotiation phase Full featured phase 335 iscsi Session Establishment Login Begins with the First Connection Initial login phase Initiator sends login with text strings for InitiatorName, TargetName, and authentication options (which are then selected by the target) Security authentication phase Authentication text exchanges (ID, password, certificates, etc) Operational negotiation phase Each side (initiator and target) negotiate the supported options using Keyword=value, or Keyword=value,value,value Amount of unsolicited buffer Types of data delivery Solicited, unsolicited, immediate, etc Full featured phase Can carry SCSI CDBs/data, task management, and responses 336 Printed in USA.

169 iscsi Session Key Points Sessions: iscsi Session = a group of TCP connections linking an initiator with a target (i.e., can be one or more connections) NOTE: A TCP connection that is part of an iscsi session will only be used to carry iscsi traffic The iscsi initiator and target use the session to communicating iscsi commands, control messages, parameters, and data to each other TCP connections can be added and removed from a session using the iscsi Login/Logout commands 337 iscsi Sessions During session establishment, the target identifies the SCSI initiator port (the I in the I_T nexus ) through the value pair (InitiatorName, ISID) Any persistent state (e.g., persistent reservations) on the target associated with a SCSI initiator port is identified based on this value pair Any state associated with the SCSI target port (the T in the I_T nexus ) is identified externally by the TargetName and portal group tag and internally in an implementation dependent way 338 Printed in USA.

170 iscsi Connection Allegiance For SCSI commands that require data transfer, the data phase and status phase must be sent over the same TCP connection used by the command phase Consecutive commands that are part of a SCSI task may use different connections within the session (linked commands) Connection allegiance is strictly per-command and not per task Multiple connections allow the iscsi session to be scaled across multiple links/devices 339 iscsi Connection Termination Session may end with logout or I/O error causing dropped connection TCP connections are closed through normal methods i.e. TCP FINs Graceful shutdowns can only occur when no outstanding tasks are on the connection and not in full-feature phase Termination of connection abnormally may require a recovery method by logout request for all connections; This prevents stale iscsi PDU s being received after going down Logout can also be issued by the target through asynchronous message PDU 340 Printed in USA.

171 iscsi Security Two types of security IPSec secures TCP/IP nodes; setup at TCP/IP startup before iscsi login Session authentication via IKE (Internet Key Exchange) Packet by packet authentication (also provides Integrity) Privacy via encryption (also provides Integrity) See SEC-IPS iscsi techniques (done/setup during iscsi Login) Authentication (ensures nodes are authorized to use the iscsi target node) may use SRP, Chap, or Kerberos 341 Challenge Handshake Authentication Protocol In-band initiator-target authentication IP-SEC is not assumed No clear text password accepted Compliant iscsi initiators and targets MUST implement the CHAP (RFC1994) Implementations MUST support use of up to 128 bit random CHAP secrets 342 Printed in USA.

172 iscsi Security Various levels of security can fit different topologies Examples: Secure main floor No security Campus LAN iscsi authentication and CRC32c (digests) Remote private WAN IPSec with session/packet authentication Remote internet WAN IPSec with privacy encryption 343 iscsi Data Integrity Basic level of end-to-end data integrity can be reasonably handled by TCP using the standard checksum iscsi CRC32c digest checks for Integrity beyond TCP/IP XOR checksum a) Header digest b) Data payload digest 344 Printed in USA.

173 Digests (Checksums) Optional header and data digests protect the integrity of the header and data, respectively; The digests, if present, are located, respectively, after the header and PDU-specific data, and cover the proper data and the padding bytes The existence and type of digests are negotiated during the login phase The separation of the header and data digests is useful in iscsi routing applications, in which only the header changes when a message is forwarded; In this case, only the header digest should be recalculated Digests are not included in data or header length fields A zero-length data segment also implies a zero-length data-digest 345 Error Recovery Two Considerations for Errors An iscsi PDU may fail the digest check and be dropped, despite being received by the TCP layer; The iscsi layer must optionally be allowed to recover such dropped PDUs A TCP connection may fail at any time during the data transfer; All the active tasks must optionally be allowed to be continued on a different TCP connection within the same session 346 Printed in USA.

174 Error Recover: iscsi Initiator A. NOP-OUT to probe sequence numbers of the target B. Command retry C. Recovery R2T support D. Requesting retransmission of status/data/r2t using the SNACK facility E. Acknowledging the receipt of the data F. Reassigning the connection allegiance of a task to a different TCP connection G. Terminating the entire iscsi session to start fresh 347 Error Recover: iscsi Target A. NOP-IN to probe sequence numbers of the initiator B. Requesting retransmission of data using the recovery R2T feature C. SNACK support D. Requesting that parts of read data be acknowledged E. Allegiance reassignment support F. Terminating the entire iscsi session to force the initiator to start over 348 Printed in USA.

175 Error Recover Classes Within a command (i.e., without requiring command restart) Within a connection (i.e., without requiring the connection to be rebuilt, but perhaps requiring command restart) Connection recovery (i.e., perhaps requiring connections to be rebuilt and commands to be reissued) Session recovery 349 Error Levels Level determined during logon text negotiation Error recovery level is proposed by an originator in a text negotiation 350 Printed in USA.

176 iscsi PROTOCOL DETAILS IN-DEPTH 351 iscsi Key Points Tasks: A linked set of SCSI commands One and only one SCSI command at a time can be processed within any given iscsi task Initiator Task Tag (ITT) and Target Transfer Tag (TTT) Initiator tags for all pending commands must be unique initiator-wide SCSI Data PDUs are matched to their corresponding SCSI commands using tags specified in the protocol ITT for unsolicited data TTT for solicited data 352 Printed in USA.

177 iscsi Key Points Solicited or unsolicited messages: Initiator to target User data or command parameters will be sent as either solicited data or unsolicited data Solicited data is sent in response to ready to transfer (R2T) PDUs Unsolicited data can be part of an iscsi command PDU ( Immediate data ) or an iscsi data PDU The maximum size of an individual data PDU or the immediate part of the initial unsolicited burst may be negotiated during login Target to initiator Ready to transfer (R2T) message to Initiator, requesting data for a write command Command responses Asynchronous messages (SCSI and iscsi) describing an unusual or error event 353 iscsi Numbering iscsi uses command and status numbering Command numbering Session wide and is used for ordered command delivery over multiple connections within a session; It can also be used as a mechanism for command flow control over a session Status numbering per connection and is used to enable recovery in case of connection failure Fields in the iscsi PDUs communicate the reference numbers between the initiator and target During periods when traffic on a connection is unidirectional, iscsi NOP PDUs may be issued to synchronize the command and status ordering counters of the initiator and target 354 Printed in USA.

178 SCSI Command Numbering and Acks within iscsi Initiator and target device have three sequence number registers per session CmdSN Current command sequence number; Sent by initator ExpCmdSN Expected command by the target; Sent to the initiator by the target to acknowledge CmdSN; Can be used to ACK several sequences MaxCmdSN Maximum number target can receive in its queue; Can be sent to Initiator from target to adjust queue size 355 SCSI Command Numbering and Acks within iscsi iscsi supports ordered command delivery within the session Command-Sequence-Number (CmdSN) is assigned by initiator and carried in the iscsi PDU CmdSN starts at iscsi login CmdSN not assigned to data-out (DataSN used) Immediate delivery does not advance CmdSN iscsi must deliver commands to target in order of CmdSN and will not increment until executed state by target 356 Printed in USA.

179 SCSI Status Numbering and Acks within iscsi Status Sequence Number (StatSN) is used to number responses to the Initiator from the target ExpStatSN is sent by Initiator to acknowledge status Status numbering starts after Login; During login there can be only one outstanding command per connection 357 Initiator iscsi OPcodes 0x00 NOP (No operation, used as ping to target gateway) 0x01 SCSI command (Indicates encapsulated iscsi packet has a SCSI CDB for target device) 0x02 SCSI task management command 0x03 iscsi login 0x04 text command 0x05 SCSI data-out (Write data to target device) 0x06 iscsi logout 0x10 SNACK (Request retransmission from target) 0x1c-0x1e Vendor specific codes 358 Printed in USA.

180 Target iscsi OPcodes 0x20 NOP-In (No operation in, used for ping response from target 0x21 SCSI response (Indicates encapsulated iscsi packet has status or from target device) 0x22 SCSI task management response 0x23 login response 0x24 text response 0x25 SCSI data-in (Read data from target) 0x26 logout response 0x31 Ready to transfer (Sent to initiator from target to indicate it is ready to receive data) 0x32 async message (Message from target to indicate special conditions) 0x3c-0x3e vendor specific codes 0x3f reject 359 iscsi PDU s Several different types of iscsi PDUs used, each of the different iscsi Operation Codes (Opcodes) determine what iscsi PDU to use; Some of the more used PDUs are: Login and logout PDU Command and response PDU Data-In and data-out PDU 360 Printed in USA.

181 iscsi Login PDU If Set to 1 = Recovery from Failed Connection If Set to 1 Indicates Initiator Is Ready to Transit to Next Stage Initiator ID for This Connection Current Stage/Next Stage 0 Security Negotiation 1 Login Operational Negotiation - 3 Full Feature Phase Unique ID for This Connection Initiatior May Provide Initial Text Parameters in This Area 361 iscsi Login Login Phase used to: Enable TCP connection (Target listens on well known port) Authentication (CHAP) Negotiate session parameters Open security protocols Mark the TCP connection as a iscsi session and assign IDs 362 Printed in USA.

182 iscsi Text Mode During Login Some Sessions or Connection Parameters May Be Negotiated in a Text Format list = values sent in order of preference Example of values can be: MaxConnections=< > T or I Sendtargets=all I only Targetname=<iSCSI-Name> T or I SessionType=<Discovery Normal> I only Others addressed later in slides (see RFC) 363 iscsi Full Feature Phase A Connection Is in Full Feature Mode after a Completed Login iscsi PDUs can be sent PDUs must flow over same connection as login Size of PDU is negotiated during login 364 Printed in USA.

183 Data Sequencing within iscsi The iscsi PDUs used for data input and output are the 0x05 iscsi command and the 0x25 iscsi command, along with R2T (0x31 ready to transfer) DataSN is a number field and advances by 1 for each input (read) and output (write) Targets will operate in two modes, solicited (R2T) or unsolicited (non-r2t) Target operating in R2T mode can only receive solicited data from the initiator R2TSN advances by one for each received R2T during the data transfer The DataSN and R2TSN fields are for the initiator to detect missing data 365 Data-Out PDU Final Bit Say This Is the Last PDU of a Sequence LUN Number for Data Data Segment Length Based on Capabilities Exchange 366 Printed in USA.

184 Data-In PDU Final Bit say this is the last read of a sequence Acknowledge Bit used when error recovery level is 1 or higher Flags valid when S bit is set tells how to read Residual Count Status bit tells that there is meaningful data in the StatSN, Status, and Residual Count fields 367 iscsi Read Command Example 1. Initiator sends iscsi command PDU (CDB=Read) 2. Target sends iscsi data-in PDU(s) 3. Target sends iscsi response PDU Notes: Solicited data via read command PDU (Initiator requests data from the target) Target may satisfy the single read command with multiple iscsi data read PDUs (PDUs can be out-of-order) Command is not complete until all data and status is received by the initiator Good status can be sent within the last iscsi data-in PDU All iscsi data-in PDUs and the response PDU will be delivered on the same TCP connection that the command was sent on All data-in PDUs will carry the same value in the ITT field 368 Printed in USA.

185 SCSI Command PDU Lets Target Know if More Data Is to Follow along with Expected Data Transfer Length Task Attributes See RFC for Detailed Meaning 16 bytes of SCSI CDB, R=1 If the Command Is Expected to Input Data W=1 If the Command Is Expected to Output Data CRC If Capabilities Required This Some SCSI Commands Have Additional Data and This Field Is Used for the Accompanied Data 369 SCSI Response PDU SCSI Status per SAM2 Ox00 = Command Completed at Target 0x01 = Target Failure 0x08 0xff = Reserved for Vendor Response CRC Check Sums 370 Printed in USA.

186 SCSI Status and Response Fields for iscsi OpCode 0x21 The status field of the iscsi PDU is used to report status of the command back to the initiator The specific status codes are documented in the SCSI architectural model for the device Response field contains the iscsi codes that are mapped to the SAM-2 response 371 Ready to Transfer PDU When the initiator has sent a SCSI write command to the target the target can specify the blocks be delivered in a convenient order; This information is passed to the initiator in the R2T PDU Allowing an initiator to write data to a target without a R2T is agreed upon during login The target may send several R2T PDUs and have several data transfers pending if allowed by the initiator 372 Printed in USA.

187 Task Management Functions to provide the initiator a way to control management of the target device Abort the TASK Clear allegiance Logical reset Target reset Each of these and more are broken down in detail in the iscsi RFC 373 SACK, NOP-IN, NOP-OUT SACK Optional Used to request retransmission of numbered responses, data or R2T PDUs from the target NOP-IN Sent by a target as a response to a NOP-Out, as a ping to an initiator Or a means to carry a changed ExpCmdSN and/or MaxCmdSN if there is no other PDU to carry them for a long time NOP-OUT Used by Initiator as a ping command, to verify that a connection/session is still active and all its components are operational Used to confirm a changed ExpStatSN if there is no other PDU to carry it for a long time 374 Printed in USA.

188 Message Synchronization and Steering Steering of iscsi out of order TCP segments into pre-allocated buffers instead of temporary buffers To decrease reassembly time Not needing to rely on message length information Provides a synchronization method using fixed interval markers telling where the start of the next iscsi PDU is in the buffer Optional for iscsi RFC 375 List of Negotiated Parameters Prior to Going into Full Feature Mode Header Digest Data Digest Max Connections Send Targets Target Name Initiator Name Target Alias Initiator Alias Target Address Target Portal Group Tag Initial Ready 2 Transfer Immediate Data Max Rec Data Segment Length Max Burst Length First Burst Length Default Time 2 Wait Default Time 2 Retain Max Outstanding R2T Data PDU In-order Data Sequence In-order Error Recovery Level Session Type 376 Printed in USA.

189 Standards: Where to Find Details T10 Technical committee Technical committee of the National Committee on Information Technology Standards (NCITS), deals with the storage devices T11 Technical committee Technical committee of the NCITS, deals with the physical interface and transport level 377 SIMPLE ISCSI CONNECTION FLOWS EXAMPLE OF DISCOVERY SESSION WITH CHAP 378 Printed in USA.

190 iscsi Flows Initiator TCP port 1026 (Random) Discovery Session Establish Initial TCP Session Phase 0X03 Command Login Key Values Are Sent, InitiatorName, InitiatorAlias, SessionType=Discovery, AuthMethod=CHAP/none, HeaderDigest, DataDigest 0X23 Login Response Status= Accept Login (0X0000), Keyvalues Are Sent, AuthMethod=CHAP, HeaderDigest=none, DatDigest=none 0X03 Command Login Key Values Sent, InitiatorName, InitiatorAlias, SessionType=Discovery, CHAP_A=5 (CHAP with MD5) 0X23 Login Response Status=Accept Login, KeyValues CHAP_A, CHAP_I & CHAP_C Target TCP Port 3260 This Device Has Already Initialized Onto the Fibre Channel iscsi Driver 379 iscsi Flows Initiator TCP port 1026 (Random) Discovery Session 0X03 Command Login Key Values Are Sent, InitiatorName, InitiatorAlias, SessionType=Discovery, CHAP_R, CHAP_N 0X23 Login Response Final PDU in Sequence, Status= Accept login (0X0000) End of Authentication Phase Start of Parameter Negotiation Phase for Discovery Session 0X03 Command Login Key Values Sent, InitiatorName, InitiatorAlias, SessionType=Discovery, Negotiate Session Parameters 0X23 Login Response Status=Accept Login, Negotiate Session Parameters Target TCP Port 3260 iscsi Driver 380 Printed in USA.

191 iscsi Flows Initiator TCP port 1026 (Random) Discovery Session 0X04 Text Command SendTargets=all 0X24 Login Response Final PDU in Sequence, KeyValue=TargetName (iqn number along with target name configured on iscsi Target) Target TCP Port 3260 TCP port 1027 (random) Target Session #1 Note the Addition of Another TCP Session iscsi Driver Start of Target Session Authentication and Target Session Parameter Negotiation Establish TCP connection for target 0X03 Command Login Key Values sent, InitiatorName, InitiatorAlias, SessionType=Normal, TargetName, AuthMethod=CHAP,none 0X23 Login Response Status=Accept Login, AuthMethod=CHAP 381 iscsi Flows Initiator TCP Port 1027 (Random) Target Session #1 0X03 Command Login Key Values are sent, InitiatorName, InitiatorAlias, SessionType=Normal, TargetName, CHAP_A=5 0X23 Login Response Status=Accept Login, KeyValues CHAP_A, CHAP_I & CHAP_C 0X03 Command Login Key Values are sent, InitiatorName, InitiatorAlias, SessionType=Normal, CHAP_R, CHAP_N 0X23 Login Response Status=Accept Login 0X03 Command Login Target TCP Port 3260 iscsi Driver Key Values sent, InitiatorName, InitiatorAlias, SessionType=Normal, TargetName, Negotiate session Parameters 382 Printed in USA.

192 iscsi Flows Initiator TCP Port 1027 (Random) Target Session #1 0X23 Login Response Status=Accept Login, Negotiate session Parameters Target TCP Port X01 iscsi Command SCSI Inquiry CDB 0X12 0X25 iscsi Data-in (read) iscsi Driver 383 FCIP CONCEPTS 384 Printed in USA.

193 Agenda What FCIP Is About The Standards Fibre Channel T11 Standards IETF IPS Working Group Drafts Understanding FCIP Protocol Relationships to Other SCSI Transport Technologies 385 FCIP: Fibre Channel over IP FCIP provides a standard way of encapsulating FC frames within TCP/IP, allowing islands of FC SANs to be interconnected over an IP-based network TCP/IP is used as the underlying transport to provide congestion control and in-order delivery of error-free data FC frames are treated the same as datagrams It is not ifcp, mfcp, IPFC, iscsi transports or extended FC fabric 386 Printed in USA.

194 FCIP Design FC Server FC Tape Library FC Server FC Tape Library FSPF Routing Backbone FSPF Routing Backbone FC Switch FC Switch FC Server Fiber Channel SAN FC JBOD FC Switch FC Switch FCIP Tunnel IP Network Tunnel Session FC Switch FCIP Tunnel FC Switch IP Services Available at Aggregated FC SAN Level Fiber Channel SAN FC Server FC JBOD FC Switch FC Switch 387 Four (4) Specifications Define Basic FCIP ANSI: FC-SW-2 describes the operation and interaction of Fibre Channel switches, including E_Port, B_Port and fabric operation FC-BB-2 is a mapping that pertains to the extension of Fibre channel switched networks across a TCP/IP network backbone and defines reference models that support E_Port and B_Port IETF IPS working group: Fibre channel over TCP/IP covers the TCP/IP requirements for transporting Fibre Channel frames over an IP network FC frame encapsulation defines the common Fibre Channel encapsulation format 388 Printed in USA.

195 ANSI: FC-SW-2 Standard E_Ports are used at both ends of an Inter Switch Link (ISL) E_Ports forward user traffic (storage data) and control information (class F SW_ILS frames containing FSPF, zone exchanges, etc.) FC-SW-2 defines fabric merge procedures (Domain_ID assignment, zone transfers, etc.) FC-SW-2 also defines FSPF 389 ANSI: FC-SW-2 Essentials (Recap) E_Ports provide switch-to-switch connectivity E_Ports negotiate parameters such as: ELP Exchange Link Parameters ESC Exchange Switch Capabilities FSPF is enabled over E_Ports only Separate fabrics can be merged over E_Ports Zoning information is exchanged over E_Ports 390 Printed in USA.

196 IETF FCIP: Fibre Channel Over IP Each interconnection is called a FCIP link and can contain one (1) or more TCP connection(s) Each end of a FCIP link is associated to a virtual ISL link (VE_Port or B_Access Portal) VE_Ports communicate between themselves just like normally interconnected E_Ports by using SW_ILS: ELP, EFP, ESC, LKA, BF, RCF, FSPF, etc. B_Access portals communicate between themselves by using SW_ILS: EBP, LKA The result (when all goes well ) is a fully merged Fibre Channel fabric between FC switch SAN s 391 IETF FCIP IETF draft standard that allows IP connectivity to link Fibre Channel storage area networks across WANs Two methods can be used 1) Similar to Cisco STUN Nailed up tunnel 2) Similar to DLSW Dynamic peering method We will visit the details of each in later slides draft-ietf-ips-fcovertcpip Draft 12 is current, will RFC Jan/Feb Printed in USA.

197 FCIP Architecture Model FCIP Link FCIP FCIP Key: FC-0 Fibre Channel Physical Media Layer FC-1 Fibre Channel Encode and Decode Layer FC-2 Fibre Channel Framing and Flow Control Layer FC-2 FC-1 FC-0 TCP IP LINK PHY TCP IP LINK PHY FC-2 FC-1 FC-0 TCP Transmission Control Protocol IP Internet Protocol LINK IP Link Layer PHY IP Physical Layer SAN TCP/IP Network SAN 393 FCIP End-station addressing, address resolution, message routing, and other fundamental elements of the network architecture remain unchanged from the Fibre Channel model, with IP introduced exclusively as a transport protocol for an inter-network bridging function IP is unaware of the Fibre Channel payload and the fibre channel fabric is unaware of IP // Ethernet Header IP Checksum TCP FCIP FCP SCSI Data CRC // 394 Printed in USA.

198 FCIP FCIP only supports class 2, class 3, class 4, and class F frames No FC primitive signals or primitive sequences supported Physical signal sets used by FC ports to indicate events, i.e. NOS, OLS, LR IP transport is transparent to Fibre Channel topology 395 Understanding FCIP Terms FC end node A Fibre Channel device that uses the connection services provided by the FC fabric FC entity The Fibre Channel specific functional component that combines with an FCIP entity to form an interface between an FC fabric and an IP network FC fabric An entity that interconnects various Nx_Ports attached to it, and is capable of routing FC frames using only the destination ID information in a FC frame header FC fabric entity A Fibre Channel specific element containing one or more Interconnect_Ports (see FC-SW-2) and one or more FC/FCIP entity pairs FC frame The basic unit of Fibre Channel data transfer FC frame receiver portal The access point through which an FC frame and time stamp enters an FCIP data engine from the FC entity FC frame transmitter portal The access point through which a reconstituted FC frame and time stamp leaves an FCIP data engine to the FC entity FC/FCIP entity pair The combination of one FC entity and one FCIP entity 396 Printed in USA.

199 Understanding FCIP Terms (Cont.) FCIP data engine (FCIP_DE) The component of an FCIP entity that handles FC frame encapsulation, de-encapsulation, and transmission FCIP frames through a single TCP connection FCIP entity The entity responsible for the FCIP protocol exchanges on the IP network and which encompasses FCIP_LEP(s) and FCIP control and services module FCIP frame An FC frame plus the FC frame encapsulation header, encoded SOF and encoded EOF that contains the FC frame FCIP link One or more TCP connections that connect one FCIP_LEP to another FCIP link endpoint (FCIP_LEP) The component of an FCIP entity that that handles a single FCIP link and contains one or more FCIP_DE s Encapsulated frame receiver portal The TCP access point through which an FCIP frame is received from the IP network by an FCIP data engine Encapsulated frame transmitter portal The TCP access point through which an FCIP frame is transmitted to the IP network by an FCIP data engine FCIP special frame (FSF) A specially formatted FC frame containing information used by the FCIP protocol 397 FCIP Diagram FC Entity Virtual ISL FC/FCIP Entity Pair FCIP Entity VE_Port FCIP_LEP DE DE FCIP Link End Point FCIP Data Engine FCIP Link FC Frame Receiver Portal FCIP Data Engine (Detail) FCIP Frame TX RX Portal VE_Port FCIP_LEP TX RX Dynamic CONNECTION PORT for FCIP Connections TCP Ports WKP = 3225 IP Address = Ethernet Gigabit/WAN Interface Non Dynamic Connections More than One TCP Connection Is Allowed TCP Ports IP Address = WKP = 3225 Ethernet Gigabit/WAN Interface FCIP Physical Link TCP Connection FC Frames in TCP/IP Class 3 and Class F Can Be on Separate Ports or Connections 398 Printed in USA.

200 ANSI Meets IETF E-Port FC-BB-2 FCIP 399 ANSI Meets IETF B-Port FC-BB-2 FCIP 400 Printed in USA.

201 FCIP Standards Stack Details This Will Be the ISL Connection Either a Bridged connection or E_Port; Depending on FCIP Implementation selected by Vendor 401 Additional IETF Drafts SLP: Service Location Protocol draft-ietf-ips-fcip-slp Used for dynamic discovery of FCIP ports IPSec for storage draft-ietf-ips-security More details later on this requirement for FCIP MIBs draft-ietf-ips-scsi-mib draft-ietf-ips-fcmgmt-mib draft-ietf-ips-fcip-mib FC-BB Published ANSI project being superseded by BB Printed in USA.

202 ANSI: FC-BB-2 Essentials (FCIP E-Port) Defines a slightly complex model; FC-BB-2 covers the FC portion of this model (FC entity and some of above) Cisco s FCIP E_Port implementations will closely follow this model 403 IETF: FCIP Essentials (FCIP E-Port) FCIP follows the model proposed in FC-BB-2; FCIP covers the lower portion of this model (FCIP entity and below) Cisco s FCIP E_Port implementation will follow this model 404 Printed in USA.

203 ANSI: FCIP Essentials (FCIP B-Port) Again the FC side of the this model follows SW-BB-2 standards With B_Port there is no FC switching element so the B_Port device will not be seen as a switch in the fabric but as a passive device 405 IETF: FCIP Essentials (FCIP B-Port) The FCIP part of the B-Port operation is the same as FCIP for the E_Port Note in this diagram that implementations of this standard can be any number of ports from 1 to n 406 Printed in USA.

204 About FCIP Links The FCIP interface represents both the VE_Port and the FCIP link An FCIP link is defined as one or more TCP connections FCIP link endpoint (LEP) terminates FCIP links FCIP data engine: One per TCP connection TCP Ports Entity 1 VE_Port FCIP_LEP DE DE WKP = 3225 IP Address = TCP/IP Network Interface FCIP Link Class F Class About the FC Entity FC entity interfaces (internally) with FCIP entity FC entity components: Control and Services Module Provides FC frame and timestamp along with synchronization with FCIP entity Correct order delivery of FC frames Works with FCIP entity for flow control Computes end-to-end transit time Throws away expired frames Answers to authentication of TCP connection request 408 Printed in USA.

205 About the FCIP Entity FCIP entity interfaces (internally) with FC entity FCIP entity components: Provide FC frame and timestamp to FC entity Tells FC entity about discarded bytes Tells FC entity about new and lost TCP connections and reason codes Monitors special frame changes Makes request to FC entity for authentication 409 FCIP Link Endpoint: Details FCIP_LEP is the translation point between an FC entity and an IP network LEP coordinates between FC and TCP flow control mechanisms 410 Printed in USA.

206 Error Detection and Recovery Data engine uses various methods to detect errors but does not correct errors Rather, it inserts EOFa (abort) frame delimiters when possible Requests sent up to FC entity to handle recovery 411 IETF: Fibre Channel Frame Encapsulation Header Defines the encapsulation header for Fibre Channel frames Not specific to FCIP Includes timestamp, CRC and provision for special frames 412 Printed in USA.

207 Initialization of Port B_Port Link initialization Exchange link parameters Link reset E_Port Link initialization Exchange link parameters Reset link Exchange switch capabilities Exchange fabric parameters Assign domain IDs Establish routes Merge zones if required 413 Link Initialization Flow E_Port on Switch B_Port or E_Port on FCIP Device These Are All Special Ordered Sets of 8B/10B Coding LF NOS OLS LF NOS = Not Operational Sequence OLS = Offline Sequence LR = Link Reset LR AC LR Idle LRR Idle OL LR LRR =Link Reset Response AC = Activity State LR = Link Recovery State LF = Link Failure State OL = Offline State AC 414 Printed in USA.

208 Link Capture E_Port on Switch NOS B_Port or E_Port on FCIP Device LR IDLE R-RDY IDLE IDLE IDLE R-RDY IDLE LR & LRR to Initialize Flow Control Parameters Per FC-PH LRR At this Point B_Port Device Is Up and E_Port to E_Port Exchange Continues 415 ISL E_Port If It Is an E_Port FCIP Device or If the B_Port Is Now up the Switch to Switch Exchange Continues 416 Printed in USA.

209 ELP Data Bit 15 of flag will be a 1 for B_Port RA_TOV is fabric wide timer, ED_TOV is per Link PWWN & WWN, Vendor ID also Credit value is one to start to allow only one out standing frame during link start-up Class 2 & 3 supported 417 E_Port and B_Port Summary FCIP E-Port VE - Port FCIP E-Port FC SAN FC Switch IP Network FC Switch FC SAN E Port Operation Exchange Link Parameters Exchange FCIP- Link Parameters Exchange Link Parameters Exchange Fabric Parameters Exchange Fabric Parameters Exchange Fabric Parameters ESC ESC ESC FCIP E-Port FCIP B-Port VB - Port FCIP B-Port FCIP E-Port 7200 w/ PA-FC-1G 7200 w/ PA-FC-1G FC SAN FC Switch IP Network FC Switch FC SAN B Port Operation Exchange Link Parameters Exchange FCIP-Link Parameters Exchange Link Parameters Exchange Fabric Parameters ESC (Exchange Switch Capabilities) if required 418 Printed in USA.

210 FCIP: ISL Connection The E-Port or B-Port FCIP Connection Will Provide: Simple name service across the IP tunnel FC discovery between SAN islands FSPF routing services between fabric switches Management server information Buffer credits 419 Comparisons B-Port and E- Port Differences 420 Printed in USA.

211 FCIP Connection Establishment Non-dynamic TCP connection to a specific IP address Dynamic Discovery of FCIP entities using SLPv2 Use of FCIP special frame Use of options 421 Non-Dynamic TCP Connections The FCIP entity is informed of a TCP connection is needed (Most likely done by configuration parameters in the device) IP address and security features are established (Configured) Destination WWN is determined (Configured) TCP/IP parameters are set (Configured) Quality of service is determined (Configured) Connection request is made to Port 3225 or configured port 422 Printed in USA.

212 Dynamic TCP Connections SLPv2 IP security for SLP determined Enter FCIP discovery domain process Advertise availability to SLP discovery domain service agent Locate FCIP entities in the discovery domain as a SLP user agent For each discovered entity follow same process as non-dynamic method to establish connection 423 FCIP Special Frame TCP Connection Is Established Sending Side First frame sent after TCP connection is established Sending side waits for FSF echo (90 seconds) Echo is match or non-match (Non-match terminates TCP connection) Creation of FCIP_LEP and FCIP_DE Inform FC Entity of connection and usage flags 424 Printed in USA.

213 FCIP Special Frame TCP Connection Is Established Receiving Side (Listening) Listen for connections on WKN port 3225 or configured port Checks database to allow connection Checks security features Wait for FSF frame (90 seconds) Inspect FSF contents and send echo frame Connection nonce Destination FC fabric entity world wide name Connection usage flags Connection usage code 425 FCIP Special Frame Details Used to exchange WWNs, entity pair identifiers, TCP connection identifiers and to except or reject connection Identify what kind of traffic (SOFi3, SOFn3, EOF) is intended; Not enforced In conjunction with connection usage flags, connection usage code help FCIP entity apply proper QoS parameters for the connection Adjustments to FSF with use of change bits can be made when frame is echoed back If two entities are trying to send FSF connection frames simultaneously first to Rx echo wins 426 Printed in USA.

214 FC FC FCIP: Tunnel Setup as Proposed in FCIP Draft The first frame transmitted in each direction is a special frame used to identify the peers FCIP entities and to synchronize I Am WWN1, This Is my FC/FCIP Identifier Are You Fabric WWN2? Fibre Channel 1) Special Frame Sent 2)Special Frame echoed 3) FCIP Tunnel Setup Complete FCIP Device Ok WWN1, I Am WWN2 Let s Setup the Connection Fibre Channel / /9 10/11 12/13 proto 0x01 proto 0x01 Flags 0x00 version 0x01 version 0x01 Frame Len 0x12 Reserved 0x0000 ~proto 0xFE ~proto 0xFE ~Flags 0x3F CRC (Reserved in FCIP) 0x ~version 0xFE ~version 0xFE pflags 0x00 ~pflags 0x00 ~Frame Len 0x3ED Timestamp integer/fraction Reserved 0xFFFF Source FC Fabric Entity WWN (identify the fabric) Source FC/FCIP Entity Identifier 14 Connection Nonce (random number) IP WAN 15/16 17 Conn Usage flags 0x00 Connection usage code Destination FC Fabric Entity WWN Reserved 0x0000 Reserved 0xFFFF 427 pflag Breakdown 428 Printed in USA.

215 FCIP Header Format Ones Compliment for Synchronization and Error Checking FCIP header used after FSF exchange is completed 429 Connection Options TCP selective acknowledgement (SACK) Per RFC 2883 TCP window scale option Protection from sequence number wrap (PAWS) TCP keepalives (KAD) Flow control mapping between TCP and Fibre Channel 430 Printed in USA.

216 FCIP Security Requirements (Per Draft) To Support IP Network Security FCIP Entities MUST: Implement cryptographically protected authentication and cryptographic data integrity keyed to the authentication process, and Implement data confidentiality security features FCIP utilizes the IPSec protocol suite to provide data confidentiality and authentication services, and IKE as the key management protocol FCIP Security compliant implementations MUST implement ESP and the IPsec protocol suite based cryptographic authentication and data integrity [11], as well as confidentiality using algorithms and transforms as described in this section 431 FCIP Security Requirements (Per Draft) (Cont.) FCIP implementations MUST meet the secure key management requirements of IPsec protocol suite FCIP entities MUST implement replay protection against ESP sequence number wrap FCIP entities MUST use the results of IKE phase 1 negotiation for initiating an IKE phase 2 quick mode exchange and establish new SAs Note: An External Device May Be Used in Conjunction with the FCIP Implementation to Meet the Must Implement ESP Requirement 432 Printed in USA.

217 Important FC and FCIP Timers Resource Allocation Timeout Value (R_A_TOV) Timeout value that determines how long a FC frame can be in transit on the Fibre Channel network This is a fabric wide value with a default value usually at 120 sec on switch networks Error Detect Timeout Value (E_D_TOV) A value that times events and responses at the link level; Errors at the link level will cause delays of these events This value is defaulted to 10 sec and should be lower then R_A_TOV; Again this is a fabric wide setting Keep Alive Timer K_A_TOV A value that is applied to TCP connection and is used when no data is present 433 Time Stamps and Synchronization Clock synchronization is required if timestamps are used Synchronized to FC services Synchronized to IP NTP Transit time through IP network is applied via a timestamp Integer If no timestamp value is available zero will be used Fibre channel time values still apply across the ISL link and are timed-out via lack of RDY coming back End system devices such as HBA attached hosts still require normal responses to timers end-to-end (no spoofing) 434 Printed in USA.

218 Timestamps TS are the responsibility of the FC entity This allows transit through the FCIP entity to be included in the measurement This transit time should be well below R_A_TOV 435 Buffer Credits Fibre channel buffer credit methods do not change R_Rdy s will be used to control flow coming from FC switch on a per link basis Buffer credit establishment is determined at FLOGI Mechanisms to control flow of R_Rdy s to FC switch based on TCP/IP congestion is per FCIP solution FC switches do not require extended credit methods 436 Printed in USA.

219 Error Recovery Errors on FC side of local B_Port are not forwarded over the IP network; Issues such as loss of sync or a FC encapsulation error will not be set to the FC entity Errors on IP side are handled by TCP and frame is dropped if checksum is in error 437 Summary FCIP is the standards approach to connect Fibre Channel ISLs over TCP/IP LAN/WAN connections State of draft wording will most likely stay as it is worded today Security, network delay and error recovery will be biggest concerns No shipping product today conforms to the proposed FCIP draft Cisco will have several platforms supporting FCIP solutions 438 Printed in USA.

220 INTERNET FIBRE CHANNEL PROTOCOL 439 ifcp Protocol Model FC-4 ifcp ifcp FC-1 TCP TCP FC-1 FC-0 IP LINK IP LINK FC-0 Gateway Region PHY PHY Gateway Region IP Network ifcp replaces the transport layer of Fibre Channel (FC-2) with an IP network but keeps the FC-4 mapping the existing Fibre Channel transport services on TCP/IP ifcp processes differently FC-4 frame images (applications), FC-2 frame images (link service request), FC broadcast and ifcp control frames Topology within the gateway regions are opaque to the IP network and other gateway regions (they appear just like collection of N_Ports) 440 Printed in USA.

221 ifcp Network Model: isns Role An ifcp gateway cannot operate without access to an isns server Client-Server architecture isns functions: Device Discovery and fabric management Emulation of the services provided by the FC name server and RSCN Definition and management of discovery domains Definition and management of logical fabrics IP Network IFCP Gateway Region Gateway N_port-to-N_port session IFCP Gateway Gateway Region isns Queries isns N_port-to-N_port session IFCP Gateway isns Queries IFCP Gateway Gateway Region Gateway Region 441 ifcp Protocol Description: N_Ports Addresses Allocation Two different schemes: Address transparent mode (optional): The N_Port FC_IDs are unique across the whole logical fabric Address translation mode (mandatory): The N_Port FC_IDs are unique only inside the gateway region the N_Port belongs to 442 Printed in USA.

222 Address Transparent Mode All the gateways belonging to the same logical fabric cooperate to assign addresses that are unique across the gateway regions that form the logical fabric No need for address translation Not scalable (max 239 gateways) 443 Address Translation Mode ifcp gateways use aliases to map the local representation of addresses of external gateway regions to the real addresses outside the gateway region (comparable to IP NAT) Requires a rewrite of the FC_IDs in the FC frame header and in the FC payload for some ELS (i.e. ADISC) 444 Printed in USA.

223 ifcp Protocol Description: Address Translation Mechanism Give Me the Remote Gateway IP Address, N_Port ID, N_Port WWN 1) The N_Port Issues a NS Query FC NS Request FC_ID = x.x.x FC NS Reply 2) isns query/reply IFCP Gateway isns TCP/IP Remote GW IP Dest N_Port ID (y.y.y) Local N_Port alias (z.z.z) Dest N_Port WWN 4) The Gateway Sends Back to the N_Port the NS Reply (for FC_ID z.z.z) IFCP Gateway FC_ID = y.y.y 3) The Requesting GW Fills Up the Address Translation Table 445 ifcp Protocol Description: Address Translation Mechanism (Cont.) 2) The GW Makes a Table Lookup Gets the Remote GW IP Address (to Set Up the ifcp Session) and the Actual Dest N_Port ID ( to Rewrite the D_ID) PLOGI did y.y.y sid x.x.x Remote GW IP Dest N_Port ID (x.x.x) Local N_Port alias (w.w.w) Dest N_Port WWN 3) The receiving GW Fills Up Its Own Translation Table 1) The N_Port I a PLOGI to D_ID z.z.z PlogiI did z.z.z sid x.x.x IFCP Gateway TCP/IP Remote GW IP Dest N_Port ID (y.y.y) Local N_Port alias (z.z.z) IFCP Gateway PLOGI did y.y.y sid w.w.w FC_ID = y.y.y 4) The Receiving GW Rewrites the S_ID of the Incoming Request Dest N_Port WWN FC_ID = x.x.x In case of fabric reconfiguration all the address translation tables need to be recalculated with a consequent loss of every active login session 446 Printed in USA.

224 ISNS AND SLP DISCOVERY PROTOCOLS FOR THE IP-SAN 447 Discovery Approach Deploy and Interoperate in Three Stages: 1. Naming and static configuration Configure both targets and initiators Use SendTargets to reduce initiator config 2. SLPv2 for multicast and simple discovery Configure targets 3. isns for centralized management Configure central isns server 448 Printed in USA.

225 Service Location Protocol (SLP) Based on service location protocol v2 (RFC 2608) Allows hosts to search for instances of a network service they are interested in: Example: printers 449 Basic SLP Discovery Requirements Find targets by initiator s worldwide unique identifier Tell me which targets you have that I should see Find targets by target s worldwide unique identifier Where is target iscsi.com.acme.foo? Propagate attributes needed before connecting Boot information, authentication information Scaling requirements Zero-configuration, no servers in small environments Reduce or eliminate multicast in medium environments Interoperate with LDAP/iSNS in large environments 450 Printed in USA.

226 Service Location Protocol (SLP) Three Components, Two of Which Run in Our Storage Router SA Service Agent; Services register with SA UA User Agent; Queries SA or DA for registered services DA Directory Agent; Proxies for a set of SAs UA query/response SA register services services services query/ response DA register 451 Service Location Protocol for IP Storage SLP DA host Management Code SLP UA SLP SA iscsi Initiator TCP/IP IP TCP/IP device iscsi Target Management Code Service Agent (SA) Advertises services Services have attributes User Agent (UA) Finds services Zero configuration Directory Agent (DA) Optional Propagate service adverts SLP Protocol UDP or TCP Minimize multicast 452 Printed in USA.

227 Implementing SLP for iscsi Targets implement a service agent Answer multicast requests or register with DA Initiators implement a user agent Use multicast or DA to locate targets Devices containing targets register: The canonical target or individual targets Attributes of targets Register target at each of its addresses 453 SLP Summary Serverless discovery of targets Optional, generic DA to scale services Zero-configuration of hosts SLP makes careful use of multicast Access list and attribute propagation Optional message authentication Available open source implementations 454 Printed in USA.

228 What Is isns isns Facilitates Scalable Configuration and Management of iscsi, ifcp and Fibre Channel (FCP) Storage Devices in an IP Network, By Providing a Set of Services comparable to that Available in Fibre Channel Networks isns Functions There Are Four Main Functions of the isns: 1. A name server providing storage resource discovery 2. Discovery Domain (DD) and login control service 3. State change notification service 4. Open Mapping of Fibre Channel and iscsi devices 456 Printed in USA.

229 Basic: How isns Works 1 iscsi Clients Register with isns Server, Done By Adding isns IP Address to iscsi Application Driver iscsi & isns Clients IP Network Fibre Channel SAN FC 2 iscsi Targets Register with isns Server 3 3 isns Clients Query isns Server for Storage Location and Name FC 2 4 iscsi Client then Selects and Logs into iscsi Target Using Information from isns Server 457 Internet Storage Name Service (isns) isns server functions: Allows an isns client to register/deregister/query with the isns server Provides centralized management for enforcing access control of targets from specific initiators Provides a state-change notification mechanism for registered isns clients on the change of status of other isns clients Similar to the functionally provided by the FC name Server, Zone Server and the RSCN mechanism 458 Printed in USA.

230 isns Components isns protocol (isnsp) A flexible and lightweight protocol that specifies how isns clients and servers communicate Discovery Domain (DD) A grouping of storage devices much like a zone in the FCP; discovery domains help in control and manage logins and services available to the clients in the domain; Based on the FC-GS standard for fiber channel; Items like default domain are used Discovery Domain Set (DDS) A group of one or more discovery domains; A method to store sets of domains within the isns database; Multiple DDSs can be active at one time, unlike zonesets in FCP where only one can be active at a time 459 isns Components isns client The isns client is located within storage system and talk to the isns server using the isnsp within its configured device domain; client can belong to one or more DDs; isns client registers its attributes with the isns server and receives notices of changes within the domain isns database The isns database is the information repository for the isns server; it maintains information about isns clients attributes; a directory-enabled implementation of isns may store client attributes in an LDAP directory infrastructure isns server isns servers respond to isns protocol queries and requests, and initiate isns protocol state change notifications; properly authenticated information submitted by a registration request is stored in an isns database; listens on port Printed in USA.

231 isns SCN (State Change Notifications) isns clients who wish to receive SCN have to explicitly register with isns server the events in order to receive the notifications Initiator/target/object with add/remove event or to/from discovery domain are the events that can be registered isns servers generate SCN when either the state of any target device changes or when the target device itself requests an SCN to be generated using SCN event message; isns listens to FCNS to registration/deregistration 461 SCN Types Regular registrations This type of SCN is used within a DD; The discovery domain will control where the SCN message will go Management registrations Used by control nodes and can travel outside the DD from which they came Can be TCP or UDP messaging (Most implementations only using TCP for now) 462 Printed in USA.

232 Services Provided by the Discovery Domain Login control Authorization and control policies for storage targets can be maintained by isns servers only allowing authorized devices to access the targets Control of what target portals are accessible within the discovery domain Fibre Channel to iscsi device mapping isns database learns and stores naming and discovery information about FC storage devices discovery on the iscsi Gateway and iscsi devices in the IP network; This database can then be available by FC and IP isns clients 463 High Availability of isns Servers Can use SLP to discovery other isns servers Database transfers between servers using isnsp or SNMP Heartbeat mechanism used between active and backup isns servers 464 Printed in USA.

233 Internet Storage Name Service (isns) for iscsi The isns protocol (isnsp) provides: A mechanism for iscsi clients to discover other iscsi targets/initiators Enforce access control Notifications from an isns server on changes to the status of a logged in iscsi device Provide ability to discovery iscsi target on different IP network iscsi target discovery can happen through: Static configuration of initiator iscsi sendtargets command Name server/directory server (via isns) 465 isnsp Header isnsp Version C the Current Version is 0x0001; All Other Values Are RESERVED isnsp Function ID Defines the Type of isns Message and the Operation to Be Executed; isnsp PDU Length Specifies the Length of the PDU PAYLOAD Field in bytes; The PDU Payload Contains Attributes for the Operation isnsp Flags Indicates Additional Information About the Message and the Type of Network Entity That Generated the Message isnsp Transaction ID MUST Be Set to a Unique Value for Each Concurrently Outstanding Request Message; Replies MUST Use the same TRANSACTION ID Value as the Associated isns Request Message isnsp Sequence ID The SEQUENCE ID Has a Unique Value for Each PDU Within a Single Transaction isnsp PDU Payload The isnsp PDU PAYLOAD Is Variable Length and Contains Attributes Used for Registration and Query Operations Authentication Block For isns Multicast and Broadcast Messages, the isnsp Provides Authentication Capability; The isns Authentication Block Is Identical in Format to the SLP Authentication Block 466 Printed in USA.

234 isnsp Commands for iscsi The Following Are isnsp Commands Messages Used in Support of iscsi: 467 isnsp Responses for iscsi The Following Are isnsp Response messages Used in Support of iscsi: 468 Printed in USA.

235 isns Queries for iscsi isns clients can perform two types of queries: Device attribute query: isns server responds with requested attributes of one or more isns clients The isns server converts the received query to a FC name server query in the SAN FC name server will ensure that the resultant set is filtered based on zones The isns server translates each entry returned by the FC name server to the corresponding isns clients Apply filters based on iscsi access control by removing all statically configured virtual targets the query initiator is not allowed to access Device get next query: Allows an iterative query of the isns server s isns client database 469 Return Information from isns iscsi Query iscsi Name Name of Port on the IP Gateway Entity IP Address of Portal to Log to and Ask for This target 470 Printed in USA.

236 isns for ifcp Will work much the same manor as iscsi just will require other related attributes to be registered and queried Is required for ifcp Functions much like domain name server and domain ID manager Needs to be highly available service for FC devices 471 isnsp Commands for ifcp The Following Are isnsp Commands Messages Used in Support of ifcp: 472 Printed in USA.

237 isnsp Responses for ifcp The Following Are isnsp Response Messages Used in Support of ifcp: 473 SLP and isns SLP used for target discovery No configuration required for the simplest networks Small footprint; no servers required Just enough discovery for small-to-medium networks Device-centric access control model isns adds storage management capabilities Active monitoring of initiators and targets Event propagation Public key distribution Centralized access control model 474 Printed in USA.

238 Using Both SLP and isns Initiators can use both SLP and isns to discover targets Targets should use SLP only if not configured for isns Gateways or proxies may provide local SLP discovery of remote isns devices 475 TECHNICAL TOOLS AND SKILLS 476 Printed in USA.

239 Storage Networking Toolbox Test tools for Fibre Channel and IP Host based tools Network component serviceability tools Software debug tools Knowledge 477 Fibre Channel Analyzers Most units are based on dedicated hardware, and might be supplied with software tools for performance base lining Very expensive Oriented to protocol conformance testing Requires 2 GBICs interfaces to be implemented Monitoring units might have a retiming mode, to cleanup some of the timing problems on a link, and to separate them from the real problem at layer 1 Statistical software can run on these type units Collecting statistics on the status of the line, or other parameters (number of bits, exchanges ) Sharing is still a dream in most cases, it is complex to share in the field, so in most cases the portable versions are the most suitable 478 Printed in USA.

240 Fibre Channel Analyzers Snooping GBICs or fiber taps; allow to monitor without service interruption; very important for Fibre Channel work in the field Traffic probes; used to remotely monitor the state of a network without service interruption Trace viewers (free from the vendor websites) Each vendor has its own PC viewer and must be used with each capture tool; these can be found at each of their websites 479 FC Test Vendors Leaders in dedicated hardware tools: Finisar ( Xyratex ( Aglient ( I-Tech ( Ancot ( Spirent/Netcom systems ( 480 Printed in USA.

241 SCSI Host-Based Testing I/O meter I/O zone SCSI tools Xyratex disk basher Freeshare or software tools for SCSI and I/O analysis, tools for disk manufacturing Windows Tools iscsi Driver debug helpers Windows debug utilities Detail uses of O/S disk administrator to verify and check health of target devices 482 Printed in USA.

242 IP: GiGE GiGE testers $$$ Agilent Sniffer Fluke Finisar/Shomiti iscsi decodes just becoming available on most tools All your IP tools IP Ping, trace, etc. Fibre Channel ping available at iscsi Decoding Software only analyzers like Ethereal ( Hardware analyzers Can use monitor command on Cisco switches to span the iscsi GiGE port to a 10/ Printed in USA.

243 Available Certifications SNIA (Storage Networking Industry Association) Level 1 Fibre Channel storage networking professional Level 2 Fibre Channel storage networking practitioner iscsi training available at many education sources Infinity I/O, medusa, solution technology, others Other certifications that are vendor specific 485 ARCHITECTURAL DESIGN OF STORAGE AREA NETWORKS 486 Printed in USA.

244 Section Agenda Introduction Hierarchy Modularity Architecture Examples 487 INTRODUCTION 488 Printed in USA.

245 Hierarchy, Modularity and Limited Failure Domains Why Do This? (Benefits Summary): Scalable architecture Improved performance Manage change Improve service Improved security Simplified management and troubleshooting Reduced cost of ownership 489 What Problem Are We Solving? Applications Must Be Available and Perform Well Networks that Deliver on this Requirement: Have consistently high performance Are reliable, scaleable, and manageable Are secure and cost-efficient Are service and solution enabling Adapt to changing requirements 490 Printed in USA.

246 Network Design Goals Architecture Provides: Performance Reliability, availability, and scalability RAS Cost efficiencies Security A base to enable services and solutions To Meet Mission-Critical Business Objectives, Applications Need to Be Consistently Up, Available, and High-Performing 491 Architecture: Hierarchy, Modularity, and Domains Hierarchy Modularity Domains Functionally Divides the Problem Create Manageable Building Blocks Limits Scope of Potential Failures Fundamentally, We Break the Network Design Process into Manageable Blocks so that the Network will Function within the Performance and Scale Limits of Applications, Protocols and Network Services 492 Printed in USA.

247 What Does This Mean? We Build Networks that Have Structure: Access Focus of This Discussion Building Blocks Distribution Application Servers Core Backbone Enterprise Storage WAN Internet PSTN 493 Applying Design Principles to Storage Hierarchy Predictable performance Scaleable design Fault isolation Modularity Cost-effective Repeatable Domain Reliability Security Unified Storage Mgmt Core Shared Storage 494 Printed in USA.

248 HIERARCHY 495 Hierarchy: Physical and Logical Physical hierarchy Predictable performance Scaleable design Fault isolation High availability Logical hierarchy Virtual SANs Zoning Enhances physical hierarchy Physical Architecture Virtual SAN A Virtual SAN B Zone 1 Zone 2 Zone 1 Zone 2 H1 H2 H1 H2 Zone 3 Zone 4 D1 H3 D2 D1 H7 D7 Logical Architecture Logical Architecture 496 Printed in USA.

249 Hierarchy: Physical Consolidated Storage Network Cost-effective solution Benefits of consolidation Limited scalability Small to medium business Expansion can be disruptive Single fault redundancy Double fault would likely result in isolation iscsi iscsi 497 Hierarchy: Physical Collapsed Core Architecture Collapsed core High performance Multiple unequal paths Better scalability Medium to large enterprise ISLs can limit scalability Redundant Mesh topology Network survives some double faults iscsi iscsi 498 Printed in USA.

250 Hierarchy: Physical Core Edge Architecture Core Edge High performance Load balancing Consistent hop count Good scalability Large to very large enterprise Non-disruptive expansion Better fault tolerance Improved fault isolation Single fault within layer okay iscsi iscsi 499 Hierarchy: Physical Oversubscription To be expected in storage networks Typically lower factors than we see in LANs Architecture should be flexible to accommodate differing requirements for various hosts and storage subsystems Bandwidth can be modified non-disruptively by using port channels between switches Take into account any inherent over subscription in networking hardware Use actual anticipated throughput rather than link speed for calculating bandwidth requirements 500 Printed in USA.

251 Hierarchy: Physical Inter-Switch Links Inter-Switch Link ISL Physical FC link between two fabric switches forming a trunk Utilized for FC services and data traffic Port Channel Multiple FC ISLs combined to form a single aggregated trunk All links in a Port Channel must be directly connected to the same two switches Individual link state changes do not cause ISL trunk state changes iscsi iscsi ISL Port Channel 501 Hierarchy: Physical Scalability Oversubscription Higher OS acceptable for some hosts Lower OS for High performance hosts and storage devices Consider impact of multipath load balancing Determine acceptable worst case in various failure scenarios Can be non-disruptively changed by adding/ removing links to port channels 15:1 OS iscsi 1:1 OS Core iscsi 4x2Gb ISL 8x2Gb ISL 8:1 OS 3:1 OS 502 Printed in USA.

252 Hierarchy: Logical Virtual SANs VSANs provide a means to build a logical structure on top of a physical SAN Similar to how VLANs are used to scale ethernet networks VSANs help scale Fibre Channel networks Topology changes are isolated within the VSAN therefore adds, moves, and changes are not disruptive to other VSANs VSANs can be utilized to establish administrative domains Zoning provides an additional access control mechanism within each VSAN 503 Hierarchy: Logical Logical Architecture Virtual SANs Similar to Ethernet VLANs except no inter-vsan flows Enhanced ISL provides VSAN trunking (EISL) Complimentary to port channel iscsi iscsi Services scalability Independent Fibre Channel services for each VSAN Zoning is per VSAN EISL Port Channel Failure domain Faults contained within VSAN 504 Printed in USA.

253 Hierarchy: Logical Maximizing VSAN Architecture Isolate multiple paths into separate VSANs Independent FC services per VSAN Provides complete traffic isolation between redundant paths Each VSAN converges independently for faster recovery and improved fault isolation iscsi iscsi 505 Hierarchy: Combining Physical and Logical iscsi iscsi Fabric A provides one set of links and Fibre Channel services Fabric B provides an independent set of links and services A B 506 Printed in USA.

254 MODULARITY 507 Modularity: Key Elements The ability to scale the network while maintaining consistent performance Building block approach breaks network into smaller chunks that are easier to understand, replicate, and deploy Changes and additions can be made non-disruptively Provides consistent and limited failure domains Modularity can also define administrative boundaries 508 Printed in USA.

255 Modularity: Building Blocks Application Modules iscsi iscsi Functional Building Blocks Provide Scalability with Deterministic Performance Fiber Channel Core Storage Modules 509 Modularity: Utilizing VSANs Adds, moves, and changes contained within a VSAN are non-disruptive to other VSANs Using VSANs facilitates application modeling and testing Per VSAN statistics Per VSAN traffic engineering Per VSAN administration (if desired) Eliminates costs associated with separate physical fabrics 510 Printed in USA.

256 Modularity: Benefits of VSANs Overlay isolated virtual fabrics on same physical infrastructure Each VSAN contains zones and separate (replicated) fabric services VSAN membership determined by port VSANs for availability Isolate virtual fabrics from fabric-wide faults/reconfigurations Security Complete hardware isolation Scalability Replicated fabric services Thousands of VSANs per storage network Management Roll Based Access Control RBAC Provides administrative boundaries Mgmt VSAN Department/ Customer A Shared Storage Department/ Customer B VSAN-Enabled Fabric VSAN Trunks 511 Modularity: Storage Intelligence and VSANs Dept 1 VSAN Dept 2 VSAN Dept 3 VSAN VSANs created to provide isolation of fabric-wide services. Virtualization Virtualization allows physical storage to be in its own VSANs, separate from the host VSANs. Data Center VSANs VSANs provide Secure isolation of physical storage Easier configuration Dynamic configuration of fabrics Role-based access control 512 Printed in USA.

257 ARCHITECTURE EXAMPLES 513 Architecture: iscsi Scalability Less expensive alternative for host not requiring 2Gbps Recommend separate NIC TCP Offload Engine TOE Consider actual throughput requirements for scalability Host Services Appears as normal HBA Compatible with host based storage utilities multi-path, load balance, mapping, etc. Network File System TCP/IP Stack NIC Driver Applications iscsi Driver TCP/IP Stack NIC Driver TOE Block Device SCSI Generic iscsi Driver TCP/IP Stack NIC Driver File System Adapter Driver SCSI Adapter 514 Printed in USA.

258 Architecture: iscsi High-Availability Redundant connections to hosts or servers High-availability iscsi services Redundant paths to backend FC SAN Host with Multiple(iSCSI) NICs and Multipathing Software Installed Application Multipathing iscsi Driver Multiple Ethernet Switches Redundant iscsi to Fibre Channel Connections and Services Storage Array with Redundant Controller Ports 515 Architecture: iscsi Authentication SCSI routing service passes username and MD5-hashed password from initiators to AAA server AAA authentication list used to determine which service(s) to use for authentication User1/pwd1 User2/pwd2 / RADIUS Server TACACS + Server User1/pwd1 User2/pwd2 / User1/pwd1 User2/pwd2 / iscsi Hosts (Initiators) User1/pwd1 CHAP RADIUS TACACS+ Local Authentication Services List AAA Authentication Services iscsi Storage (Targets) SCSI Routing Instance IP Network iscsi Services FC Fabric 516 Printed in USA.

259 Architecture: iscsi Topology iscsi Best Practices Isolate IP storage network behind application hosts with VLANs Minimized potential for bandwidth contention Map VLANs to VSANs for manageability Dedicated ethernet interfaces on host for attachment to storage network Ethernet Switches iscsi Services Front-Side IP Network iscsi iscsi iscsi iscsi Storage Pool IP Storage Network FC Fabric Clients iscsi-enabled Hosts FC Attached Hosts with HBAs 517 Architecture: SAN Extension Technology Technology Choice Requires Matching Storage Application Requirements with Service Availability, Cost, Throughput, and Latency IP WAN CWDM FCIP FC DWDM FCIP FC SONET/SDH 518 Printed in USA.

260 Architecture: High Availability for SAN Extension: FC Utilize disparate paths and portchannel for high availability Utilize VSANs to limit the failure domain in the event of lost connectivity Fabric A FC CWDM FC Fabric A Fabric B PortChannel DWDM Fabric B SONET/SDH Both fabrics remain connected if one of the paths fails Use of portchannel prevents state change on link failure 519 Architecture: High Availability for SAN Extension: FCIP Utilize disparate paths and portchannel for high availability Utilize VSANs to limit the failure domain in the event of lost connectivity Recommend not using etherchannels Fabric A FCIP IP WAN FCIP Fabric A Fabric B PortChannel IP WAN PortChannel Fabric B Both fabrics remain connected if one of the paths fails Use of portchannel prevents state change on link failure 520 Printed in USA.

261 Architecture: Legacy Storage Implementation Storage is captive behind applications Inefficient allocation of storage resources Multiple administrative domains SAN Islands Captive Storage Blocks Application Servers Campus Clients Remote Clients Internet Clients LAN Core Backbone 521 Architecture: Factors for Determining Architecture Current size and anticipated growth for both application servers and storage elements Baseline performance requirements for servers and storage Business continuance requirements SAN extension Administrative domains Migration plans Interoperability considerations Costs 522 Printed in USA.

262 Architecture: Collapsed Core Architecture Servers and storage elements connected to collapsed core Some scalability especially with iscsi Redundant paths Achieves economical storage consolidation VSANs can add scalability and management benefits Unified Storage Mgmt iscsi Application Servers Shared Storage 523 Architecture: Large Scale Architecture Application servers connect to edge switches Storage devices connect to edge switches Highly scalable Highly redundant Highly modular Multiple equal paths VSANs limit the size of any one SAN Unified Storage Mgmt Shared Storage iscsi Application Servers 524 Printed in USA.

263 Network Design Goals Architecture Summary: Performance Planned hierarchy, managed oversubscription, and modular design Reliability, Availability, and Scalability RAS Limited failure domains, leveraged VSANs, and modular design Cost efficiencies Consolidated storage, central management, and leveraged resources Security Limited domains, RBAC management, and consistent architecture A base to enable services and solutions Business continuance and disaster recovery Management of heterogeneous storage elements Ubiquitous access to storage from anywhere Infrastructure for storage virtualization 525 Architecture: End-to-End SAN Architecture FC Highly Scalable Storage Networks FC FC FC FC iscsi iscsi iscsi FC FC iscsi iscsi FC FC iscsi FC FCiSCSI FC Multiprotocol/Multiservice SONET Network iscsi iscsi iscsi FC FC iscsi iscsi FC FC FC iscsi iscsi FC FC iscsi- Enabled Storage Network Ethernet Switches SONET Network FC FC Asynchronous Replication FCIP over SONET Optical Network Resilient Optical Transport Networks Synchronous Replication Optical (FCIP/FC) FC FCIP Remote Storage Access FC FC FC FC FC Intelligent Workgroup Storage Networks FC FC FC FC FC 526 Printed in USA.

264 Q & A 527 Complete Your Online Session Evaluation! WHAT: WHY: Complete an online session evaluation and your name will be entered into a daily drawing Win fabulous prizes! Give us your feedback! WHERE: Go to the Internet stations located throughout the Convention Center HOW: Winners will be posted on the onsite Networkers Website; four winners per day 528 Printed in USA.

265 529 EXTRAS 530 Printed in USA.

266 Port RX TX FC LOOP OPERATIONS 531 Single Port ARB AL_PA 2A IDLE RX Port TX IDLE 1. The Loop is initially filled with IDLES AL_PA Each port is in the monitoring state 3. Because of no activity CFW = Idle RX Port AL_PA B2 4. Rx IDLES are replaced with CFW TX IDLE AL_PA EF Port IDLE RX TX 532 Printed in USA.

267 Port RX TX Port RX TX Single Port ARB AL_PA 2A ARB(01) RX Port TX IDLE 1. Port_01 begins to arbitrate for access to the Loop AL_PA Port_01 changes its CFW from IDLE to ARB(01) RX Port AL_PA B2 3. Port_01 transmits ARB(01) when a fill word is required TX IDLE AL_PA EF Port IDLE RX TX 533 Single Port ARB AL_PA 2A ARB(01) RX Port TX ARB(01) AL_PA ARB(01) is Rx by the next port and updates its CFW to ARB(01) 2. Whenever a fill word is required ARB(01) is used; With no other activity on the loop ARB(01) is sent RX TX Port AL_PA B2 IDLE AL_PA EF Port ARB(01) RX TX When a Port Discards Rx Fill Words and Transmits the CFW this Allows the Port to Compensate for Clock Differences Between Rx Data Stream and Tx Data Stream 534 Printed in USA.

268 Port RX TX Port RX TX Single Port ARB AL_PA 2A ARB(F0) OPN RX Port TX 1. When Port_01 receives its own ARB(01) it wins arbitration AL_PA Port_01 sends on OPN to open a loop circuit and changes its CFW to ARB(F0) RX Port AL_PA B2 3. Port_01 discards any Rx ed ARB(x) TX ARB(01) AL_PA EF Port RX TX 535 Single Port ARB AL_PA 2A IDLE RX Port TX ARB(F0) 1. As each port Rx s the ARB(F0) it updates its CFW to ARB(F0) AL_PA Assuming that no other port is arbitrating, ARB(F0) travel the complete loop RX Port AL_PA B2 3. When ARB(F0) is Rx ed by Port_01 the CFW in Port_01 is changed to IDLE TX ARB(F0) AL_PA EF Port ARB(F0) RX TX 536 Printed in USA.

269 Port RX TX Port RX TX Single Port ARB AL_PA 2A IDLE RX Port TX IDLE AL_PA Each port receives the IDLE and updates its CFW to IDLE 2. Assuming the no other port is arbitrating and the IDLES travel the complete loop 3. As long as Port_01 owns the loop it discards any Rx ed IDLE or ARB(x) and continues to send its CFW when necessary AL_PA EF ARB(F0) Port IDLE RX TX Port AL_PA B2 RX TX Discarding the Receiving Arb(x) Prevents Any Other Port from Winning Arbitration 537 Multiple Port ARB AL_PA 2A ARB(01) RX Port TX IDLE AL_PA Port_01 begins arbitrating for access to the loop; Done by replacing IDLE and ARB(x) with ARB(01) 2. Port_B2 also begins arbitrating for the loop; It replaces Idle and ARB(x) with ARB(B2) IDLE AL_PA EF Port ARB(B2) RX TX Port AL_PA B2 RX TX 538 Printed in USA.

270 Port RX TX Port RX TX Multiple Port ARB AL_PA 2A ARB(01) RX Port TX ARB(01) AL_PA The ARB(01) gets to Port_2A which updates its CFW with ARB(01) and transmits this when the CFW is needed RX Port AL_PA B2 2. The ARB(B2) also travels to Port_EF which updates its CFW with ARB(B2) TX ARB(B2) AL_PA EF Port ARB(B2) RX TX 539 Multiple Port ARB AL_PA 2A ARB(01) RX Port TX ARB(01) AL_PA When Port_B2 receives ARB(01) it changes its CFW to ARB(01) because of 01 has higher priority(lower AL_PA wins) 2. When Port_01 receives ARB(B2) it is replaced with ARB(01) RX TX Port AL_PA B2 ARB(B2) AL_PA EF Port ARB(01) RX TX Because Port_B2 s ARB(B2) Is Replaced with ARB(01) It Will Not Win Arbitration at this Time 540 Printed in USA.

271 Port RX TX Port RX TX Multiple Port ARB AL_PA 2A ARB(F0) and OPN RX Port TX ARB(01) 1. ARB(01) is Rx by Port_01 and wins arbitration AL_PA Port_01 then opens the loop circuit and updates it s CFW with ARB(F0) when a fill word is required RX TX Port AL_PA B2 3. Port_B2 is still arbitrating but is lower priority AL_PA EF ARB(01) Port ARB(01) RX TX 541 Multiple Port ARB AL_PA 2A ARB(F0) RX Port TX ARB(F0) AL_PA Port_2A receives ARB(F0) and updates the CFW to F0 2. Port_B2 replaces the lower-priority ARB(F0) and transmits ARB(B2) RX TX Port AL_PA B2 ARB(01) AL_PA EF Port ARB(B2) RX TX 542 Printed in USA.

272 Port RX TX Port RX TX Multiple Port ARB AL_PA 2A ARB(F0) RX Port TX ARB(F0) 1. Port_EF updates its CFW to ARB(B2) and transmits on to Port_01 AL_PA Port_01 Tx s ARB(F0) 3. Port_B2 continues to replace F0 with B2; Port_01 discards all Rx ed ARB(x) ordered sets RX TX Port AL_PA B2 ARB(B2) AL_PA EF Port ARB(B2) RX TX When Port_01 Relinquishes Control of the Loop It Changes Its CFW to ARB(B2) Allowing Port_B2 to Win 543 Lower Priority Port ARB AL_PA 2A IDLE RX Port TX IDLE 1. Port_B2 begins to arbitrate for the loop by changing CFW to B2 AL_PA Each Rx ed IDLE and lower-priority ARB(x) is discarded by Port_B2 and the ARB(B2) is substituted in its place RX TX Port AL_PA B2 IDLE AL_PA EF Port ARB(B2) RX TX 544 Printed in USA.

273 Port RX TX Port RX TX Lower Priority Port ARB AL_PA 2A IDLE RX Port TX IDLE AL_PA ARB(B2) propagates around the loop to Port_EF 2. Port_EF changes its CFW to ARB(B2) and Tx s the ARB(B2) whenever a fill word is needed RX TX Port AL_PA B2 ARB(B2) AL_PA EF Port ARB(B2) RX TX 545 Lower Priority Port ARB AL_PA 2A ARB(B2) RX Port TX IDLE AL_PA The ARB(B2) propagates around the loop to Port_01 2. Port_01 changes its CFW to ARB(B2) and Tx s ARB(B2) whenever a fill word is needed RX TX Port AL_PA B2 ARB(B2) AL_PA EF Port ARB(B2) RX TX 546 Printed in USA.

274 Port RX TX Port RX TX Lower Priority Port ARB ARB(01) AL_PA 2A Port RX TX ARB(01) ARB(B2) XX AL_PA Port_01 begins arbitrating after a single ARB(B2) has passed 2. Port_01 has higher priority than Port_B2 and discards ARB(B2) and replaces it with ARB(01) 3. The single ARB(B2) travels around the loop to Port_2A. Port_2A passes the ARB(B2) 4. When ARB(01) is Rx ed at Port_2A its CFW is changed from B2 to 01 RX TX Port AL_PA B2 ARB(B2) AL_PA EF Port ARB(B2) RX TX 547 Lower Priority Port ARB AL_PA 2A ARB(01) RX Port TX ARB(01) ARB(B2) AL_PA The single ARB(B2) is Rx ed by Port_B2 which wins arbitration and begins to discard any Rx d ARB(x) 2. Port_B2 changes its CFW to ARB(F0) RX Port AL_PA B2 TX ARB(B2) AL_PA EF Port ARB(F0) and OPN RX TX 548 Printed in USA.

275 Port RX TX Lower Priority Port ARB AL_PA 2A AL_PA 01 ARB(01) RX Port Port TX ARB(01) 1. Port_EF changes its CFW to ARB(F0) and sends it on to Port_01 2. Port_01 substitutes ARB(01 for every ARB(F0) it receives 3. Port_B2 discards the ARB(01) and sends ARB(F0) as its fill word 4. When Port_B2 relinquishes the loop, it will change its CFW to ARB(01) and allow Port_01 to win the loop ARB(F0) AL_PA EF ARB(F0) RX TX Port AL_PA B2 RX TX 549 Printed in USA.