Best Practice and Deployment of the Network for iscsi, NAS and DAS in the Data Center Samir Sharma, Juniper Networks Author: Samir Sharma, Juniper Networks
SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. 2
Abstract Best Practice and Deployment of the Network for iscsi, NAS and DAS in the Data Center Low Latency, high bandwidth, lossless, spanning tree, IO Convergence, Layer 2, Layer 3, Storage Resource Management,Two Tier, Three Tier, and Flat when deploying iscsi, NAS and DAS what should the network topology be? How can network topology affect Large, Medium and Small Scale network deployments? In this presentation we will discuss best practice and deployments of the network for iscsi, NAS and DAS in the Data Center. 3
Agenda 1 2 3 4 5 6 Scope of this presentation? Definitions The Storage Puzzle Benefits and where are they being deployed? Lossless or Lossy That is the Question Ideal and Non Ideal Ethernet Storage Network Deployments Mapping the Deployment to what is Ideal? 4
Agenda 1 Scope of this presentation? 5
Scope of this presentation? Background The pieces of the puzzle that make up Ethernet network storage deployments. Topologies Ethernet Network topologies that are conducive to storage deployments. Summary and final thoughts. 6
Agenda 2 Definitions The Storage Puzzle 3 6 7
Definitions The Storage Puzzle 8
File Level Storage File Level storage : is the most common storage system that we find with our hard-drives, NAS systems, etc. In this type of storage, the storage disk is configured with a particular protocol (Like NFS, etc) and files are stored and accessed from it as such, in bulk. Advantages of File Level Storage System: File level storage system is simple to implement and simple to use. It stores files and folders and is visible as such, to both the systems storing the files and the systems accessing it. File level storage systems are generally inexpensive, when compared to block level storage systems. File level storage systems are more popular with NAS based storage systems Network Attached Storage. They can be configured with common file level protocols like NTFS (Windows), NFS (Linux), etc. File level storage systems are well suited for bulk file storage. The file level storage device itself can generally handle operations like access control, integration with corporate directories, etc. 9
Block Level Storage Block level storage : is raw blocks (storage volumes) are created and each block can be controlled like an individual hard drive. Generally, these blocks are controlled by the Server based Operating Systems. Each block/ storage volume can be individually formatted with the required file system. Advantages of Block level storage systems: Block level storage systems offer a better performance/ speed than file level storage systems. Each block / storage volume can be treated as an independent disk drive and are controlled by external Server OS. Each block / storage volume can be formatted with the file system required by the application (NFS / NTFS / SMB, etc). Block level storage systems are very popular with SAN Storage Area Networks. Block level storage systems are more reliable, and their transport systems are very efficient. Block level storage can be used to store files and also provide the storage required for special applications like Databases, VMFS (Virtual Machine File Systems), etc. They can support external boot-up of the systems connected to them. 10
Direct-attached Storage (DAS) Direct-attached storage (DAS) refers to a digital storage system directly attached to a server or workstation, without a storage network in between. Protocols used in DAS: ATA, SATA, esata, SCSI, SAS, and Fibre Channel. Pros: Cheep, Compact great for very small business. Con: It is hard to share data or unused resources with other servers. 11
Network-attached storage (NAS) Network-attached storage (NAS) is file-level computer data storage connected to a computer network providing data access to heterogeneous clients. It provides file-based storage. Protocols used in NAS: uses file-based protocols NFS, SMB/CIFS, or AFP. NAS units rarely limit clients to a single protocol. Pros: Often a striped down OS and hardware is needed. Cons: Typically not conducive for large scale deployments. 12
Internet Small Computer System Interface (iscsi) Internet Small Computer System Interface (iscsi)- is a transport for iscsi using TCP/IP (typically native 1GbE, 10GbE, 40GbE and 100GbE) Pros: Cost effect components, hard drives, servers, network cards, cables, switches, optics.. Leverages TCP/IP to guarantee lossless traffic and in order frame delivery. Built in security, authentication with RADIUS servers and can leverage IPSec, MACSec. Cons: Rely on overlay protocols to get lossless and in order frame delivery. Causing complexity and overhead in the network. 13
Fibre Channel over Ethernet (FCoE) Fibre Channel over Ethernet (FCoE) is an encapsulation of Fiber Channel frames over Ethernet networks. Why is this important? FCoE will carry FC frames (eg SCSI commands) directly over Ethernet. This is one of the key drivers that enables SAN LAN convergence. Ethernet HD FC Frame Ethernet FCS FCoE Frame = Ethernet Frame that has an FC frame inside it. 14
Agenda 3 Benefits and where are they being deployed? 15
Benefits of Ethernet Storage COST: Transport ultimately more economical than other storage transports. In-order/ Lossless: TCP/IP = In-order, guaranteed delivery DCB = Lossless, traffic seperation SPEED: Higher bandwidth 10GbE 40GbE 100GbE EQUIPMENT More cost effective Storage equipment Servers Networking equipment 16
Where is it being deployed? FCoE Converged Access Large End to end FCoE & iscsi Medium Business iscsi Small Business 17
Agenda 4 Lossless or Lossy That is the Question 18
Lossless or Lossy.. That is the Question. Very Important Rule: Thy shall never loose or drop a storage frame. Lossless Network Topology: No Frame drop between initiator and target. Lossy Network Topology: The network allows for frame drop within the network and rely on overlay protocols to handle frame drop (eg TCP/IP). 19
Realities of Traditional Ethernet in the Data Center Possible Problem Areas Frame loss Traditionally not meant to transport lossless traffic Forced to rely on protocols like TCP/IP to achieve a lossless topology -> Resulting in protocol overhead/congestion due to retransmission. (iscsi used TCP/IP) Network Topologies Multi-tier Multiple device to manage Spanning Tree adds inefficiencies 50% of links blocked Heavily Oversubscribed 10:1, 5:1, 4:1 take your pick Network Congestion Mix Traffic Types Video Running on Ethernet Voice Running on Ethernet Traditional Applications Running on Ethernet Now add Storage and Server Virtualization to the mix
Storage over Ethernet What do you need? Relying TCP/IP for the Solution. Lossless Network Topology Need: No frame drops between initiator and target Solution : If a frame is dropped TCP/IP will retransmit Result: This adds latency and can exacerbate congestion In order frame delivery Need: Frames must be in order between initiator to target Solution : TCP/IP guarantees inorder delivery per session Fibre Channel s Solution. Lossless Network Topology/ In order frame delivery Need: No frame drops between initiator and target Uses a buffer credit mechanism to create a lossless network. Eliminates frame drop and frame retransmission
Solution : Map storage flows to PFC Creates a lossless network PFC and DCB (dcbx allows the creation of arbitrary application tlv s based on higher level protocol to easily capture and configure for cifs,nfs etc ) Lossless Ethernet Storage Fabric No need for retransmission (Closest way to emulate FC Credit Mechanism) Mapping Different Flows on Separate Priority Result True Traffic Separation
Agenda 5 Ideal and Non Ideal Ethernet Storage Network Deployments 23
Realities of Ethernet in the Data Center The challenges Multi-tier legacy network Too slow Too expensive Too complex Network Complexity TCP/IP can t scale N Unnecessary layers add hops and latency Up to 50% of the ports interconnect switches, not servers or storage Complexity W Up to 75% of traffic E S Spanning Tree disables up to 50% of bandwidth Scale
Non-Ideal Network Topology Ethernet L3 Core Switch/Router Ethernet L2/L3 Aggregation Switch Ethernet L2/L3 Access Switch POD 1 POD 2 POD 3 POD 4 25
Non-Ideal Network Topology Ethernet L3 Core Switch/Router Ethernet L2/L3 Aggregation Switch Ethernet L2/L3 Access Switch POD 1 POD 2 POD 3 POD 4 The Network gets in the way 7 switches 10 chances to drop the frame 7 devices to manage.
Summary of issues.. Three Tier design (access, aggregation and core) design flaws: To many hops add latency. Spanning Tree removes ~50% of the links thus increasing congestion. Lossy fabric rely on overlay protocols like TCP/IP this adding to congestion. 27
Simple solutions.. Three Tier design (access, aggregation and core) design flaws: To many hops add latency. Solution Flatten the Network Topology Spanning Tree removes ~50% of the links thus increasing congestion. Use overlay protocols like MC-LAG Lossy fabric rely on overlay protocols like TCP/IP this adding to congestion. Create a lossless fabric by mapping all storage flows to a lossless queue (use DCB) 28
Traditional Network Topology Aggregation Layer DCB Enabled Storage Array Access Layer DCB Enabled Servers 29
Ideal Network Topology One way.. Storage Array Spine MC-LAG 40GbE/100GbE Leaf 10GbE/40GbE Servers Connected to Leaf/Access Layer Use MC-LAG to remove the ills of Spanning Tree. 30
Ideal Network Topology Desired way.. Flat Fabric Everything is connected to everything (Pooling all resources together) Access is always 1 hop away. Removing the ills of Spanning Tree. 31
Agenda 6 Mapping the Deployment to what is Ideal? 32
Hadoop Network Topology One way Spine MC-LAG 40GbE/100GbE Leaf 10GbE/40GbE 33
Hadoop Network Topology Desired Way. Flat Fabric Note No need for STP everything is connect to everything else one hop away. Best Latency, least amounts of hops. 34
Mixing DAS, NAS, iscsi and FCoE FC SAN DCB Enabled Fabric FC SAN 35
Example: DCB Enabled Fabric/Network Mixing DAS, NAS, iscsi in a DCB Enabled Fabric MC-LAG Note Keep Traffic separated on different priorities on all network devices. Eg) Priority 1 = Tape Traffic (on all switches) Priority 2 = iscsi Traffic (on all switches) Priority 3 = FCoE Traffic (on all switches) Priority 4 = NAS Traffic (on all switches) 36
Example: DCB Enabled Fabric/Network Mixing DAS, NAS, iscsi in a DCB Enabled Fabric DCB Enabled Fabric Note Keep Traffic separated on different priorities on network device. Eg) Priority 1 = Tape Traffic (on all switches) Priority 2 = iscsi Traffic (on all switches) Priority 3 = FCoE Traffic (on all switches) Priority 4 = NAS Traffic (on all switches) Flat Topology: Easy to do operationally when there is only one device to manage 37
Remote office Data Replication LOSSLESS LOSSLESS LOSSLESS Data Center Frames in-fight Backup Data Center PAUSE Things to think about Does your WAN devices have enough buffer to handle frame that are in-flight after pause is sent. Campus Distance correlates directly to the amount of buffer needed. 38
Remote office Data Replication LOSSLESS Data Center LOSSY with Protocol Overlay TCP/IP LOSSLESS Backup Data Center Note WAN Routers should be able buffer several flows to account for frame drop and ensure retransmission. Campus 39
Remote office Data Replication LOSSLESS Data Center LOSSY with Protocol Overlay MPLS/VPLS LOSSLESS Backup Data Center Campus Note WAN links are dedicated to just storage traffic and no other traffic. Speed is pre-negotiated to ensure no packet drop. 40
Final Thoughts Loop in the Network (Spanning Tree) Flatten your fabric to remove loops, this allows better pooling of Ethernet storage. Or Use protocols like MC-LAG (Spin-Leaf) to remove loop. Lossless Fabric/Networks Map flows on the same priority on all network devices. For long distances be mindful of the amount of buffer on the network device. Else frame loss will be introduced. Lossy Fabric/Networks Use a protocol like TCP/IP to ensure proper frame delivery. If you are brave and just want to rely on Ethernet. (Not suggested) Dedicated to just storage traffic and no other traffic. Pre-negotiated speed to ensure no packet drop. 41
Q&A / Feedback Many thanks to the following individuals for their contributions to this tutorial. - SNIA Education Committee Joseph White Simon Gordon Andy Ingram Kishore Inampudi Haruki Sonehara Send any questions or comments on this presentation to SNIA: tracktutorials@snia.org 42