High Throughput File Servers with SMB Direct, Using the 3 Flavors of RDMA network adapters

Transcription

1

2 High Throughput File Servers with SMB Direct, Using the 3 Flavors of network adapters Jose Barreto Principal Program Manager Microsoft Corporation

3 Abstract In Windows Server 2012, we introduce the SMB Direct protocol, which allows file servers to use high throughput/low latency network interfaces. However, there are three distinct flavors of, each with their own specific requirements and advantages, their own pros and cons. In this session, we'll look into iwarp, InfiniBand and RoCE, outline the differences between them. We'll also list the specific vendors that offer each technology and provide step-by-step instructions for anyone planning to deploy them. The talk will also include an update on performance and a customer case study.

4 Summary Overview of SMB Direct (SMB over ) Three flavors of Setting up SMB Direct SMB Direct Performance SMB Direct Case Study

5

6 SMB Direct (SMB over ) New class of SMB file storage for the Enterprise Minimal CPU utilization for file storage processing Low latency and ability to leverage high speed s Fibre Channel-equivalent solution at a lower cost File Client Application File Server Traditional advantages of SMB file storage Easy to provision, manage and migrate Leverages converged network No application change or administrator configuration SMB Client User Kernel SMB Server Required hardware -capable network interface (R-) Support for iwarp, InfiniBand and RoCE Network w/ support Network w/ support NTFS SCSI Uses SMB Multichannel for Load Balancing/Failover R- R- Disk

7 What is? Remote Direct Memory Access Protocol Accelerated IO delivery model which works by allowing application software to bypass most layers of software and communicate directly with the hardware benefits Low latency High throughput Zero copy capability OS / Stack bypass Client SMB Client SMB Direct Memory Memory SMB Server SMB Direct File Server Hardware Technologies Infiniband iwarp: over TCP/IP RoCE: over Converged Ethernet NDKPI NDKPI Ethernet or InfiniBand

8 SMB over TCP and User Kernel Client 1 Unchanged API Application SMB Client 2 Memory File Server 4 1 Memory SMB Server 1. Application (Hyper-V, SQL Server) does not need to change SMB client makes the decision to use SMB Direct at run time TCP/ IP NDKPI SMB Direct SMB Direct NDKPI TCP/ IP 3 3. NDKPI provides a much thinner layer than TCP/IP 3 4. Remote Direct Memory Access performed by the network interfaces. 4 Ethernet and/or InfiniBand

9

10 Low CPU Utilization under load Low latency Comparing Technologies Type (Cards*) Pros Cons Non- Ethernet (wide variety of s) TCP/IP-based protocol Works with any Ethernet switch Wide variety of vendors and models Support for in-box teaming (LBFO) Currently limited to 10Gbps per port High CPU Utilization under load High latency iwarp (Intel NE020*, Chelsio T4) TCP/IP-based protocol Works with any 10GbE switch traffic routable Currently limited to 10Gbps per port* RoCE (Mellanox ConnectX-2, Mellanox ConnectX-3*) Ethernet-based protocol Works with high-end 10GbE/40GbE switches Offers up to 40Gbps per port today* traffic not routable via existing IP infrastructure Requires DCB switch with Priority Flow Control (PFC) InfiniBand (Mellanox ConnectX-2, Mellanox ConnectX-3*) Offers up to 54Gbps per port today* Switches typically less expensive per port than 10GbE switches* Switches offer 10GbE or 40GbE uplinks Commonly used in HPC environments Not an Ethernet-based protocol traffic not routable via existing IP infrastructure Requires InfiniBand switches Requires a subnet manager (on the switch or the host) * This is current as of the release of Windows Server 2012 RC. Information on this slide is subject to change as technologies evolve and new cards become available.

11 Mellanox ConnectX -3 dual-port Adapter with VPI (InfiniBand and Ethernet) Mellanox provides end-to-end InfiniBand and Ethernet connectivity solutions (adapters, switches, cables) Connecting data center servers and storage Up to 56Gb/s InfiniBand and 40Gb/s Ethernet per port Low latency, Low CPU overhead, InfiniBand to Ethernet Gateways for seamless operation Windows Server 2012 exposes the great value of InfiniBand for storage traffic, virtualization and low latency InfiniBand and Ethernet (with RoCE) integration Highest Efficiency, Performance and return on investment For more information: Gilad Shainer,

12 Intel 10GbE iwarp Adapter - NE020 In production today Supports Microsoft s MPI via ND in Windows Server 2008 R2 and beyond See Intel s Download site ( for drivers (search NE020 ) Drivers inbox since Beta for Windows Server 2012 Supports Microsoft s SMB Direct via NDK Uses the IETF s iwarp technology that is built on top of IP The only WAN-routable, cloud-ready technology Uses standard ethernet switches Beta drivers available from Intel s Download site ( for drivers (search NE020 ) For more information: david.l.fair@intel.com

13 Chelsio T4 line of 10GbE adapters (iwarp) Contact:

14

15 Setting up SMB Direct Install hardware and drivers Get-NetAdapter Get-NetAdapterRdma Configure IP addresses Get-SmbServerNetworkInterface Get-SmbClientNetworkInterface Establish an SMB Connection Get-SmbConnection Get-SmbMultichannelConnection Verify client Performance Counters Activity 1/interface SMB Direct Connection 1/connection SMB Client Shares 1/share Verify server Performance Counters Activity 1/interface SMB Direct Connection 1/connection SMB Server Shares 1/share SMB Server Session 1/session Similar to configuring SMB for regular network interfaces

16 Cards Mellanox ConnectX-2 Mellanox ConnectX-3 InfiniBand details Configure a subnet manager on the switch Using a managed switches with a built-in subnet manager Or use OpenSM on Windows Server 2012 Included as part of the Mellanox package New-Service Name "OpenSM" BinaryPathName "`"C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe`" --service -L 128" - DisplayName "OpenSM" Description "OpenSM" -StartupType Automatic

17 iwarp details Cards Intel NE020 Chelsio T4 Configure the firewall SMB Direct with iwarp uses TCP port 5445 Enable-NetFirewallRule FPSSMBD-iWARP-In-TCP Allow cross-subnet access (optional) iwarp technology can be routed across IP subnets Set-NetOffloadGlobalSetting -NetworkDirectAcrossIPSubnets Allow

18 RoCE details Cards Mellanox ConnectX-3 Make sure to configure the for Ethernet Configuring Priority Flow Control (PFC) on Windows Install-WindowsFeature Data-Center-Bridging New-NetQosPolicy RoCE NetDirectPortMatchCondition PriorityValue8021Action 4 Enable-NetQosFlowControl Priority 4 Enable-NetAdapterQos InterfaceAlias 1 Set-NetQosDcbxSetting willing 0 New-NetQoSTrafficClass "RoCE" -Priority 4 -Bandwidth 60 -Algorithm ETS Configuring PFC on the Switch

19

20 SMB Direct Performance 1 x 54GbIB SMB Client IO Micro Benchmark SMB Client IO Micro Benchmark SMB Client IO Micro Benchmark 10GbE IB QDR IB FDR Single Server IO Micro Benchmark SMB Server 10 GbE SMB Server IB QDR SMB Server IB FDR Fusion Fusion Fusion Fusion IO IO IO IO Fusion Fusion Fusion Fusion IO IO IO IO Fusion Fusion Fusion Fusion IO IO IO IO Fusion Fusion Fusion Fusion IO IO IO IO

21 SMB Direct Performance 1 x 54GbIB Workload: 512KB IOs, 8 threads, 8 outstanding Configuration BW MB/sec IOPS 512KB IOs/sec %CPU Privileged Non- (Ethernet, 10Gbps) 1,129 2,259 ~9.8 (InfiniBand QDR, 32Gbps) 3,754 7,508 ~3.5 (InfiniBand FDR, 54Gbps) 5,792 11,565 ~4.8 Local 5,808 11,616 ~6.6 Workload: 8KB IOs, 16 threads, 16 outstanding Configuration BW MB/sec IOPS 8KB IOs/sec %CPU Privileged Non- (Ethernet, 10Gbps) ,160 ~21.0 (InfiniBand QDR, 32Gbps) 2, ,446 ~85.9 (InfiniBand FDR, 54Gbps) 2, ,388 ~84.7 Local 4, ,225 ~90.4 *** Preliminary *** results from two Intel Romley machines with 2 sockets each, 8 cores/socket Both client and server using a single port of a Mellanox network interface PCIe Gen3 x8 slot Data goes all the way to persistent storage, using 4 FusionIO iodrive 2 cards Preliminary results based on Windows Server 2012 beta

22 File Client (SMB 3.0) SMB Direct Performance 2 x 54GbIB Single Server SQLIO File Server (SMB 3.0) SQLIO Hyper-V (SMB 3.0) File Server (SMB 3.0) VM SQLIO RAID Controller RAID Controller RAID Controller RAID Controller RAID Controller RAID Controller RAID Controller RAID Controller RAID Controller RAID Controller RAID Controller RAID Controller

23 SMB Direct Performance 2 x 54GbIB Configuration BW MB/sec IOPS 512KB IOs/sec %CPU Privileged Latency milliseconds 1 Local 10,090 38,492 ~2.5% ~3ms 2 Remote 9,852 37,584 ~5.1% ~3ms 3 - Remote VM 10,367 39,548 ~4.6% ~3 ms Preliminary results based on Windows Server 2012 RC

24 SMB Direct Performance

25 SMB Direct Performance 3 x 54GbIB File Client (SMB 3.0) SQLIO Workload BW MB/sec IOPS IOs/sec %CPU Privileged Latency milliseconds File Server (SMB 3.0) Storage Spaces 512KB IOs, 100% read, 2t, 8o 16,778 32,002 ~11% ~ 2 ms 8KB IOs, 100% read, 16t, 2o 4, ,665 ~65% < 1 ms RAID Controller HBA HBA HBA HBA HBA Preliminary results based on Windows Server 2012 RC

26

27 Case Study

28

29 Summary Overview of SMB Direct (SMB over ) Three flavors of Setting up SMB Direct SMB Direct Performance SMB Direct Case Study

30 Related Content Blog Posts TechEd Talks WSV328 The Path to Continuous Availability with Windows Server 2012 VIR306 Hyper-V over SMB: Remote File Storage Support in Windows Server 2012 Hyper-V WSV314 Windows Server 2012 Teaming and SMB Multichannel Solutions WSV334 Windows Server 2012 File and Storage Services Management WSV303 Windows Server 2012 High-Performance, Highly-Available Storage Using SMB WSV330 How to Increase SQL Availability and Performance Using WS 2012 SMB 3.0 Solutions WSV410 Continuously Available File Server: Under the Hood WSV310 Windows Server 2012: Cluster-in-a-Box,, and More

31