Experiences with Wide-Area RDMA: The most fun you can have with your WAN

Transcription

1 Experiences with Wide-Area RDMA: The most fun you can have with your WAN Ezra Kissel, Martin Swany Indiana University July 16 th 2012 Internet 2 Joint Techs

2 High-performance data movement Increasing demand on networks Better connectivity, more data Petabyte-scale computing Increasing WAN capacity 10G -> 40G -> 100G Virtual circuits, SDN, smarter networks End of CPU frequency scaling 10Gb/s TCP fully taxes a core New approaches needed Known issues surrounding existing transport protocols over LFNs 7/16/2012 Summer 2012 Internet2 Joint Techs 2

3 Phoebus A performance-enhancing network service based on XSP extensible Session Protocol XSP sessions allow for interaction and control between applications/services Negotiate features, metadata exchange Phoebus aims to help bridge the end-to-end performance gap Shorter RTTs from the edge to PG Automatic tuning on a per-segment basis Use of alternative protocols over the WAN Standard interfaces at the edge, innovation in the core 7/16/2012 Summer 2012 Internet2 Joint Techs 3

4 Session Layer Burst Switching (SLaBS) Apply burst switching concepts at gateways (Phoebus) Take advantage of large, fast buffers in the network Eliminate contention across WAN segments, avoid congestive loss Multiplex incoming connections into right-sized bursts, called slabs Use more efficient protocols, reduce overhead ADDRESS RKEY * 0x70df x65a3a x39c4d /16/2012 Summer 2012 Internet2 Joint Techs 4

5 Improving performance Reading and writing to/from buffer medium can be slow Fast memory is needed User space copying incurs significant overhead Traditional transport mechanisms are scaling poorly Socket splicing can minimize overheads in some cases Hypothesis: We can reduce overheads by adopting a low-level protocol using an asynchronous communication pattern Our solution: Bulk Asynchronous Get (BAG) Operate on in-memory chunks of data, as in slabs Exchange data locations (i.e. metadata) ahead of the transfer A consumer remotely gets data from the producer when ready Acknowledgements are sent out-of-band 7/16/2012 Summer 2012 Internet2 Joint Techs 5

6 Remote Direct Memory Access (RDMA) Connects virtual buffers which may be located in different physical address space even across a network No kernel buffer copies or OS context switching to transfer data Virtual-to-physical address translation done by the NIC 7/16/2012 Summer 2012 Internet2 Joint Techs 6

7 RDMA over Converged Ethernet (RoCE) Infiniband made to work over existing Ethernet networks Annex A16 of IB Architecture Specification Converged Ethernet (i.e. Data Center Bridging) is not a requirement for RoCE Will work over WANs Requires certain guarantees about path characteristics 7/16/2012 Summer 2012 Internet2 Joint Techs 7

8 RDMA operations Infiniband supports reliable (RC) and unreliable (UC) RDMA connections We are interested in the one-sided channel semantics RDMA READ/WRITE : Application pins memory, exchanges pointers and keys, and then posts the desired operations RDMA READ is the Get in BAG All completed RDMA events are written to a completion queue Queue is polled for events All operations are asynchronous Supported through IB verbs and connection manager libraries ibverbs, rdmacm OpenFabrics Enterprise Distribution (OFED) 7/16/2012 Summer 2012 Internet2 Joint Techs 8

9 Filling the pipe when sending real data Must pipeline operations to keep the network saturated Decouple reading and writing to/from buffer regions Fill the transmit queue Configurable ring buffer keeps data in-flight ready partitions can be posted as soon as possible Further partitioned based on demand 7/16/2012 Summer 2012 Internet2 Joint Techs 9

10 Bulk Asynchronous Get with SLaBS SLAB buffer information is communicated over XSP control channel Pipeline Get operations up to transmit queue depth Wait for SLAB acknowledgements Consumer performs RDMA READ to copy remote buffers SLAB buffering also applied for end-to-end applications GridFTP driver via XIO framwork 7/16/2012 Summer 2012 Internet2 Joint Techs 10

11 SC11 SRS Demo (10G network) 7/16/

12 SC11 SRS Demo Tests (78ms path) Native RDMA application #./xfer_test -c t 180 -i 2 -a 4 -o 27 r x sc-viz/ : port=18515 ib_port=1 tx_depth=16 sl=0 duplex=0 cma=1 Created SLAB buffer with SIZE: PARTITIONS: 4 raddr: 0x2aaaaf6ed010, laddr: 0x2aaaaf873010, size: raddr: 0x2aaab16ee010, laddr: 0x2aaab , size: raddr: 0x2aaab36ef010, laddr: 0x2aaab , size: raddr: 0x2aaab56f0010, laddr: 0x2aaab , size: Metadata exchange complete [ sec] 2.32 GB 9.26 Gb/s [ sec] 2.42 GB 9.66 Gb/s [ sec] 2.45 GB 9.79 Gb/s CPU core utilization 70-90% - 1 stream TCP 3-4% - RDMA GridFTP using Phoebus-SLaBS RDMA Gateway # globus-url-copy -t 60 -vb -p 1 -dcstack phoebus:'phoebus_path= /5006# /5006, xsp: xsp_hop=sc-viz/5006;xsp_net_path=openflow ftp:// :2811/dev/zero ftp:// :2811/dev/null Source: ftp:// :2811/dev/ Dest: ftp:// :2811/dev/ zero -> null bytes MB/sec avg MB/sec inst [ 9.3 Gb/s ] 7/16/2012 Summer 2012 Internet2 Joint Techs 12

13 Tunable environments 6-node, 10G connected testbed Mellanox ConnectX EN and VPI adapters, native RoCE support Netem and PSpacer enable latency and bottleneck emulation Spirent XGEM network impairment device Validates netem XGEM 7/16/2012 Summer 2012 Internet2 Joint Techs 13

14 Phoebus-SLaBS performance GridFTP transfers over dedicated 10G path, increasing WAN latency, 4ms LAN RTT and.001% edge loss 7/16/2012 Summer 2012 Internet2 Joint Techs 14

15 RoCE over bottleneck links 120ms WAN latency 7/16/2012 Summer 2012 Internet2 Joint Techs 15

16 Overhead reduction Linux perf profiling of SLaBS buffer implementation. Comparing TCP to splicing and RDMA transport. Comparing CPU load of SLaBS gateway using TCP, TCP-splicing, and RDMA. 1. Ezra Kissel and Martin Swany. Evaluating High Performance Data Transfer with RDMA-based Protocols in Wide-Area Networks, In Proceeding of the 14th IEEE International Conference on High Performance Computing and Communications (HPCC-12), June /16/2012 Summer 2012 Internet2 Joint Techs 16

17 Summary In short: RoCE can work well over the WAN Reduced overhead, consistently good performance Requires rnics, clean paths, proper buffer management Bulk Asynchronous Get Take advantage of RDMA semantics to decouple OS involvement Out-of-band signaling with XSP RDMA for Phoebus-SLaBS Transparently brings RDMA over WAN advantages to existing applications 40G+ testing in progress over ANI testbed 39Gb/s over 8000 mile path (96ms) 7/16/2012 Summer 2012 Internet2 Joint Techs 17

18 Acknowledgements and resources Supported in part by NSF grant nos. OCI , OCI , and CNS Energy Sciences Network Inder Monga, Brian Tierney, Eric Pouyoul Prototype Phoebus deployment (1G) on Internet /16/2012 Summer 2012 Internet2 Joint Techs 18