Interconnecting Future DoE leadership systems

Transcription

1 Interconnecting Future DoE leadership systems Rich Graham HPC Advisory Council, Stanford, 2015

2 HPC The Challenges 2

3 Proud to Accelerate Future DOE Leadership Systems ( CORAL ) Summit System Sierra System 5X 10X Higher Application Performance versus Current Systems Mellanox EDR 100Gb/s InfiniBand, IBM POWER CPUs, NVIDIA Tesla GPUs Mellanox EDR 100G Solutions Selected by the DOE for 2017 Leadership Systems Deliver Superior Performance and Scalability over Current / Future Competition 3

4 Exascale-Class Computer Platforms Communication Challenges Challenge Very large functional unit count ~10M Large on- node functional unit count ~500 Deeper memory hierarchies Smaller amounts of memory per functional unit May have functional unit heterogeneity Component failures part of normal operation Data movement is expensive Solution focus Scalable communication capabilities: point-topoint & collectives Scalable Network: Adaptive routing Scalable HCA architecture Cache aware network access Low latency, high b/w capabilities Support for data heterogeneity Resilient and redundant stack Optimize data movement Independent remote progress Independent hardware progress Power costs Power aware hardware 4

5 Standardize your Interconnect 5

6 Technology Roadmap Always One-Generation Ahead Mellanox 20Gbs 40Gbs 56Gbs 100Gbs 200Gbs Terascale Petascale Exascale 3 rd TOP Virginia Tech (Apple) 1 st Roadrunner Mellanox Connected Mega Supercomputers

7 The Future is Here Enter the World of Scalable Performance At the Speed of 100Gb/s! 7

8 The Future is Here Entering the Era of 100Gb/s 100Gb/s Adapter, 0.7us latency 150 million messages per second (10 / 25 / 40 / 50 / 56 / 100Gb/s) 36 EDR (100Gb/s) Ports, <90ns Latency Throughput of 7.2Tb/s Copper (Passive, Active) Optical Cables (VCSEL) Silicon Photonics 8

9 Enter the World of Scalable Performance 100Gb/s Switch Switch-IB: Highest Performance Switch in the Market 7 th Generation InfiniBand Switch Analyze 36 EDR (100Gb/s) Ports, <90ns Latency Throughput of 7.2 Tb/s InfiniBand Router Adaptive Routing Store 9

10 Enter the World of Scalable Performance 100Gb/s Adapter ConnectX-4: Highest Performance Adapter in the Market InfiniBand: SDR / DDR / QDR / FDR / EDR Ethernet: 10 / 25 / 40 / 50 / 56 / 100GbE Connect. Accelerate. Outperform 100Gb/s, <0.7us latency 150 million messages per second OpenPOWER CAPI technology CORE-Direct technology GPUDirect RDMA Dynamically Connected Transport (DCT) Ethernet offloads (HDS, RSS, TSS, LRO, LSOv2) 10

11 Higher is Better Bandwidth (MBytes/sec) Bandwidth (MBytes/sec) Connect-IB Provides Highest Interconnect Throughput Unidirectional Bandwidth Bidirectional Bandwidth ConnectX2-PCIe2-QDR ConnectX3-PCIe3-FDR Sandy-ConnectIB-DualFDR Ivy-ConnectIB-DualFDR K 16K 64K 256K 1M K 16K 64K 256K 1M Message Size (bytes) Message Size (bytes) Source: Prof. DK Panda Gain Your Performance Leadership With Connect-IB Adapters 11

12 Standard! Standardized wire protocol Mix and match of vertical and horizontal components Ecosystem build together Open-source, extensible interfaces Extend and optimize per applications needs 12

13 Extensible Open Source Framework Application Protocols/Accelerators TCP/IP Sockets RPC Packet processing Extended Verbs Storage MPI / SHMEM / UPC Vendor Extensions HW A HW B HW C HW D 13

14 Scalability 14

15 Latency (us) Latency (us) Collective Operations Reduce Collective 3000 Topology Aware Hardware Multicast processes (PPN=8) Without FCA With FCA Separate Virtual Fabric Offload Scalable algorithms Barrier Collective Processes (PPN=8) Without FCA With FCA 15

16 Process p-1 Process 1 p*(cs+cr)/2 Process 0 The DC Model Dynamic Connectivity Each DC Initiator can be used to reach any remote DC Target No resources sharing between processes process controls how many (and can adapt to load) process controls usage model (e.g. SQ allocation policy) no inter-process dependencies cs cr cs cr Resource footprint Function of node and HCA capability Independent of system size cs Fast Communication Setup Time cr cs concurrency of the sender cr=concurrency of the responder 16 16

17 Dynamically Connected Transport Key objects DC Initiator: Initiates data transfer DC Target: Handles incoming data 17

18 Host Memory Consumption (MB) Dynamically Connected Transport Exascale Scalability 1,000,000,000 1,000,000 8 nodes 2K nodes 10K nodes 100K nodes 1,000 1 InfiniHost, RC 2002 InfiniHost-III, SRQ 2005 ConnectX, XRC 2008 Connect-IB, DCT

19 Reliable Connection Transport Mode 19

20 Dynamically Connected Transport Mode 20

21 More QoS Servers Topologies l l Administrator m b h 1 HCA 1..h h r HCA 1..h IPC InfiniBand Subnet Storage Routers Net. Mgt. Filer Gateway Distance Storage FAT Tree SM FAT Tree SM FAT Tree SM 21

22 Scalability Under Load 22

23 Adaptive Routing Purpose Improved Network utilization: choose alternate routes on congestion Network resilience: Alternative routes on failure Supported Hardware SwitchX-2 Switch-IB: Adaptive routing notification added 23

24 Mellanox Adaptive Routing Hardware Support For every incoming packet the adaptive routing process has two main stages Route Change Decision (to adapt or not to adapt) New output port selection Mellanox hardware is NOT topology specific SDN concept separates the configuration plane from the data plane Every feature is software controlled Fat-Tree, Dragonfly and Dragonfly+ are fully supported New hardware features introduced to support Dragonfly and Dragonfly+ 24

25 Is the Packet Allowed to Adapt? AR Modes Static traffic is always bound to a specific port Time-Bound traffic is bound to the last port used if not more than Tb [sec] passed since that event Free traffic may select a new out port freely Packets are classified to be either Legacy, Restricted or Unrestricted Destinations are classified to be either Legacy, Restricted, Timely-Restricted or Unrestricted A matrix maps possible combinations of packet and destination based classification to AR modes 25

26 Mellanox Adaptive Routing Notification (ARN) The reaction time is critical to Adaptive Routing Traffic modes change fast A better AR decision requires some knowledge about network state Internel switch to switch communications Faster convergence after routing changes ARN Fast notification to decision point Fully configurable (topology agnostic) 3) ARN cause new output port selection 2) ARN forwarded, can t fix here 1) Sampling prefers the large flows Faster Routing Modifications, Resilient Network 26

27 Network Offload 27

28 Scalability of Collective Operations Ideal Algorithm Impact of System Noise

29 Scalability of Collective Operations - II Offloaded Algorithm Nonblocking Algorithm - Communication processing 29

30 CORE-Direct Scalable collective communication Asynchronous communication Manage communication by communication resources Avoid system noise Task list Target QP for task Operation Send Wait for completions Enable Calculate 30

31 Example Four Process Recursive Doubling Step Step

32 Four Process Barrier Example Using Managed Queues Rank 0 32

33 Nonblocking Alltoall (Overlap-Wait) Benchmark CoreDirect Offload allows Alltoall benchmark with almost 100% compute 33

34 Optimizing Non Contiguous Memory Transfers Support combining contiguous registered memory regions into a single memory region. H/W treats them as a single contiguous region (and handles the non-contiguous regions) For a given memory region, supports non-contiguous access to memory, using a regular structure representation base pointer, element length, stride, repeat count. Can combine these from multiple different memory keys Memory descriptors are created by posting WQE s to fill in the memory key Supports local and remote non-contiguous memory access Eliminates the need for some memory copies 34

35 Optimizing Non Contiguous Memory Transfers 35

36 On Demand Paging No memory pinning, no memory registration, no registration caches! Advantages Greatly simplified programming Unlimited MR sizes Physical memory optimized to hold current working set 0x1000 0x2000 0x3000 0x4000 0x5000 0x6000 Address Space PFN1 PFN2 PFN3 PFN4 PFN5 PFN6 IO Virtual Address 0x1000 0x2000 0x3000 0x4000 0x5000 0x6000 ODP promise: IO virtual address mapping == Process virtual address mapping 36

37 Connecting Compute Elements TODO: x86-> +GPU -> +ARM/POWER 37

38 1 1 GPUDirect RDMA Receive Transmit System Memory 1 CPU GPUDirect 1.0 CPU System Memory GPU Chip set Chip set GPU InfiniBand InfiniBand GPU Memory GPU Memory System Memory 1 CPU CPU System Memory GPU Chip set Chip set GPU GPU Memory InfiniBand GPUDirect RDMA InfiniBand GPU Memory 38

39 Mellanox PeerDirect with NVIDIA GPUDirect RDMA HOOMD-blue is a general-purpose Molecular Dynamics simulation code accelerated on GPUs GPUDirect RDMA allows direct peer to peer GPU communications over InfiniBand Unlocks performance between GPU and InfiniBand This provides a significant decrease in GPU-GPU communication latency Provides complete CPU offload from all GPU communications across the network Demonstrated up to 102% performance improvement with large number of particles 102% 39

40 Storage 40

41 K 4K IO Size Disc access 4K IO [micsec] InfiniBand RDMA Storage Get the Best Performance! Transport protocol implemented in hardware Zero copy using RDMA only 131K IOPs iscsi (TCP/IP) 1 x FC 8 Gb port 4 x FC 8 Gb port iser 1 x 40GbE/IB Port iser 2 x 40GbE/IB Port (+Acceleration) KIOPs IOPs iscsi/rdma 5-10% the latency under 20x the workload 41

42 Data Protection Offloaded Used to provide data block integrity check capabilities (CRC) for block storage (SCSI) Proposed by the T10 committee DIF extends the support to main memory 42

43 Power 43

44 Motivation For Power Aware Design Today networks work at maximum capacity with almost constant dissipation Low power silicon devices may not suffice to meet future requirements for low-energy networks The electricity consumption of datacenters is a significant contributor to the total cost of operation Cooling costs scale as 1.3X the total energy consumption Lower power consumption lowers OPEX Annual OPEX for 1KW is ~$1000 * According to Global e-sustainability Initiative (GeSI) if green network technologies (GNTs) are not adopted 44

45 Increasing Energy Efficiency with Mellanox Switches Voltage/Frequency scaling Dynamic array power-save ASIC level Width/Speed reduction Energy Efficient Ethernet Link level Aggregate Power Aware API FW / MLNX-OS / UFM Port/Module/System shutdown Switch Level Energy aware cooling control System Level Low Energy COnsumption NETworks 45

46 power relative to highest speed Prototype Results - Power Link Level Speed Reduction Power Save [SRPS] % 95.00% 90.00% 85.00% 80.00% 75.00% 70.00% 65.00% 60.00% FDR (56Gb-IB} QDR(40Gb-IB} SDR(10Gb-IB} speed Width Reduction Power Save [WRPS] 46

47 Total SX power [W] power relative to full connectivity Prototype Results - Power System Level Internal port shutdown (Director switch) - ~1%/port Load based power scaling Total system power for SX % 95.00% 90.00% 85.00% 80.00% 75.00% 70.00% 65.00% 60.00% Power decrease percentage due to internal port closing number of closed ports Fan system analysis for improved power algorithm % 20% 30% 40% 50% 60% 70% 80% 90% 100% %BW per port WRPS enable (auto mode) WRPS disable 47

48 Monitoring and Diagnostics 48

49 Unified Fabric Manager Automatic Discovery Central Device Mgmt Fabric Dashboard Congestion Analysis Health & Perf Monitoring Advanced Alerting Fabric Health Reports Service Oriented Provisioning 49

50 ExaScale! 50

51 Mellanox InfiniBand Connected Petascale Systems Connecting Half of the World s Petascale Systems Mellanox Connected Petascale System Examples 51

52 The Only Provider of End-to-End 40/56Gb/s Solutions Comprehensive End-to-End InfiniBand and Ethernet Portfolio ICs Adapter Cards Switches/Gateways Host/Fabric Software Metro / WAN Cables/Modules From Data Center to Metro and WAN X86, ARM and Power based Compute and Storage Platforms The Interconnect Provider For 10Gb/s and Beyond 52

53 Thank You