Terabit Networking with JASMIN

Transcription

1 Terabit Networking with JASMIN Jonathan Churchill JASMIN Infrastructure Manager Research Infrastructure Group Scientific Computing Department STFC Rutherford Appleton Labs

2 Terabit Networking with JASMIN What is JASMIN? Why is it needed? 3 year Growth Pains Network design criteria Network design issues ECMP CLOS Architecture Advantages and Disadvantages Some issues we ran into Science DMZ (if time).

3 JASMIN s Purpose CEDA data storage & services Curated data archive Archive management services Archive access services (HTTP, FTP, Helpdesk,...) Data intensive scientific computing Global / regional datasets & models High spatial, temporal resolution Private cloud Flexible access to high-volume & complex data for climate & earth observation communities Online workspaces Services for sharing & collaboration

4 JASMIN is a world leading, unique hybrid of: 16PB high performance storage (~250GByte/s) High-performance computing (~4,000 cores) Non-blocking Networking (> 3Tbit/sec), and Optical Private Network WAN s Coupled with cloud hosting capabilities To address one of NERC s most strategically important challenges: the improvement of predictive environmental science. Prof. Duncan Wingham, NERC Chief Exec.

5 Panasas Parallel file system (cf Lustre, GPFS, pnfs etc) Single Namespace 140GB/sec benchmarked (95 shelves PAS14) Access via PanFS client/nfs/cifs Posix filesystem out of the box. Mounted on Physical machines and VMs 103 shelves PAS shelves PAS Shelves PAS16 Each shelf connected at 10Gb (20Gb PAS14) 2,684 Blades JASMIN - Largest single realm in the world One Management Console TCO: Big Capital, Small Recurrent but JASMIN2 /TB < GPFS/Lustre offerings

6 Three year growth pains X.0/21 = 2,000 IPs X.0/21 Flat Overlaid L2 160->240 10Gb

7 4 x VMware Clusters vjasmin 156 cores, 1.2TB 40x 10Gb 385 x 10Gb Panasas 20PBytes 15.1 PB (usable) Overview 12.5M,38 s, 850Amps, 25 tonnes, 3Terabit/s bandwidth Lotus HPC Cluster 468 x 10Gb 32x 10Gb NetApp+Dell 1010TB + (VM VMDK images) A network : 1,100 10GbE vcloud cores, 1.5TB 12.8TB MPI network (10Gb low latency eth) hosts, 2.2K-3.6K cores. RHEL6, Platform LSF, MPI 40x 10Gb 1&2Gb/s and 10Gb/s: Leeds, UKMO, Archer, (KNMI), CEMS-ISIC

8 Floor Plan Network distributed ~30m x ~20m JASMIN 1 JASMIN 4,5 ( ) ) JASMIN 3 JASMIN 2 Science DMZ

9 Network Design Criteria Non-Blocking (No network contention) Low Latency ( < 20uS MPI. Preferably < 10uS) Small latency spread. Converged (IP storage, SAN storage, Compute, MPI) Gb Expansion to 1,600 ports and beyond wo forklift. Easy to manage and configure Cheap later on: Replaces JASMIN1 240 ports in place.

10 Cabling Costs: Central Chassis Switch 192x 10Gb Cat6 (Power?) 80x Active Optic Twinax 40x Twinax Compute Network Compute 80x Fibre 80x Active Optic Twinax 192x Twinax Compute Network Compute 1,000 Fibre Connections = K JASMIN Gb Connections

11 Cabling Costs: Central Chassis Switch + ToR Compute Network Compute 312 Twinax Fully Populated ToR 6x S4810 ToR Switches 48x Active Optic QSFP ToR e.g Force10 S4810P 48 x 10Gb SFP+ 4x 40Gb QSFP+ 1,000 Fibre Connections = K JASMIN Gb Connections 1:1 Contention ToR 20x S4810 ToR Switches 80x Active Optic QSFP Lots of core 40Gb ports needed. MLAG to 72 Ports?... Chassis switch? expansion/cost

12 1,104 x 10GbE Ports CLOS L3 ECMP OSPF 768 Ports max. no expansion so 12 spines Max 36 leaf switches :1,728 10GbE Non-Blocking. Zero Contention (48x10Gb = 12x 40Gb uplinks) Low Latency (250nS L3 / per switch/router). 7-10uS MPI Cheap!.. (ish)

13 Four routed ECMP hopshops Fast

14

15 ECMP CLOS L3 Advantages Massive scale High performance Low latency with fixed switches Standards based supports multiple vendors Very small blast radius upon network failures Small isolated subnets Deterministic latency with a fixed spine and leaf Pay as you grow start small and increment

16 ECMP CLOS L3 Issues Managing scale: #s of IPs, subnets, VLANs, Cables Monitoring Routed L3 network: Reqs dynamic OSPF routing (100 s routes per switch) No L2 between switches (VMware: SAN s, vmotion) Reqs: DHCP Relay, VXLAN Complex traceroute seen by users.

17 IP and Subnet Management

18 Subnet / IP Management 2x /21 Panasas 4x /24 Internet Connects 55x /26 Fabric Subnets 264x /30 Inter switch links VMs & Growing quickly 288 Servers 2,244 Blades 5,000 IPs & Growing Another 1,000 this month! ~260 VLAN IDs

19 Monitoring / Visualisation Complex Cacti >30 Fabric Switches >50 Mgmt Switches 100 s links to monitor Nagios bloat

20 DHCP Relay Across the Fabric Panasas non-std DHCP port Which interface to monitor in ECMP? Outbound / return not the same. DHCP Relay configs trees.

21 VXLAN ESXi MultiCast PIM IGMP Snooping MTU 50 Byte overhead VMs no MTU 9000 IP ESXi Routing ESXi Auto deploy/dhcp No Panasas Mounts

22 Science DMZ High Speed Data Transfer

23 Questions? Contact: LinkedIn