CONGA: Distributed Congestion-Aware Load Balancing for Datacenters

Transcription

1 CONGA: Distributed Congestion-Aware Load Balancing for Datacenters Mohammad Alizadeh Tom Edsall, Sarang Dharmapurikar, Ramanan Vaidyanathan, Kevin Chu, Andy Fingerhut, Vinh The Lam, Francis Matus, Rong Pan, Navindra Yadav, George Varghese 1

2 Motivation DC networks need large bisection bandwidth for distributed apps (big data, HPC, web services, etc) Single-rooted tree Ø High oversubscription Core Agg Access 1000s of server ports 2

3 Motivation DC networks need large bisection bandwidth for distributed apps (big data, HPC, web services, etc) Multi-rooted tree [Fat-tree, Leaf-Spine, ] Ø Full bisection bandwidth, achieved via multipathing Spine Leaf 1000s of server ports 3

4 Motivation DC networks need large bisection bandwidth for distributed apps (big data, HPC, web services, etc) Multi-rooted tree [Fat-tree, Leaf-Spine, ] Ø Full bisection bandwidth, achieved via multipathing Spine Leaf 1000s of server ports 4

5 Multi-rooted!= Ideal DC Network Ideal DC network: Big output-queued switch 1000s of server ports Ø No internal bottlenecks è predictable Ø Simplifies BW management [EyeQ, FairCloud, pfabric, Varys, ] 5

6 Multi-rooted!= Ideal DC Network Ideal DC network: Big output-queued switch Multi-rooted tree Can t build it L 1000s of server ports 1000s of server ports Ø No internal bottlenecks è predictable Ø Simplifies BW management [EyeQ, FairCloud, pfabric, Varys, ] Possible bottlenecks 6

7 Multi-rooted!= Ideal DC Network Ideal DC network: Big output-queued switch Multi-rooted tree Can t build it L 1000s of server ports 1000s of server ports Ø No internal bottlenecks è predictable Ø Simplifies BW management [EyeQ, FairCloud, pfabric, Varys, ] Need precise load balancing 7

8 Today: ECMP Load Balancing Pick among equal-cost paths by a hash of 5-tuple Ø Approximates Valiant load balancing Ø Preserves packet order H(f) % 3 = 0 8

9 Today: ECMP Load Balancing Pick among equal-cost paths by a hash of 5-tuple Ø Approximates Valiant load balancing Ø Preserves packet order Problems: - Hash collisions (coarse granularity) H(f) % 3 = 0 9

10 Today: ECMP Load Balancing Pick among equal-cost paths by a hash of 5-tuple Ø Approximates Valiant load balancing Ø Preserves packet order Problems: - Hash collisions (coarse granularity) - Local & stateless (v. bad with asymmetry due to link failures) 10

11 Dealing with Asymmetry Handling asymmetry needs non-local knowledge 11

12 Dealing with Asymmetry Handling asymmetry needs non-local knowledge 12

13 Dealing with Asymmetry Handling asymmetry needs non-local knowledge 13

14 Dealing with Asymmetry Scheme Thrput 30G (UDP) 30G ECMP (Local Stateless) Local Cong- Aware Global Cong- Aware (TCP) 14

15 Dealing with Asymmetry: ECMP Scheme Thrput 30G (UDP) 30G ECMP (Local Stateless) Local Cong- Aware 60G Global Cong- Aware 10G (TCP) 20G 15

16 Dealing with Asymmetry: Local Congestion-Aware Scheme Thrput 30G (UDP) 30G ECMP (Local Stateless) Local Cong- Aware 60G Global Cong- Aware 10G (TCP) 20G Interacts poorly with TCP s control loop 16

17 Dealing with Asymmetry: Local Congestion-Aware Scheme Thrput 30G (UDP) 30G ECMP (Local Stateless) Local Cong- Aware 60G 50G Global Cong- Aware 10G (TCP) 10G Interacts poorly with TCP s control loop 17

18 Dealing with Asymmetry: Global Congestion-Aware Scheme Thrput 30G (UDP) 30G ECMP (Local Stateless) Local Cong- Aware 60G 50G Global Cong- Aware 10G (TCP) 10G 18

19 Dealing with Asymmetry: Global Congestion-Aware Scheme Thrput 30G (UDP) 30G ECMP (Local Stateless) Local Cong- Aware 60G 50G Global Cong- Aware 70G 5G (TCP) 35G 19

20 Dealing with Asymmetry: Global Congestion-Aware Scheme Thrput 30G (UDP) 30G ECMP (Local Stateless) Local Cong- Aware 60G 50G Global Cong- Aware 70G 5G (TCP) Global CA > ECMP > Local CA 35G Local congestion-awareness can be worse than ECMP 20

21 Global Congestion-Awareness (in Datacenters) Datacenter Opportunity Challenge Latency Topology Traffic microseconds simple, regular volatile, bursty 21

22 Global Congestion-Awareness (in Datacenters) Datacenter Opportunity Challenge Latency Topology Traffic microseconds Simple & simple, Stable regular volatile, Responsive bursty Key Insight: Use extremely fast, low latency distributed control 22

23 CONGA in 1 Slide 1. Leaf switches (top-of-rack) track congestion to other leaves on different paths in near real-time 2. Use greedy decisions to minimize bottleneck util Fast feedback loops between leaf switches, directly in dataplane L0 L1 L2 23

24 CONGA S DESIGN 24

25 Design CONGA operates over a standard DC overlay (VXLAN) Ø Already deployed to virtualize the physical network VXLAN encap. L0è L2 H1è H9 L0è L2 H1è H9 L0 L1 L2 H1 H2 H3 H4 H5 H6 H7 H8 H9 25

26 Design: Leaf-to-Leaf Feedback Track path-wise congestion metrics (3 bits) between each pair of leaf switches Dest Leaf L1 L2 Path CongesPon- To- Leaf Rate Measurement Module measures link utilization L L1 L2 H1 H2 H3 H4 H5 H6 H7 H8 H9 26

27 Design: Leaf-to-Leaf Feedback Track path-wise congestion metrics (3 bits) between each pair of leaf switches Dest Leaf L1 L2 Path CongesPon- To- Leaf L0è L2 Path=2 CE=0 3 L L0è L2 Path=2 CE=0 CE=5 L1 pkt.ce ç max(pkt.ce, link.util) L2 Src Leaf L0 L1 Path CongesPon- From- Leaf L0è L2 Path=2 CE=5 5 H1 H2 H3 H4 H5 H6 H7 H8 H9 27

28 Design: Leaf-to-Leaf Feedback Track path-wise congestion metrics (3 bits) between each pair of leaf switches Dest Leaf L1 L2 Path Src Leaf L0 L1 Path CongesPon- To- Leaf L0è L2 L2è L0 FB- Path=2 FB- Metric=5 CE=0 L L1 L2 CongesPon- From- Leaf H1 H2 H3 H4 H5 H6 H7 H8 H9 28

29 Design: LB Decisions Send each packet on least congested path flowlet [Kandula et al 2007] Dest Leaf L1 L2 Path CongesPon- To- Leaf L0 è L1: p * = 3 L0 è L2: p * = 0 or L0 L1 L2 H1 H2 H3 H4 H5 H6 H7 H8 H9 29

30 Why is this Stable? Stability usually requires a sophisticated control law (e.g., TeXCP, MPTCP, etc) Source Leaf Dest Leaf 30

31 Why is this Stable? Stability usually requires a sophisticated control law (e.g., TeXCP, MPTCP, etc) Source Leaf (few microseconds) Feedback Latency Dest Leaf Adjustment Speed (flowlet arrivals) Near-zero latency + flowlets è stable 31

32 How Far is this from Optimal? Given traffic demands [λ ij ]: max!!"#$%!! min!!"#$%&'" Price of Anarchy with CONGA max!!"#$%!! bottleneck routing game (Banner & Orda, 2007) L0 L1 L2 H1 H2 H3 H4 H5 H6 H7 H8 H9 32

33 How Far is this from Optimal? Given traffic demands [λ ij ]: max!!"#$%!! min!!"#$%&'" Price of Anarchy with CONGA max!!"#$%!! bottleneck routing game (Banner & Orda, 2007) Worst-case L0 L1 L2 H1 H2 H3 H4 H5 H6 H7 H8 H9 Theorem: PoA of CONGA = 2 33

34 Implementation Implemented in silicon for Cisco s new flagship ACI datacenter fabric Ø Scales to over 25,000 non-blocking 10G ports (2-tier Leaf-Spine) Ø Die area: <2% of chip

35 Evaluation Link Failure fabric links 32x10G 32x10G 32x10G 32x10G Testbed experiments Ø 64 servers, 10/ switches Ø Realistic traffic patterns (enterprise, data-mining) Ø HDFS benchmark Large-scale simulations Ø OMNET++, Linux TCP Ø Varying fabric size, link speed, asymmetry Ø Up to 384-port fabric 35

36 HDFS Benchmark 1TB Write Test, 40 runs Cloudera hadoop cdh3u5, 1 NameNode, 63 DataNodes no link failure ~2x better than ECMP Link failure has almost no impact with CONGA 36

37 Decouple DC LB & Transport TX H1 H2 H3 H4 H5 H6 H7 H8 H9 Big Switch Abstraction (provided by network) ingress & egress (managed by transport) H9 H8 H7 H6 H5 H4 H3 H2 H1 RX 37

38 Conclusion CONGA: Globally congestion-aware LB for DC implemented in Cisco ACI datacenter fabric Key takeaways 1. In-network LB is right for DCs 2. Low latency is your friend; makes feedback control easy 38

39 Thank You! 39