GoPlane: Open Source BUM-less Networking for Large Scale Docker Deployment

Transcription

1 GoPlane: Open Source BUM-less Networking for Large Scale Docker Deployment August 19, 2015 Soramichi Akiyama Software Innovation Center, NTT, Japan

2 A bit about Myself Innovation Center, NTT Had been working on Virtual Machine Live Migration for 5 years Ask me if you want to know migration technologies other than trace and replay Current work on software defined networking for DCs using baremetal switches More on me: 2

3 A bit about NTT Japanese counterpart of AT&T telephone, telegraph, ASDL, FTTH, ISP, DC, mobile Changing toward timely and open FLOSS we develop Ryu Openflow controller Sheepdog distributed storage (merged in QEMU upstrm) Lagopus software switch 3

4 Large Scale Docker Deployment i) Load balancing across multiple racks Internet iii) Disaster recovery across the Internet DC1 DC2 ii) User affinity across multiple DCs in the same provider 4

5 How Docker Apps are Deployed (Ideally) Deployable as developed This requires network tenant isolation No (visible) one else on the NW Use any IP you want No (visible) one else on the NW Use any IP you want Your Macbook (or Linux machine) VM Development Deployment 5

6 Overcoming L3 boundary Large scale Docker deployment hits L3 boundaries Across racks, DCs, Internet Large scale single L2 domain is in feasible Original Docker app in a single L2 domain

7 Overcoming L3 boundary Large scale Docker deployment hits L3 boundaries Across racks, DCs, Internet Large scale single L2 domain is in feasible Load-balanced version across an L3 NW DC1 DC L3 boundary?? 7

8 L2 over L3 using VxLAN VxLAN: Virtual extensible LAN Encapsulate L2 packets with VxLAN headers L2 networks can be extended over L3 networks L2 packet VxLAN switch VxLAN switch L3 boundary 8

9 L2 over L3 using VxLAN VxLAN: Virtual extensible LAN Encapsulate L2 packets with VxLAN headers L2 networks can be extended over L3 networks UDP packet vxlan header L2 packet VxLAN switch VxLAN switch L3 boundary 9

10 L2 over L3 using VxLAN VxLAN: Virtual extensible LAN Encapsulate L2 packets with VxLAN headers L2 networks can be extended over L3 networks VxLAN switch VxLAN switch L2 packet L3 boundary 10

11 Problem: How to find the next hop? VxLAN tunnels over L3 NW ?? ping Where to send BUM packets? Broadcast Unknown Unicast Multicast 11

12 Naïve method: what Socketplane does Dataplane flood and learn Just like normal L2 network Broadcasting over L3 networks! b-cast to ff:ff:ff:ff:ff:ff ARP

13 Existing approaches: Nicira service node BUM packets are sent to *the* service node Reduce CPU load for packet replication Figure cited from 13

14 Existing approaches: Nicira service node BUM packets are sent to *the* service node Reduce CPU load for packet replication Figure cited from ARP No help to the congestion problem! 14

15 Existing approaches: Cisco floodless mode Implemented in Cisco Nexus switches Mac addresses are reported to/distributed from ToR Cisco Nexus (3) Mac addr reported (4) Mac addr distributed to the other hosts (2) Mac addr detected Host w/ special module Host w/ special module VM (1) VM invoked VM VM VM 15

16 Existing approaches: Cisco floodless mode Implemented in Cisco Nexus switches Mac addresses are reported to/distributed from ToR Cisco Nexus (3) Mac addr reported (4) Mac addr distributed to the other hosts (2) Mac addr detected Host w/ special module Host w/ special module VM (1) VM invoked VM VM VM Vendor lock-in, Cannot go across an L3 network 16

17 Our solution: OVS + BGP (EVPN) EVPN (Ethernet VPN): Simplified overview Extended ethernet across L3 networks Standardized in RFC7432 on Feb 2015 Exchange MAC addresses thru BGP 11:22:33:44:55:66 11:22:33:44:55:66 60:57:18:XX:YY:ZZ Border Gateway Protocol E4:1D:2D:AA:BB:CC E4:1D:2D:AA:BB:CC 60:57:18:XX:YY:ZZ 17

18 Our solution: OVS + BGP (EVPN) System Components GoBGP: Fully open and API-capable BGP daemon GoPlane: Data plane management Open vswitch: VxLAN tunneling and flow control GoBGP GoPlane Linux What we built Existent things 18

19 How it works: Mac address exchange (1) Container invoked (2) MAC addr notified to GoBGP Linux GoBGP GoPlane (5) VxLAN tunnel established Packet forwarding or modifying rule (OpenFlow term) (4) Insert flow Linux GoBGP GoPlane (3) MAC addr sent with BGP L3 network (e.g. Inter-rack, Inter-DC, Inet-net) 19

20 Flows Inserted 1. Remote tunnel selection flow Mac address VxLAN tunnel beyond which the container with the mac exists 2. ARP responder flow Mutates an ARP request to a corresponding ARP response (described in the coming slides) 20

21 How it works: ARP & Response GoBGP GoPlane (1) ARP packet sent to FF:FF:FF:FF:FF (b-cast) ARP GoBGP GoPlane Linux Linux 21

22 How it works: ARP & Response GoBGP GoPlane (2) OVS receives the ARP, mutates it to the response (as OVS knows the MAC addresses!) GoBGP GoPlane ARP Linux Linux 22

23 How it works: ARP & Response GoBGP GoPlane (3) Container receives the response, ARP packets never be shot to actual NW ARP response GoBGP GoPlane Linux Linux 23

24 How it works: Unicast Every unicast is known If the target mac is unknown by the OVS, that address does not exist ping

25 How it works: Unicast Every unicast is known If the target mac is unknown by the OVS, that address does not exist ARP resolvable locally (previous slides) ping 11:a:b:c:d:

26 How it works: Unicast Every unicast is known If the target mac is unknown by the OVS, that address does not exist The only one VxLAN tunnel chosen by OVS

27 Discussion Pros of our solution Fully open, no proprietary technology Flexibility by API-capable BGP daemon Even the containers themselves can manage the network if you want (looking for a good use case) Cons of our solution BGP doesn t consider coherence Special extension needed for IP management 27

28 GoBGP Open Source BGP daemon we develop Controlled with GRPC (via http) Suitable for SDN Leverage multi-core CPUs The most famous existing software BGP is single threaded 28

29 EVPN Tokyo 2015 Juniper QFX GoBGP + GoPlane Cisco Nexus 9xxx 29

30 Summary L2 over L3 NW is required for large scale Docker deployment Existing solutions do not fulfill requirements Service node: no help for congestion Cisco floodless mode: vendor lock-in, ToR only We built a fully open solution, by a combination of BGP (EVPN) + OVS GoPlane and GoBGP available at GitHub! 30

31 Q&A 31