Routing on the Host. Benefits of a modern data center design with Cumulus Networks. White Paper WHITE PAPER

Transcription

1 Routing on the Host Benefits of a modern data center design with Cumulus Networks White Paper WHITE PAPER

2 Routing on the Host : Table of Contents Table of Contents Introduction... 2 Network Complexity Reduction... 3 Subnet Freedom and Mobility... 6 Increased Redundancy - Hosts Connect to "N" Leaf Switches... 8 Stateless Anycast Conclusion

3 Routing on the Host : Introduction Introduction Today s modern data centers are evolving to a layer 3 networking design. Legacy data centers that are entirely layer 2 have numerous issues, including large broadcast and failure domains, spanning tree (STP) challenges, and the need for proprietary protocols such as MLAG for redundancy. A layer 3 Clos topology breaks up the broadcast and failure domains, eliminates spanning tree dependencies, and utilizes standardized routing protocols like OSPF or BGP. Even with those benefits, many data centers utilizing a layer 3 fabric still use layer 2 connectivity to the hosts. While the issues referenced above have been reduced by moving layer 3 closer to the edge, they are not eliminated. Hosts still need to be dual attached to switches for redundancy, thus spanning tree and MLAG are still deployed. Routing on the Host enables hosts to participate in the layer 3 routing fabric by running routing protocols directly on the host and completely eliminates these layer 2 challenges, while adding an array of additional benefits. Making use of robust, mature, and sophisticated open source routing suites that are available for Linux, routing can also be run on Linuxbased hosts, unifying an all layer 3 data center. The same open source tools that are used to configure and monitor both the Linux-based hosts and the Linux-based switches can be also be used to install, configure, and monitor the routing suite on the Linux-based hosts. Cumulus Networks has enhanced the Quagga routing suite so it can run on both hosts and Cumulus Linux switches. This paper highlights the benefits of hosts participating in the modern data center layer 3 fabric with Cumulus Linux on and switches and Cumulus Quagga on the hosts. By leveraging Cumulus Quagga, the data center becomes unified, realizing many advantages, four of which are examined here: Network complexity reduction Subnet freedom and IP mobility Support for increased redundancy Stateless anycast 2

4 Routing on the Host : Network Complexity Reduction Network Complexity Reduction Routing on the Host enables a layer 3 network design throughout the entire data center, unifying the data center and allowing easier troubleshooting. Both BGP and OSPF are mature standard routing protocols used in highly scaled environments and can be used for full data center connectivity. In the past, layer 2 was required in the data center for a variety of reasons pertaining to legacy applications. For example, some older applications use layer 2 broadcast/multicast for application discovery and to maintain heartbeat. Layer 2 connectivity was also needed for applications like VMware vmotion to move virtual machines (and their IP addresses) from one rack to another. However, there are several issues with an all layer 2 design: Spanning tree complexity When redundancy is required, the spanning tree protocol (STP) is needed to avoid bridging loops. Spanning tree avoids loops by causing some redundant links to be blocked, which decreases available bandwidth and creates significant convergence times. Proprietary multi-chassis link aggregation group (MLAG) In order to increase link utilization while still providing redundancy, many data centers deploy MLAG to "fool" spanning tree and link bonding into thinking there is only one remote chassis and as long as both remote chassis stay in sync with each other on the control plane, it can be effective to allow full utilization of all links as well as redundancy. MLAG, however, is typically proprietary as no standard exists. Only two peered, redundant switches are supported, which limits bandwidth when one of the MLAG switches is down for any reason, such as a hardware failure or maintenance. In addition, typically two extra ports are used for intra-mlag control plane communication and the backup data path. Lack of layer 3 segmentation Deploying a data center with a layer 2 network creates a single broadcast and failure domain. This means that one spanning tree failure could affect the entire data center. Troubleshooting difficulty Bridging does not have mechanisms for discarding a frame if it has been forwarded multiple times, so a spanning tree loop can cause a network meltdown. Spanning tree issues cannot be investigated while the problem is occurring, which makes spanning tree notoriously difficult to troubleshoot. Due to the above mentioned issues, many data centers deploy a layer 3 fabric between the and the while keeping layer 2 to the host. Layer 3 segments the layer 2 network broadcast and failure domain and eliminates the need for spanning tree between the and switches. Since layer 3 has a TTL (time to live) mechanism, a routing loop does not cause a network meltdown, making it easier to troubleshoot and fix a problem while it is occurring while protecting network assets from being overwhelmed. Tools like traceroute and ping are available for troubleshooting and BGP can keep track of AS numbers in the path to help follow the path a packet will take. 3

5 Routing on the Host : Network Complexity Reduction L3 Routing Extra swps needed Max 2 MLAG switches per host VLANs VLANs Spanning tree domain Spanning tree domain Hosts Figure 1. Layer 2 to the Host Hosts As seen in Figure 1, even with a layer 3 fabric between the and, people still deploy spanning tree and MLAG to the host when redundancy is required and can still experience the above mentioned issues. Deploying a routing protocol on the host and advertising the host's /32 address (or /128 ipv6 address) eliminates all of the above layer 2 issues all the way down to the host. As seen in Figure 2, routing to the host completely eliminates the need for STP and MLAG in the data center and provides better redundancy than MLAG along with a much simpler, easier solution. 4

6 Routing on the Host : Network Complexity Reduction Routing on the Host simplifies data center network design L3 Routing Host Host Figure 2. Routing on the Host Data Center Design For certain applications, layer 2 still has to be extended between hosts, and Routing on the Host can provide benefits here as well. With a layer 3 fabric, a network virtualization overlay such as VXLAN is typically used. Routing on the host allows the /32 (or /128 ipv6 address) Virtual Tunnel EndPoint (VTEP) addresses residing on the hosts to be advertised into the routing domain with full redundancy without the use of MLAG. OpenStack Neutron with built-in or third-party network virtualization for multi-tenancy are great use cases. Cumulus Networks, Dell, and Red Hat demonstrated this scalable solution at the 2016 OpenStack Summit. You can find more information on the OpenStack demonstration with Routing on the Host here. Modern applications have uplevel communications and do not rely on layer 2 connectivity between hosts. The architecture of clustered applications and containerization using frameworks like Kubernetes and Mesos support the layer 3 model natively. For example, Kubernetes prefers DNS instead of layer 2 broadcast/multicast for discovery, so a fully routed model is ideal for these environments. Cumulus Quagga simplifies Routing on the Host deployments by supporting features such as BGP and OSPF unnumbered to the host, making configuration simple and easy. The same automation tools used for other services on the host can also be used for configuring BGP or OSPF on the host. Cumulus Quagga also supports JSON output, easing machine readability. Eliminating layer 2 from the network by enabling Routing on the Host creates a much simpler design that is easier to troubleshoot, provides redundancy, removes the large layer 2 failure domain, and reduces the proliferation of layer 2 broadcast and multicast. Furthermore, bandwidth can be increased since the previous two switch ports that were used for intra-mlag control plane communication can be freed up to connect to additional hosts. 5

7 Routing on the Host : Subnet Freedom and Mobility Subnet Freedom and Mobility Routing on the host provides complete subnet freedom and mobility as the IP address of the host can be completely independent of its location in the data center. With the legacy layer 2 to the host architecture, all the hosts in a rack had to be tied to a specific VLAN that is accessible on that specific rack. If a host had to relocate and keep the same IP address, it needed to move within that same subnet/vlan by using an overlay. See Figure 3. L2 Overlay Blue /24 Yellow /24 Red /24 L3 Routing Blue /24 Yellow /24 Red /24 VLANs Blue, yellow, red VLANs Blue, yellow, red VLANs VLANs Hosts Hosts Figure 3. Layer 2 to the Host Design with Layer 2 Overlay However, deploying Routing on the Host completely eliminates these restrictions. By enabling OSPF or BGP, the host can advertise its IP address(es) directly into the routing domain and host placement is no longer an issue. Power and rack space can be distributed independently of the required IP address for the host. If the host is relocated, and its IP address is moved along with it, the route to that IP address will converge and reachability is maintained at its new physical location. Deploy any host, anywhere in the data center 6

8 Routing on the Host : Subnet Freedom and Mobility Deploying Cumulus Quagga BGP unnumbered or OSPF unnumbered makes automation, moving a physical host, and IP address management even easier. Both deployments do not require specific addresses to be configured on host or switch interfaces as with traditional BGP or OSPF deployments. The only IP address advertised into the routing domain is the host s loopback address. Not only does this simplify configuration, but it also drastically reduces the number of IP addresses needed to be deployed and managed. For example, a /24 subnet allocates 253 host IP addresses within it. Advertising a /32 or /128 allows more granular IP assignment and eliminates any wasted host IP address that results from subnetting. No VLAN configurations and no switched virtual interfaces (SVIs) are needed either. No configurations change when physically moving a host and connecting to a different set of switches. See Figure 4. L3 Routing / /32 Host Host Figure 4.t Advertising /32 from the Host Routing on the Host decouples the host from its rack, thus giving much needed network flexibility and freedom. Hosts can be deployed anywhere within the data center, re-cabled to any, and still be reachable with the same address, requiring no configuration changes. 7

9 Routing on the Host : Increased Redundancy - Hosts Connect to "N" Leaf Switches Increased Redundancy - Hosts Connect to "N" Leaf Switches Routing on the Host provides the mechanisms to support greater redundancy than MLAG since routing does not limit the host s number of adjacent switches. A layer 2 to the host design with MLAG has a limitation of only two peered switches. A switch failure reduces the bandwidth to a host by 50% after an initial traffic loss. See Figure 5. 50% BW lost Layer 2 MLAG Hosts Figure 5. MLAG Limits Redundancy Routing on the Host does not have this limitation, and the host can connect to any number of switches to provide even more redundancy with no STP or MLAG. During a failure, the routing protocol converges to allow continued connectivity over the remaining links to the host. 8

10 Routing on the Host : Increased Redundancy - Hosts Connect to "N" Leaf Switches During a maintenance window, routing protocols allow a graceful rerouting of traffic around a switch prior to taking it offline. For example, BGP can be configured for AS-path prepend both to the and the host, which gracefully forces traffic to avoid that specific switch and thus avoid traffic loss. See Figure 6. For maintenance, can gracefully move traffic off L3 Routing Only 33% BW lost Layer 3 ECMP Host Host Host Host Figure 6. Increased Redundancy and Resiliency 9

11 Routing on the Host : Stateless Anycast Stateless Anycast Routing on the Host enables load balancing, redundancy, and locality through the use of anycast addressing. Many load balancing designs are based on extending the layer 2 network between every rack that hosts the same application. These designs rely on MLAG between the s and s for redundancy. Alternatively, to maintain a layer 3 / network but still extend layer 2 connectivity, an overlay technology such as VXLAN is used. Both of these methods generally introduce an expensive load balancer, add complexity, and result in large layer 2 broadcast and failure domains. However, announcing services with anycast, leveraging Routing on the Host, provides similar load balancing functionality in a simplified fashion. By assigning the same IP address to several hosts and advertising each host's IP address into the routing domain, several routes to the same anycast destination are advertised and seen by the switches. Each switch then has an equal cost multipath (ECMP) route to the same IP address, and through the use of resilient hashing on the switches, connections from remote nodes are distributed between these hosts. The anycast solution also inherently provides redundancy to the hosts through route convergence; the hosts are configured with the same virtual IP address and run the same stateless applications and services (see Figure 7). For example, Network Time Protocol (NTP) and Domain Name Service (DNS) are popular applications that can use anycast for load balancing. You can find more information on anycast DNS at Anycast DNS, Part 1 - Overview. client A client B Spine Routing Table /32 via swp /32 via swp /32 via swp3 Leaf Routing Table /32 via swp /32 via swp4 L3 Routing / /32 Figure 7. Using Anycast for Load Balancing 10

12 Routing on the Host : Stateless Anycast Routing on the Host simplifies anycast for stateless load balancing between hosts Routing on the Host with anycast can also provide locality with redundancy. Configuring the same virtual IP address on hosts running the same application in different data centers keeps the traffic local while also providing access to the remote host in a failure scenario. For example, this solution eases configuration by requiring only one DNS server IP address on each host. Figure 8 depicts a local host requesting DNS from the local DNS service, while also having access to a remote DNS service during a local DNS server outage. Data Center Interconnect SITE A SITE B /32 via swp4 cost /32 via swp5 cost /32 via swp6 cost 10 swp 5-6 swp 4 DNS Request to / /32 Primary Path Higher Cost Path Figure 8. Using Anycast for Locality with Redundancy 11

13 Routing on the Host : Conclusion Conclusion Allowing the host to participate in the layer 3 routing fabric unifies and provides many benefits to a data center. It can reduce complexity by completely eliminating the need for VLANs, spanning tree, or MLAG. It provides full subnet freedom and mobility. Rather than tying a host IP address to a rack, complete IP address independence can be achieved by advertising the host's IP address directly into the routing domain. This way, if the host moves, the route moves with it. Routing on the Host also provides built-in anycast capability by advertising the same host IP address from two or more hosts. The Cumulus Networks implementation of Quagga is deployed in some of the largest data centers in the world. Since Cumulus Quagga is a Linux application, it also runs on any Linux host. Cumulus Quagga can be installed, configured, and monitored the same as any switch running Cumulus Linux, either manually or using the same automation tools. As a native Linux network OS, Cumulus Linux along with Cumulus Quagga on the hosts unify the entire data center, making it the ideal solution for your modern scalable data center. About Cumulus Networks Cumulus Networks demystifies the complexity of networking and enables better, faster, easier networks to support your business. Our network operating system, Cumulus Linux, allows you to build and operate your network with the mindset of web-scale pioneers like Google and Amazon, radically reducing the costs and complexities of modern data center networks. More than 400 organizations, including some of the largest-scale data center operations in the world, run Cumulus Linux. Cumulus Networks has received venture funding from Andreessen Horowitz, Battery Ventures, Sequoia Capital, Peter Wagner and four of the original VMware founders. For more information visit cumulusnetworks.com or 2016 Cumulus Networks. All rights reserved. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the Marks ) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their respective owners G 12