VIABILITY OF DEPLOYING ebgp AS IGP IN DATACENTER NETWORKS. Chavan, Prathamesh Dhandapaani, Jagadeesh Kavuri, Mahesh Babu Mohankumar, Aravind

Transcription

1 VIABILITY OF DEPLOYING ebgp AS IGP IN DATACENTER NETWORKS Chavan, Prathamesh Dhandapaani, Jagadeesh Kavuri, Mahesh Babu Mohankumar, Aravind Faculty Advisors: Jose Santos & Mark Dehus A capstone paper submitted as partial fulfilment for the degree of Masters in Telecommunications at the University of Colorado at Boulder

2 Viability of implementing ebgp as IGP in datacenters 2 ABSTRACT: Advent of cloud based infrastructure and continuous growth of web based services has motivated service providers to set up large scale Datacenters (DC). Traditionally, DCs utilize Border Gateway Protocol (BGP) to route the customer traffic to and from the Internet. Link state protocols such as Open Shortest Path First (OSPF) and Intermediate System-Intermediate System (IS-IS) are used for routing the traffic within DC. The disadvantages of OSPF and IS-IS are: they require large number of CPU cycles to process the link state updates and maintain an identical link state database within the area; these protocols also impose a challenge due to the necessity of redesigning DC routing architecture when the number of network devices increase. Few large scale DC operators have taken a novel approach in the routing design by adopting exterior Border Gateway Protocol (ebgp) as an Interior Gateway Protocol (IGP). The significant advantages of using ebgp as an IGP are: i) It is a stable and robust routing protocol consuming less number of CPU cycles ii) iii) It can handle large number of routes without having to re-design the network It is simple to operate and troubleshoot Our research will focus on the feasibility, architecture and the impacts of implementing ebgp as an IGP in large scale DC environments. We will simulate this environment with ebgp as the only routing protocol and compare the network performance against traditional IGP s. We will also analyze the impact of the fundamental datacenter services such as seamless mobility for virtual machines, extension of Layer 2 adjacency (VLANs) within the datacenter and load balancing applications critical for the operation of high density server environments. The goal of our research is to evaluate the incentives and the viability for large scale datacenters to migrate to an e-bgp only design.

3 Viability of implementing ebgp as IGP in datacenters 3 Table of Contents I. Introduction... 4 Statement of the Problem... 5 Research Problem... 6 Research Sub-problems... 6 II. Literature Review... 7 Traditional Datacenter topology... 7 Advantages of using CLOS design... 8 Choosing the appropriate routing protocol to be used in the Datacenter... 9 Impact on legacy datacenter applications Current state of the art technology III. Research Methodology Layer 2 Performance Analysis Layer 3 performance analysis OSPF vs BGP IV. Research Results Layer 2 Performance evaluation Layer 3 performance results under normal operations Layer 3 performance analysis during convergence V. Discussion of Research results: VI. Conclusion and Future work: VII. Appendix VIII. References... 23

4 Viability of implementing ebgp as IGP in datacenters 4 I. Introduction The growing demand for virtualization and cloud based services along with traditional network practices has created challenges in designing large scale datacenters (DC), which typically includes more than 20,000 Virtual Machines and nearly 2000 networking devices [1]. Following are some of the significant requirements of a large scale DC [2]: i) Selecting an appropriate topology that supports the increasing demand for server to server communication (east-west traffic) within the Datacenter, by adding more number of links and cheap commodity devices with identical port density. ii) Choosing a single control plane protocol (if possible) that could handle large number of routes, eliminates the need for inter-operability with other control plane protocols and are supported by most of the network equipment vendors. iii) Choosing a routing protocol that has a simple implementation in terms of the protocol design and ease of operational support. iv) Reduce the scope of failure caused by equipment or protocol issues. Designing the control plane architecture is the most challenging part. Control plane is a critical component of the router or a switch architecture that has sufficient intelligence to build the network map (dynamically or statically) and determines how an individual network device interacts with its neighbors. It is the signaling component of the network. Few of the control plane functions include management/configuration of the network device and exchange of routing information. Control plane protocols are essential for efficient forwarding of the data traffic between any two devices. Multiple Spanning Tree (MST) may be considered as the control plane protocol on a pure Layer 2 environment, whereas OSPF and BGP may be considered as the control plane protocol used in a Layer 3 environment.

5 Viability of implementing ebgp as IGP in datacenters 5 CLOS topology is preferred in large scale DCs as this topology can be horizontally scaled by increasing the port density in the devices or by adding similar devices to the topology. This saves the cost involved in buying high performance devices and avoiding the re-design of the network [3]. In a large scale DC, the cost of the networking infrastructure amounts to 10-15% of the entire Capital Expenditure (CAPEX) [4]. There is a constant drive to cut down costs and therefore DCs prefer networking elements of the same hardware type with minimal features [2]. Traditionally datacenters utilize Border Gateway Protocol (ebgp), a distance vector protocol, on Wide Area Networks(WAN) to route the traffic to and from the Internet and link state protocols such as OSPF and IS-IS in the Local Area networks (LAN) [5] to efficiently route the traffic within the datacenter. Having two control plane protocols within a DC increases the probability of network misconfiguration and poses challenges in making both the protocols interact with each other, thereby demanding a design with simpler control plane protocol that reduces the chance of network failure [2]. A single control plane protocol makes the network design simple and easy to troubleshoot; which is one of the fundamental requirements of a large scale data center. Statement of the Problem BGP is mostly viewed as the exterior routing protocol used for routing traffic between networks managed by different administrative bodies. The significant reasons for using BGP are; it provides a better control over the traffic flow because of its built-in attribute set, it can handle a large number of routes because of its limited flooding scope and simpler protocol design [6]. Link state Protocols such as OSPF and IS-IS are designed for routing traffic within the same administrative domain; as they have a complete view of the network topology within the area and can converge faster [5]. However, as the number of network devices and routes continues to increase, it creates the following scalability challenge in the network design:

6 Viability of implementing ebgp as IGP in datacenters 6 a. The scope of Link State Advertisements (LSA) flooding is the entire area which causes all the network devices within the area to process all the LSAs generated and compute SPF algorithms. The above processes may consume large number of CPU cycles on a network device as the instability in the network and number of network devices increase. The most widely implemented solution to address the above challenges is to re-design the network by segmenting a single area into multiple areas, so as to reduce the processing load on the network device [2]. Re-designing DC network requires careful planning, scheduled network outages, possibility of misconfiguration and administrative overhead. Currently, the IETF group is working on standardizing an alternate and innovative solution to the above challenges by deploying BGP as an IGP, instead of traditional OSPF and IS-IS [2]. The above routing architecture is the basis of our research in analyzing the practicality and viability of deploying ebgp as an IGP in large scale Datacenters. Research Problem What are the incentives for firms operating large scale datacenters to migrate to an ebgp only design? Although the design permits, what is the viability of implementing ebgp as an IGP? Does the new design of deploying BGP as the only control plane protocol affect any existing services and applications offered by datacenters? Research Sub-problems i. Simulating an optimally designed large scale DC network to analyze the viability of deploying ebgp as an IGP ii. Network Performance evaluation of Layer 2 and Layer 3 control protocols a. Layer 2 performance analysis

7 Viability of implementing ebgp as IGP in datacenters 7 b. Layer 3 performance analysis (OSPF vs. BGP) iii. Analyzing the possible impacts of deploying ebgp as an IGP II. Literature Review Traditional Datacenter topology Traditional DC environment utilizes a mix of Ethernet based Layer 2 technologies with three levels of hierarchy namely access, aggregation and core layers [7]. A typical DC network consist of a hierarchical tree like topology with progressively sophisticated high performance network devices at the core layer of the network and commodity network devices at the access layer of the network. These traditional datacenters utilize Spanning Tree Protocols between the access and distribution layer and link state routing protocols between the distribution and core layer. BGP is used for providing external connectivity for the DC. The use of a Layer 2 solution at the access layer causes redundant uplinks on the access layer switches to be blocked by the Spanning Tree Protocol (STP) to avoid Layer 2 loops; this effectively reduces the net utilization of the up-links [3] because of an active-passive approach. The advent of Multi-Chassis Link Aggregation protocol (M-LAGs) provides an active-active approach and offers increased throughput. However, most of the M-LAG implementations are proprietary and have a little multi-vendor support [2]. The need of STP in a flat Layer 2 network can be avoided with the introduction of TRILL that addresses most of the challenges faced by STP. However, TRILL requires new network devices and has limited implementation experiences. Finally, both TRILL and M-LAG does not eliminate the primary concerns of a shared broadcast domain such as: large amounts of broadcast and unknown-unicast storms [2].

8 Viability of implementing ebgp as IGP in datacenters 8 Advantages of using CLOS design Until recently, the traffic flows were commonly North-South traffic (enter and exit the datacenter) and tree-topology served the requirements of these type of traffic with high oversubscription ratios and high density line cards on the core network devices. But, recently many large scale DC host applications that require distributed computing across the servers, causing large amount of East- West traffic flow. The traditional tree topologies do not scale horizontally to match these bandwidth demands. The fat-tree topology (also known as CLOS or leaf-spine topology), on the other hand, uses similar type of commodity Ethernet switches to provide cheaper scalable interconnection bandwidth between the servers. The fat tree topology can be easily scaled by adding more number of stages or increasing the port density of the network element as shown in the figure below: Figure 1.a: Two stage CLOS topology to support 4 server subnets Figure 1.b: Scaling CLOS topology

9 Viability of implementing ebgp as IGP in datacenters 9 The above discussed CLOS topology presents a new interconnect architecture that leverages commodity network elements and meets the scalability, bandwidth and cost requirements of a large scale datacenter. Figure 1.a shows a two stage fully non-blocking CLOS topology to support four servers. The CLOS topology can be scaled to support eight servers by either increasing the port density on the two stage CLOS topology or by moving to a three stage CLOS topology as shown in figure 1.b.By appropriate choice of routing protocol, that inherently supports load balancing, the hosts in the network can communicate at the full bandwidth of its local interface. Choosing the appropriate routing protocol to be used in the Datacenter IP routing is generally used at the core layer and above. However, Next-generation datacenters incorporate IP routing intelligence to the access distribution layers to have efficient utilization of the uplinks, improved network stability and scalability [7]. It also confines Layer 2 broadcast domains and thereby significantly reducing the OPEX. Usually the layer 3 routing protocol is either OSPF or IS-IS, as it is assumed to provide faster convergence and better loop avoidance features in comparison to their distance vector counterparts [5]. However, as the DC grows in size and scale, the existing IGPs had to be constantly revisited and re-designed to reduce the amount of processing required on the routers within an area. Lapukhov from Microsoft and Premji from Arista Networks have recently identified several problems with the existing IGPs [8]. They are actively working with the IETF group in proposing a novel design to utilize BGP as the only IGP in large-scale datacenter environments because of the following advantages [2]: i. A network device running BGP advertises only the best path available to its directly connected neighbors and is less prone to flooding events, which makes it more scalable without the need to re-design [9].

10 Viability of implementing ebgp as IGP in datacenters 10 ii. BGP is the most widely implemented WAN protocol in a datacenter environment and extending its implementation in the LAN would simplify the operation overhead. iii. BGP has several attributes built-in to control the flow of routing information. It also supports third party resolved next hops that allow unequal cost load balancing and application defined forwarding paths [10]. iv. BGP is simple to troubleshoot when compared to OSPF because of the availability of adjacent Routing Information Base In (RIB-In) and Routing Information Base out (RIB-out) structures with the corresponding Network Layer reachability Information that can be easily correlated [10]. Impact on legacy datacenter applications Incorporating Layer 3 intelligence close to the access layer will break the layer 2 adjacency requirements of large scale DCs. There are several critical Datacenter applications such as seamless mobility for virtual machines, application load balancing using Layer 2 DSR architecture that require layer 2 adjacency for their operation.. However with the advent of several Layer 2 tunneling protocols such as GRE tunnel, L2TPv3, OTV and VXLAN, these services can still be achieved by extending the Layer 2 domain across the Layer 3 backbone. The pros and cons of these tunneling protocols will be discussed in the appendix section. Current state of the art technology The current research by IETF does not compare the performance of OSPF and BGP in a large scale DC environment and evaluate the incentives for firms to migrate to an e-bgp only design. Our research will attempt to extend the current IETF s work and analyze the viability of deploying BGP as an IGP in datacenter networks and the incentives to migrate to the ebgp-only design by

11 Viability of implementing ebgp as IGP in datacenters 11 i) Comparing the CPU utilization of OSPF and BGP during normal operations and reconvergence ii) Comparing the convergence time of OSPF and BGP iii) Exploring the most scalable Layer 2 tunneling solutions to address the Layer 2 adjacency requirement of critical datacenter applications. III. Research Methodology Our research evaluates the feasibility of deploying ebgp as an IGP in large scale Datacenter networks. The first sub-problem of our research was to simulate this environment in a lab; while real world datacenter networks consist of large number of network devices, we replicated a small subset of DC network to conduct our measurements. Fig 2.a Data plane Fig 2.b Management plane We implemented the above specialized and scalable network topology (Fig2.a) that satisfies the first requirement of the large scale datacenters as mentioned in Section I [3]. This topology is broken down into three layers: core (Tier 1), distribution/aggregation (Tier 2) and access (Tier 3) layers. The above network topology was built using 12 Cisco 3550 multi-layer switches, and 1000

12 Viability of implementing ebgp as IGP in datacenters 12 loopbacks were added on each Tier-3 device to simulate large number of servers. Next, to prevent the production traffic being affected by the management traffic, we implemented a separate data plane and management plane as shown in Figure 2.a and 2.b. The management plane was used to monitor and collect the network performance statistics through automated scripts. Layer 2 Performance Analysis We conducted a layer 2 performance analysis by considering three factors: CPU utilization, delay and bandwidth. A layer 2 network was simulated by configuring Multiple Spanning Tree (MST) in the test bed. We simulated instability in the network to test the MST convergence and measure the CPU utilization while the network converged. Next, we simulated traffic flows from one cluster to another to measure the delay and bandwidth in the network under various levels of broadcast. Layer 3 performance analysis OSPF vs BGP The following section describes the design and performance analysis of the layer 3 network with OSPF and BGP as the only control plane protocol. OSPF Design and optimization: Figure 3: IGP design using OSPF

13 Viability of implementing ebgp as IGP in datacenters 13 The network consisted of 12 devices in a single OSPF area (Area 0). We optimized OSPF to achieve faster convergence and utilize less CPU cycles by incorporating the following measures: a. Point-to-Point networks To avoid Designated Router (DR) elections b. Reduced Hello timers- To achieve faster re-convergence c. Incremental SPF (ISPF)- To reduce the CPU utilization for SPF computations d. Throttle timers- To achieve Faster convergence by reducing the time between SPF computations and LSA flooding All the above features are supported by most of the commodity switches, and hence satisfies requirement II of the large scale datacenters as mentioned in Section 1. Performance analysis of OSPF: We tested the CPU utilization under two states: normal operations (stable) and during reconvergence (unstable). For both the states, we simulated two scenarios: one with 8 device CLOS topology and other with 12 device CLOS topology to check whether the performance degrades with more number of devices. The following table illustrates the methodology used to conduct performance analysis for both states: State Reason for CPU utilization Methodology Link state database (LSDB) refreshed every 30 minutes (LS Refresh time). minutes Normal operations Re-convergence Flapping the server subnets (Loopbacks in Tier 3 devices) Flapping the transit links CPU utilization from all network devices was recorded every 30 CPU utilization from all network devices was recorded till the network converged (for approx. 2 minutes).

14 Viability of implementing ebgp as IGP in datacenters 14 IGP Network design using ebgp: The following diagram illustrates the BGP design scheme used in our network topology: Figure 4: IGP design using ebgp Internal Routing: This section describes the ebgp design scheme used for internal routing. External BGP (ebgp) peering sessions were established over point-to-point links between all network devices. All the devices in Tier-1 were assigned a unique Autonomous System Number (ASN), each group of Tier 2 devices in a cluster was assigned a unique ASN, and each Tier 3 device within a cluster was assigned a unique ASN as shown in figure 4. The 16bit ASNs allows the use of only 1023 unique private ASNs within a datacenter. Since, large scale datacenters may have more than 2000 network devices, the ASNs will have to be re-used at the Tier 3 level. Considering that ASN s are used in BGP as a loop avoidance feature [11]; the reuse of ASN s at the Tier 3 level will prevent routes advertised from a tier 3 device in one cluster being accepted by a tier 3 device in another cluster. In order to overcome this challenge, we utilized Allow AS In feature of BGP which allows a BGP enabled device to accept routes that contain its own ASN in the AS-PATH attribute. This feature is mostly supported on all commodity network devices [12]. It is assumed that the

15 Viability of implementing ebgp as IGP in datacenters 15 Datacenter network topology is symmetrical and does not cause routing loops when implementing this feature. External routing: The above BGP design routes the traffic within the datacenter, however doesn t provide Internet access. To do so, we have used another border cluster as shown in Figure 4. The Edge devices of the border cluster peer with the ISP and the core routers of the Border cluster peer with the Tier-1 devices of the datacenter. As private ASNs were used inside the datacenter; we use the Remove private-as feature in the border cluster to strip off the private AS numbers when advertising the Datacenter routes to the Internet. BGP optimization: As BGP is a stable routing protocol consuming less CPU cycles, we modified BGP to achieve faster convergence by incorporating the following measures: a. Reduced the default BGP advertisement interval value to zero so that a BGP peer exchanges the routing updates immediately. Even though the signaling overhead increases, it does not cause a considerable effect on the performance b. BGP, by default does not install multiple paths in the routing table. In order to utilize redundant paths, we enabled maximum paths feature available in BGP. BGP Performance evaluation: Performance evaluation for BGP was carried out in a similar manner as OSPF performance evaluation in order to achieve comparable results. The following table illustrates the methodology used to conduct performance analysis for both states:

16 Viability of implementing ebgp as IGP in datacenters 16 State Reason for CPU utilization Methodology BGP scanner process that runs CPU utilization from all network Normal operations every 60seconds by default devices was recorded after every Re-convergence Flapping the server subnets (Loopbacks on Tier 3 devices) Flapping the transit links minute. CPU utilization from all network devices was recorded till the network converged (for approx. 2 minutes). Convergence testing for OSPF and BGP: In order to test the convergence time of BGP and OSPF, continuous traffic was sent from a server connected to a Tier-3 device ( S9) in cluster 1 to a server connected to another Tier -3 device (S12) in cluster 2. The worst case scenario (in terms of processing overhead and convergence) for both the protocols happens when (Refer figure 5): a) We disable a redundant link L1, and force all the traffic through L2 b) We again enable link L1 and disable link L2 immediately. The above scenario was simulated, because the convergence depends on the neighbor establishment, advertisement of routes or link state packets and calculation of the best path. Figure 5: Network performance evaluation

17 Viability of implementing ebgp as IGP in datacenters 17 IV. Research Results In this section we present the performance evaluation of MST, OSPF and BGP, and briefly describe our results with graphs wherever necessary. Layer 2 Performance evaluation We observed that the CPU utilization of all the network devices was under 2% (for both stable and unstable cases) with MST and we were able to achieve convergence under one second in worst case scenarios The following figure illustrates the relationship between the amount of broadcasts and delay in a flat Layer 2 network. Performance of a Layer 2 network Traffic Delay (Seconds) Percentage of Broadcasts Figure 6: Layer 2 Performance evaluation

18 Viability of implementing ebgp as IGP in datacenters 18 Layer 3 performance results under normal operations OSPF CPU utilization in stable conditions Percentage CPU Utilization Number of Routes = Number of Nodes Tier - 1 Tier - 2 Figure 7a: CPU Utilization of OSPF under normal operations BGP CPU utilization in stable conditions Percentage CPU Utilization Number of Routes = Number of nodes Tier - 1 Devices Tier - 2 Devices Figure 7b: CPU Utilization of BGP under normal operations Figure 7.a and 7.b show the CPU utilization of OSPF and BGP under stable conditions. From the above graphs, it can be observed that the CPU utilization of OSPF increases as the number of network devices increase. However, it can be observed from the graph that the CPU utilization of BGP stays constant with the addition of more number of network devices.

19 Viability of implementing ebgp as IGP in datacenters 19 Layer 3 performance analysis during convergence CPU utilization of OSPF and BGP during convergence Percentage of CPU utilization OSPF - Tier 1 OSPF - Tier 2 OSPF - Tier 3 BGP - Tier 1 BGP - Tier 2 BGP - Tier 3 Average utilization over 60 sec Average utilization over 5 sec Figure 8: Performance metrics of OSPF and BGP during convergence. From the figure 8, it can be observed that the CPU utilization of OSPF increases to a maximum value of 56% (approx.) during convergence. Similarly it can be also observed that the CPU utilization of BGP increases to a maximum value of 37% (approx.) during convergence. V. Discussion of Research results: The CPU utilization and convergence of a Layer 2 network with MST was comparable with the Layer 3 network performance. However, broadcasts are inherently present throughout the Layer 2 network and this effectively reduces the network capacity and increases the delay in the network. It also consumes the CPU cycles of all the hosts connected in the DC, as all the hosts present in the same broadcast domain will have to process the broadcast traffic. Therefore, as the number of hosts in the Datacenter network increase, the flat Layer 2 design does not scale well. From the Layer 3 performance analysis tests, we observed that the CPU utilization of BGP was better than the CPU utilization of OSPF under stable conditions and also during network

20 Viability of implementing ebgp as IGP in datacenters 20 convergence. Inherent properties of link state protocols such as SPF computation complexity and periodic link state updates cause the CPU utilization to increase as the number of network devices increase. BGP being a distance vector protocol is not impacted by the above factors. Normally, the convergence of link state protocols is better than distance vector protocols [5]. However, based on our performance test results, we were able to achieve equivalent convergence time for BGP by optimizing it as discussed in section IV. VI. Conclusion and Future work: We first determined that a flat Layer 2 network performance in terms of CPU and convergence was comparable with the Layer 3 network performance. However, we also identified that as the size of the network increases, the Layer 2 design will have high amount of broadcasts which would cause delays and affect the overall throughput of the network. As we moved towards a Layer 3 approach, we found that the performance of OSPF degrades as the number of network devices increases; this performance degradation can be addressed by redesigning OSPF in to multiple areas. However, the redesigning approach requires planning and scheduled maintenance often, as the network scales. The use of ebgp as the IGP overcomes the above design challenges and does not require datacenter operators to re-design the complete routing architecture. Further, our results also showed that the CPU utilization of BGP is better than OSPF, and BGP can be optimized to achieve comparable convergence time as OSPF. We also found that a fully routed design with BGP/OSPF breaks the layer 2 adjacency, which is required for high availability services in a datacenter. However, we found out that several overlay technologies like GRE tunneling, L2TPv3, OTV and VXLAN can be used to achieve layer 2 adjacency over a fully routed design.

21 Viability of implementing ebgp as IGP in datacenters 21 In summary, our research shows that using ebgp as an IGP provides several operational advantages such as: single control plane protocol within the network which is easy to operate and troubleshoot, avoids network misconfigurations due to inter-operability of different control plane protocols, and capability to handle large number of routes without having to re-design the routing architecture. The above advantages provide large scale datacenter operators an incentive to migrate to an ebgp only design. Further, our approach of using a single control plane protocol within the network simplifies the migration steps towards Software Defined Networking.

22 Viability of implementing ebgp as IGP in datacenters 22 VII. Appendix - I Overlay Technologies The ebgp as an IGP design would have a complete Layer 3 design in the datacenters. There are critical applications such as Virtual machine migration, Layer 2 direct server return, etc. which require Layer 2 adjacency. This requirement can be satisfied by use of overlay technologies such as Layer 2 Generic routing encapsulation [GRE], Layer 2 transport protocol [L2TP], Overlay transport virtualization [OTV] and Virtual extensible local area network [VXLAN]. The above mentioned overlay technologies were analyzed to ensure ease of deployment and scalability. The advantage of GRE and L2TP is that they are easy to deploy and have less configuration overhead, but each of them create point to point tunnels and therefore become complex to deploy as the network scales. The recently developed overlay technologies such as OTV and VXLAN are scalable as they create dynamic tunnels to traverse through the routed access layer. The performance and configuration overhead between OTV and VXLAN is comparable. OTV primarily addresses overlaying layer 2 over layer 3 whereas VXLAN also addresses the shortage of vlan ids (only 4094) which is a realistic issue with increasing scale of datacenters. Also, OTV is a Cisco proprietary protocol which would necessitate all the devices in the Tier 3 to be cisco devices whereas VXLAN is currently in IETF draft stage with wide vendor support. Appendix II Further information relating to the implementation and configurations used for various tests conducted can be found at: s/edit?usp=sharing