Optimizing OpenFlow Load-Balancing with L2 Direct Server Return

Transcription

1 Optimizing OpenFlow Load-Balancing with L2 Direct Server Return Marc Koerner Technische Universitaet Berlin Department of Telecommunication Systems Complex and Distributed IT Systems Einsteinufer 17, Berlin, Germany Odej Kao Technische Universitaet Berlin Department of Telecommunication Systems Complex and Distributed IT Systems Einsteinufer 17, Berlin, Germany Abstract Load-balancers have an essential role in enterprise data-centers, as their existence and performance is decisive for service provision and availability. The state of the art is that today s load-balancers are a piece of specialized embedded hardware with a proprietary operating system and application modules. In this paper we propose a concept to break up this black box model and to show how to develop a high performance LB with OpenFlow technology. In this approach we use OpenFlow network components to process the loadbalancing and in order to reduce the maintenance effort and to lower the hardware cost factor. Furthermore, in case of failover it is redundant by concept and easier to replace by another device. The experimental evaluation of this concept proves that the concept is developed beyond a theoretical concept and the implemented solution is ready for deployment. I. INTRODUCTION Load-balancers have a centralized management and control function in data-centers and ensure the availability of the services that they provide. They distribute incoming service requests to an array of redundant servers to split their workload. This guarantees a fixed minimum amount of available bandwidth and latency for serving all clients. The efficiency increase comes with the problem that load-balancers often represent a single point of failure in a common data-center networking concept. An other problem is that proprietary LB hardware is coupled to a specific administration knowledge regarding the configuration, which is highly vendor depended. To avoid these impacts on configuration and maintenance by keeping the advantages of the devices, the proposed concept avoids the usage of proprietary load-balancing hardware as well as of the proprietary administration interfaces, by using the open standard for software defined networking called OpenFlow. The basic idea of this approach is that OpenFlow standard delivers an interface for performing several forwarding, dropping and modification actions for packets on vendor comprehensive switching and routing hardware. The aim is to show the usage of this new technology and provide business application on this platform. Feasibility and performance values were collected in a local OpenFlow testbed at the Technical University of Berlin. This testbed is part of a European OpenFlow infrastrucutre which is distributed over several European countries. It was created in scope of the project OpenFlow in Europe: Linking Infrastructures and Applications [1]. The remaining of the paper is organized as follows. The following section II introduces all information from the idea and previous investigations up to the required tools for the deployment. Section III explains the architecture and the processing model. Section IV gives an overview about the implementation followed by the evaluation in section V. Finally, section VI summarizes the results and experiences. II. BACKGROUND AND RELATED WORK The OpenFlow [2] paradigm is an approach to create a vendor-independent and a standardized interface for software defined networking. Basically, the idea of OpenFlow standard is to decouple the data- and the control-path of the packet forwarding hardware into two different modules. The switch, which acts as a plain data-path element and the extracted controller, which is completely separated and externally hosted. The protocol provides a manipulation mechanism for the forwarding tables in switches or routers. These forwarding tables are physically realized in ternary content addressable memory (TCAM). OpenFlow provides an abstraction layer for TCAM called flow tables. They contain the flow mods with a packet pattern, an associated actions and a counter. These flow mods are specified by the OpenFlow controller into the flow tables in the data-path element. The concept works with a network of packet processing devices connected to the centralized controller. This means that one single controller is able to control a single switch up to a complete network. OpenFlow controllers can be developed by implementing the specification, which is currently available from version 1.0 [3] up to version 1.3 [4]. Also available are several existing controllers, like NOX [5] or Floodlight [6] with an adaptable application interface (API). This is a comfortable solution for developers because they can focus on the flow processing algorithm or model. NOX is an older controller and has a C++ and Python API. In contrast, Floodlight is a newer controller implementation and is based on the Java Enterprise Edition. FloodLight can be controlled through a Java API. Basically, developers have the opportunity to implement modules or applications working on top of these controllers.

2 Deploying OpenFlow in data-centers can lead to several advantages, from decreasing management complexity up to network and service improvements, e.g. flexibility and scalability [7]. One important data-center element with significant impact on many performance factors is given by load-balancers. Mostly they have a centralized role and also a strong influence on the serving performance of the data-center. Multiple operation modes and balancing algorithms are available. Related papers dealing with OpenFlow based load-balancing focus on HTTP traffic [8], [9] and also does not deliver performance measurements. As observed during former investigations with a destination network address translation (DNAT) method, based on an OpenFlow load-balancing controllers, working with embedded switching hardware [10], there are major performance restrictions. These restrictions belong to the processed operation and the network layer. Especially network address rewrite operations are currently redirected from the application specific integrated circuit (ASIC) to the embedded core of the switch and processed in software. This is the reason for most of the performance issues. This behaviour is currently shown by all hardware OpenFlow switches. Switches are typical layer two devices and most of them used ASIC s or OpenFlow vendor implementations on the switch itself are not able to process layer three manipulation operations with line-rate. A procedure of layer two load-balancing is called direct server return (L2DSR) [11]. The idea of this concept is that a load-balancer in the local server network forwards and distributes service requests and packets directly to an array of servers with the same virtual IP address, equal to the LB network address. The LB solely substitute the destination MAC address and forwards the packet. This mechanism is called MAC address translation (MAT) and works similar as NAT. The server answering the request returns the packets directly to the router in the broadcast domain where they seem to come from, indicated through the layer two source address which is not manipulated by the LB. III. ARCHITECTURE This section introduces the fundamentals how to use existing switch based OpenFlow hardware to deploy a full functional load-balancer which is able to deliver line-rate performance. The main idea of this networking concept is to use the active network components for the load-balancing and thus replace the static LB-Hardware with an OpenFlow controller which balances the load directly on the switching hardware. The focus of this particular approach is a performance concept for load-balancing considering opportunities of OpenFlow. The balancing algorithm is optional replaceable and adoptable to the specific requirements of the service and will not further discussed in this paper. As mentioned in section II a network-layer-three approach has not enough performance for a suitable load-balancing solution as needed by data-centers. So this model uses a layer two concept to improve the processing performance on OpenFlow switching devices. In this architecture, a similar procedure to L2DSR is used for an OpenFlow controller implementation for efficient load balancing. Fig. 1. hardware Load-Balancing concept with L2DSR on OpenFlow switching As depicted in figure 1 the servers are directly connected to an OpenFlow-switch in a routed data-center broadcast domain. The OpenFlow-controller labelled with LBC is responsible for the load-balancing which decouples the servers against the rest of the network in order to avoid address conflicts. As described in section II, L2DSR works with virtual server IP s. This is not needed in this particular case, because the servers are isolated through the OpenFlow-switch or more precisely through the traffic handling of the controller. The controller is not directly forwarding any L2 traffic, he blocks ARP broadcasts and response directly. This means the controller forces the switch to act as a device with an own non-transparent interface in contrast to normal switches. This is realized by handling every kind of address resolution traffic to the legacy network and advertising the associated MAC and IP. This IP is also assigned to every server in the server network array. Internal address resolution requests in the server array network part are directly answered by the switch with the corresponding information of the controller MAT table. All MAC broadcast are not flooded neither in the server network nor in the datacenter network to prevent layer-three address conflicts. If there is no information available, the switch forwards the packet by replacing the MAC address with its own address and answers the request, if it gets the reply form the requested machine. The switch forwards traffic with his associated IP and rewrites the destination MAC address with an address out of the server MAC pool. Which address is used for this process depends on the balancing algorithm. The server reply is also forwarded with a modified source MAC. The switch solely removes the server layer-two source address and replaces it with his own associated one. This ensures a deterministic addressing in the legacy network and an error free operation. The basic procedure is similar to a routing operation. The OpenFlow load-balancer gets packets and forwards them with a new destination data link layer address. The particular difference in that case is that the LB not only separates the broadcast domains, but it also forwards the traffic which is appointed to the server array also using a dedicated loadbalancing algorithm for the traffic distribution among them. The controller is aware of the mapping between the source IP and the destination MAC and uses this knowledge for a statefull packet transfer between client and server due to a fixed flow entry.

3 Src. MAC Dst. MAC Src. IP Dst. IP 1 C n R eth0 C n OF S eth0 2 R eth1 OF S eth0 C n OF S eth0 3 R eth1 S n C n OF S eth0 4 S n R eth1 OF S eth0 C n 5 OF S eth0 R eth1 OF S eth0 C n 6 R eth0 C n OF S eth0 C n TABLE I L2 AND L3 PACKET HEADER MODIFICATION In following we explain the details of packet-processing using the example network depicted in figure 1. Table I shows all header manipulations in the packet on the way through the network, from the arrival at the router, over the load-balancer up to the server and vice versa. An arriving packet (row 1) with the IP of the load-balancer in this particular case the IP, which is associated with the OpenFlow switch by his LBcontroller is forwarded into the local network. Therefore the router replaces the source and the destination address in the layer-two header of the packet (row 2). This corresponds to the standard packet routing operation. The network destination of the packet is the OpenFlow switch (OFS) with his emulated IP and MAC address. The switch forwards the packet to a server in the server network array and replaces the destination MAC address again (row 3). The choice of the MAC address and of the corresponding server depends on the MAC pool as well as on the selected load-balancing algorithm. The destination IP is not replaced, as both addresses are identical. This means that S n and OF S eth0 have the same IP. Also the source IP is not modified because the server directly replies to the requesting device (row 4). The OFS just forwards the packet and replaces the source MAC address. It is again changed to the OFS MAC (row 5). Finally, the router is forwarding the packet by a standard routing operation to the client in the Internet (row 6). IV. IMPLEMENTATION The load-balancer implementation is based on the NOX controller and deployed as a NOX plug-in, which can be loaded during NOX start by using the name of the plug-in as a parameter in the command line interface (CLI). It has no dependencies to other plug-ins and is a stand-alone implementation. The NOX controller uses an event driven model, which calls the corresponding functions of the implemented component container. The plug-in is written in C/C++ and uses the architecture proposed in section III. For sake of simplicity, the initial L2 load-balancing implementation uses a round robin algorithm for distributing the incoming requests to the servers. The packet processing can be separated into two different processing methods. For example, in case of ARP methods the controller directly forces the switch to send out a packet as reaction on a received one. This means, the switch interface acts as an independent device. It is a slow operation, because every interaction has to be approved by the controller. On the other hand, one can use direct forwarding actions with manipulation of the L2 address by OpenFlow actions executed on the switch by an installed FlowMod- Entry. This is a fast operation, because the controller approved the flow only once. Every further packet which matches the FlowMod-Entry is directly modified and forwarded by the switch without additional approval by the controller. The direct forwarding entries consist of a matching pattern and two OpenFlow actions, which are installed in form of a FlowMod in the switch. The first action is the manipulation assignment for the destination MAC address. The second action is the output action which causes the forwarding of the packet. Basically, the important functions of the implementation are responsible for answering ARP requests, pushing down L2 rewrites FlowMod s and handling the load-balancing. Additional functions are used for the management of the LB entries and ARP mappings. The method responsible for the LB matches every incoming new packet except link layer discovery protocol (LLDP) or ARP. If a new incoming flow is detected, the controller directs the flow rules with the rewrite information down to the switch. This procedure requires two FlowMod s per client server mapping which delivers the same scalability regarding FlowMod s as a switching plug-in. The algorithm maps depending on the source IP of the requesting client a server MAC address and stores this information in the internal LB mapping table. The entry is removed, if the corresponding flow expire event from the switch is received. V. EVALUATION For the experimental evaluation we used the local OFELIA OpenFlow testbed at TU Berlin. The testbed consists of three meshed gigabit OpenFlow switches, supporting OpenFlow version 1.0, and three servers. One server for the OFELIA control framework (CF) and two servers for hosting the user VM s. The used virtualization technology is the XEN hypervisor, which provides para-virtualized Debian VM s. These servers for the user VM s provide three experimental gigabit interfaces each, as well as a dedicated control interface. The CF mainly presents to the user a web interface for resource allocation. Resources are VM s as well as a network flowspace. Fig. 2. Physical OFELIA testbed installation at TUB

4 The hardware details of the testbed depicted by figure 2 are the following. The CF is hosted on IBM server with two Intel Xeon quad-core CPU s with 2.4 GHz using a Suse Linux Enterprise Server operating system. The User-VM servers are equipped with an Intel E quad-core CPU with 16GB RAM running a Debian Squeeze. The deployed OpenFlow switches are three NEC IP8800/ S TW and a legacy HP5400 segregation switch as well as an Ixia T1600 testing system with two four port 1GBit line cards and one eight port 100MBit card. Figure 2 also shows the connection between the devices. The black coloured connections are for the management network, the green connections are for the user control network and blue for the OpenFlow experimental network. Only the blue and green networks are accessible and transparent for experimenters or researchers working on the testbed. For testing the implementation on the hardware testbed is only needed a free of charge OFELIA account, which is available on the website. With this account OpenVPN [12] access to OFELIA facility can be used to allocate a VLAN based OpenFlow network slice with the web-based control framework. This slice was connected to the controller, which was also hosted on a VM in the test-bed. For getting comparable results the same experimental installation, as during former research [10], is used in the same configuration. Only the OpenFlow controller implementation was changed and the IP addresses of the server VM s were adapted. flow, the improvement of the old implementation is visible in a considerable performance gain. This increase is caused by using the implemented layer-two concept with packet header rewrite operations which are completely processable on the ASIC. This is also indicated through the CPU consumption of the embedded core. The usage of the switching device core was nearly idle in opposite to the previous layer-three rewrite approach and operation, which were completely processed in software and caused a high core load of approximated 95 percent. The values printed in figure 3 and 4 were collected with iperf and ping application on the Debian 6.0 VM s. They are collected between the client and server VM s to show the performance of the OpenFlow instance in the switch, which connects them. In both diagrams, the blue series is the old DNAT implementation and the red series presents the new layer-two implementation. Fig. 4. Bandwidth Fig. 3. Latency In this experimental evaluation former L3 DNAT measurement results and measurements from the new developed L2 DSR controller and their performance in the data-path are compared with each other. The focus of these measurements is set on the evaluation of processing performance in the datapath and the feasibility for a data-center usage. Except from the flow-mod installation time, where the switch communicates with the controller and asks how to proceed with the As depicted by figure 3 and also summarized in table II, the latency was reduced to a quarter. Also the scatter of the round trip time (RTT) was reduced to a maximum of 0.2ms with a standard deviation of 27ns. While bandwidth with TCP traffic was increased up to 939 MBit per second. This means an increased bandwidth performance by factor 145 and a latency decrease to an average of about 0.5ms. The only exception is the first packet where a new flow have to be approved by the controller. The RTT value for this first packet including the approval and installation of the flow-mod through the controller was measured with 7.62ms. L3DNAT L2DSR Latency: ms ms Bandwidth: 6.46 MBit/s 939 MBit/s TABLE II MEASUREMENT AVERAGE VALUES

5 VI. CONCLUSION This paper presents a novel approach for implementing load-balancing without deployment of dedicated hardware components and solely on the top of an OpenFlow enabled network. We achieve a significant performance gain compared with our previous approaches. The described load-balancing procedure improves the first concept and delivers nearly line-rate bandwidth throughput. With this OpenFlow loadbalancing controller model it is possible to process L2 LB on switching hardware with no limitations regarding performance issues. This result is a further step in the direction of OpenFlow applications for data-centers. When the time has come where software defined networks will replace the normal infrastructure of commercial data-center due to their whole advantages like central management, flexibility, modularity, reliability and so on, this load-balancing controller application is an approach for a real business scenario. The experimental evaluation and the measured real-world results prove that software-defined networking like OpenFlow can cover demands of productive data-centers. The architecture improves an evaluated implementation of a known concept realized with a new technology. REFERENCES [1] Openflow in europe: Linking infrastructure and applications, October 2011, [2] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner, Openflow: Enabling innovation in campus networks, ACM SIGCOMM Computer Communication Review, April [3] T. O. Consortium, Openflow switch specification / version 1.0.0, Decmber 2009, [4] O. N. Foundation, Openflow switch specification / version 1.3.0, May 2012, [5] N. Gude, T. Koponen, J. Pettit, B. Pfaff, M. Casado, N. McKeown, and S. Shenker, Nox: Towards an operating system for networks, ACM SIGCOMM Computer Communication Review, July [6] B. S. Networks, Floodlight, August 2012, openflowhub.org/. [7] A. Tavakoli, M. Casado, T. Koponen, and S. Shenker, Applying nox to the datacenter, Eighth ACM Workshop on Hot Topics in Networks, [8] N. Handigol, S. Seetharaman, M. Flajslik, N. McKeown, and R. Johari, Plug-n-serve: Load-balancing web traffic using openflow, ACM SIGCOMM Computer Communication Review, [9] R. Wang, D. Butnariu, and J. Rexford, Openflow-based server load balancing gonewild, In Hot-ICE, [10] M. Koerner and O. Kao, Multiple service load-balancing with openflow, in Proceedings of the IEEE 13th Conference on High Performance Switching and Routing, ser. IEEE. IEEE publishers, [11] T. Bourke, Server load balancing, O Reilly Media, August [12] Openvpn, September 2012,