WEBridge: west east bridge for distributed heterogeneous SDN NOSes peering

Transcription

1 SECURITY AND COMMUNICATION NETWORKS Security Comm. Networks 05; 8:96 94 Published online 4 August 04 in Wiley Online Library (wileyonlinelibrary.com)..030 SPECIAL ISSUE PAPER WEBridge: west east bridge for distributed heterogeneous SDN NOSes peering Pingping Lin, Jun Bi* and Yangyang Wang Institute for Network Sciences and Cyberspace, Department of Computer Science, Tsinghua University, Tsinghua National Laboratory for Information Science and Technology (TNList), FIT Building, Beijing, 00084, China ABSTRACT Large networks are often partitioned by the network operators into several smaller networks when deploying software-defined networks (SDNs). Additionally, a dedicated network operating system (NOS) is deployed for each of these SDNs. Each NOS can learn the local network view that enables control of how data packets are forwarded within its network. Controlling the flow of data packets in an entire network requires each NOS to have a global network view to determine the next NOS hop. Hence, NOSes are required to share or exchange reachability and topological information. How such information is efficiently exchanged has not been well addressed so far, especially in the case of multi-vendor NOSes. This paper proposes a west east bridge mechanism for distributed heterogeneous NOSes to cooperate in enterprise/data center/intra-autonomous system networks. We propose to simplify physical networks into virtual networks and only exchange the simplified virtual network information to construct the global network view. To achieve a resilient peer-to-peer control plane of distributed heterogeneous NOSes, we propose a maximum connection degree -based connection algorithm. Considering the security issue, we adopt controller identity authentication. We implement the west east bridge and analyze the performance obtained: about 00% of enterprises and data centers, and about 99.5% of autonomous systems can adopt to this solution. The deployment in three SDNs (CERNET, Internet, and CSTNET) proves the feasibility. Copyright 04 John Wiley & Sons, Ltd. KEYWORDS distributed heterogeneous SDN peers; network view virtualization; resilience control plane; peer to peer *Correspondence Jun Bi, Institute for Network Sciences and Cyberspace, Department of Computer Science, Tsinghua University, Tsinghua National Laboratory for Information Science and Technology (TNList), FIT Building, Beijing, 00084, China. [email protected]. INTRODUCTION Currently, the architecture of the network devices are closed systems. Closed systems do not favor network innovation especially considering protocol development or network service evolution. Software-defined network (SDN) [] decouples the vertical and tightly coupled network architecture and reconstructs the network as a modular one. At the same time, it standardizes forwarding technology and opens up the control plane and the associated protocol. Thus, the architecture in network device is no longer closed. In this way, software-defined network promotes the rapid innovation and the evolution of the network. The idea of SDN is well received by both of the academic researchers and industry researchers, network operators, and the larger networking industry. Softwaredefined networking has also become a popular topic in recent years and is considered as a promising way to re-architect the networks. In addition, some research organizations have been formed such as Open Networking Laboratory []. The Open Networking Foundation [3] is responsible for standardizing SDN technologies and has the support of more than 50 companies who jointly accelerate the creation of standards, products, and applications (APPs), such as NEC, Google, Facebook, Microsoft, IBM, VMware, Juniper, and Cisco. The SDN separates the control plane from the network data plane and moves it to a centralized software network operating system (NOS) or controller. The NOS takes charge of all the functions in control plane. Switches in the network are SDN switches including but not limited to OpenFlow [9] switches. The SDN switch retains only the basic data forwarding function. Furthermore, the NOS can run on a normal host or server to control the data packet forwarding in OpenFlow switches through a standardized OpenFlow protocol and an optional secure sockets layer (SSL) channel. When deploying SDN in real networks, large networks are often partitioned by the network operators into several 96 Copyright 04 John Wiley & Sons, Ltd.

2 P. Lin, J. Bi and Y. Wang smaller networks (referred to as sub-networks hereafter) because of numerous reasons [9]: Scalability. The number of devices an SDN controller can manage feasibly is limited. Large networks thus may have to deploy multiple controllers. Privacy. A carrier may choose to implement different privacy policies in different SDN sub-networks because, for instance, an SDN sub-network may be dedicated to a set of customers who implement their own highly customized privacy policies, which may require that some networking information in these sub-networks (e.g., network topology) should not be disclosed to certain external entities. Incremental deployment. Carrier s network may consist of portions of legacy and SDN infrastructure. Dividing the network into multiple, individually manageable SDN sub-networks allows flexible incremental deployment and a more seamless network evolution. Network faults isolation. If a fault appears, the network operator hopes that it only affects a small area of the network and tries to minimize the spread of more network problems. Each sub-network runs one NOS or controller such as NOX [0], Maestro [], Beacon [4], Floodlight [8], Trema [8], and so on. Each NOS just has a local network view. The network view refers to topology, entities, reachability, network abilities, and network state. However, many network APPs need a global view of the entire network rather than just one sub-network. This requirement can be illustrated using two examples. As a first example, inter-domain path (which is across multiple sub-networks) computation APPs running on a specific NOS need to have a global view of multiple intra-domain networks to calculate an end-to-end path. As a second example, data centers (DCs) of an enterprise are usually located in different areas, and each DC may run only one NOS. Virtual machine migration, network virtualization, and traffic engineering across DCs also require a global view of the entire network... Statement of the problem and aim of the paper On the basis of on such global network view requirements analyzed earlier, controllers should be able to construct a global network view and provide it to APPs. In order to do so, multiple NOSes may need to communicate with each other to exchange individual network view information or share a global network view database. There are several distributed NOSes such as ONOS [6], Onix [7], HyperFlow [8], and DIFANE [7], which seek to provide this capability. However, these NOSes are currently not interoperable because their west east communication interfaces are private. The purpose of this paper is to support the same SDN administrative domain (enterprise/dc/intra-autonomous system (AS) networks) to run distributed heterogeneous NOSes and construct a resilient peer-to-peer control plane. At the same time, we design what is the network information that needs to be exchanged and how such information should be exchanged. Considering the privacy and scalability, we propose to virtual the SDN network view and only exchange the virtualized network view... SDN domain We refer to as an SDN domain a sub-network that runs one NOS and is partitioned from a larger network. In this paper, an SDN domain can be a sub-network in a DC or an enterprise network or an AS. According to the properties of individual SDN domains, we can divide all SDN domains into four categories, referred to as SDN peers: (i) SDN AS peers; (ii) SDN intra-domain sub-network peers belonging to the same administrative domain; (iii) DCs of an individual company located at different places; and (iv) enterprise networks located at different areas connected by the wide area network. This paper mainly focuses on the latter three scenarios. The main contributions of this paper are as follows. (i) We propose that different SDN domains can run distributed heterogeneous NOSes at the same time. (ii)we design a peer-to-peer-based high-performance network view exchange mechanism named west east bridge (WEBridge). We define what network view information can be exchanged and how such information is efficiently exchanged among distributed heterogeneous SDN peers. (iii) We propose a maximum connection degree -based connection algorithm, to achieve a resilient peer-to-peer control plane of distributed heterogeneous NOSes, and give a solution to controller identity authentication. (iv) We also propose to abstract a physical SDN network into a virtual network for privacy and scalability. The rest of the paper is organized as follows: Section presents the related work. Section 3 describes the design of the WEBridge among SDN peers. Our work is implemented and deployed in Section 4 and will be evaluated in Section 5. Finally, Section 6 concludes the paper.. RELATED WORK Currently, there are mainly two kinds of NOSes: (i) single NOS and (ii) distributed NOS. Single NOS is such as Floodlight [8], NOX [0], Maestro [], Beacon [4], SNAC [], and Trema [8]. Distributed NOS is such as Onix [7], HyperFlow [8], and DIFANE [7]. In the SDN centralized control model, all the routes are determined by the NOS, so the first packet of each data flow is sent to the central NOS. Then, the NOS will compute a forwarding path for each data flow and install the forwarding path into the related OpenFlow switches according to a global network view and the real-time network state. Here, the first packet of each data flow is called the packet-in. To improve the setting up speed of flow initialization requests, some NOSes such as Beacon Security Comm. Networks 05; 8: John Wiley & Sons, Ltd. 97

3 P. Lin, J. Bi and Y. Wang and Maestro are trying to improve the performance of the controller by multi-thread technology. The NOS software system is often deployed on a multicore host or server. However, for large-scale DCs or networks, the request processing capability of a single controller is limited: (i) NOX could process about 30 K requests [3] per second; (ii) Maestro could process about 600 K requests per second. In fact, large-scale network environments always have vast amounts of data flows: (i) a 500-server cluster might generate 00 K requests per second [4]; (ii) a 00-switch DC might generate 0 million packet-in messages per second []. To achieve a scalable control plane, distributed NOSes such as HyperFlow [8], Onix [7], and DIFANE [7] are proposed. Such solutions use more than one NOS instance to form a powerful aggregate logical NOS. Then, the scalability in the control plane evolves into a cooperative performance problem of multiple NOSes. Such, the west east interface and communication has been considered as a very import module in software-defined network architecture, such as SDNi [5] and SDN at Google [6]. HyperFlow [8] uses multiple controllers to construct a distributed control plane, and each controller takes charge for a small area of network. To learn a global network view, HyperFlow adopts a distributed file system named WheelFS [5], which is designed for the wide area network environment. Each HyperFlow controller has the permissions to interact with network events within a certain local area, and the events that will affect the global network should be announced at periodic intervals. Once other controllers learn the event information, they should replay the event to achieve the synchronization of the global view. However, this approach can only deal with events that do not occur frequently, such as link status changes. For some DCs or networks, the administrator is required to know all the realtime events. This is currently not supported by HyperFlow. WEBridge proposed in this paper can also work in the HyperFlow architecture and substitute the WheelFS to better synchronize the network view of each domain. Onix [7] is a distributed controller for production network. It focuses on large scale network. It uses the distributed hash table (DHT) to store the global network information. Each database in one SDN domain just stores a part of the whole network view. All the Onix instances learn the global network view by sharing the DHT database. WEBridge proposed in this paper, however, focuses on SDN domains such as sub-networks within intra-domain networks, DCs, and enterprise networks. The target of WEBridge is to achieve a high performance and high available network view synchronization. We will show later on that in the aforementioned scenarios, the performance of WEBridge is better than the DHT used by Onix. Besides, Onix is a distributed controller, while the WEBridge is a mechanism, and it is compatible with different third-party controllers/noses with different local network view storage systems. WEBridge is designed to support different NOSes working together. In such a way, WEBridge can encourage third parties to engage in contributing to the NOS and promote competition and coexistence in the NOS s environment. Previously, Yin et al. had proposed a message exchange protocol for software-defined networks across multiple sub-network domains named SDNi [5]. That proposal only defined several basic messages such as the reachability information and the flow setup/tear-down/update processes. But it did not describe how the message should be stored and exchanged in a performance effective manner. In this paper, we explore the west east message exchange protocol and build a resilient peer-to-peer control plane of distributed heterogeneous NOSes. 3. WEST EAST BRIDGE FOR SDN PEERS The WEBridge is a high-performance network view exchange mechanism for multiple SDN domains. It is compatible with different third-party NOSes and network view information storage systems. We illustrate the WEBridge in Figure. The WEBridge mechanism mainly includes the controller or NOS discovery process, controller authentication, definition of network view information in different scenarios, network view information storage and transfer format, and a high-performance network view exchange mechanism. We will present these components one by one. A. Controller discovery Before exchanging the network view, the first important work for controllers to do is to discover each other. The algorithm for controllers from different SDN domains or sub-networks to find each other can be a distributed controller discovery algorithm or a centralized management system such as a registration center. Because this paper focuses on the SDN peers belonging to the same administrative domain (DC/enterprise/intra-AS sub-domains), a registration server is adopted by WEBridge. In this paper, because all the SDN peers belong to the same administrative domain, the network administrator can just establish a central controller list on the registration server. Each controller can obtain the information of other controllers from the controller list. The list can be defined by the administrator at their will because it is private. The controller list usually has a basic set of information. A reference design can be as follows: <Controller_ID, name, version, IP_address, is_ssl(secure Sockets Layer), is_tcp (Transmission Control Protocol), is_restapi, SSL_port, TCP_port, Rest_port >. B. Controller identity authentication Network attackers may fake controllers. A similar lesson learnt from Border Gateway Protocol (BGP) [33] 98 Security Comm. Networks 05; 8: John Wiley & Sons, Ltd.

4 P. Lin, J. Bi and Y. Wang Figure. West east bridge for distributed heterogeneous NOSes cooperation. routing scenario is the famous prefix-hijacking attack: an attacker can easily fake BGP router and announce IP prefixes that are owned by others. Because there is no router identity authentication in BGP, such attach can easily disrupt the routing system. Thus, in WEBridge, there should be a controller identity authentication mechanism. The Internet X.509 Public Key Infrastructure is commonly used to authenticate clients and servers, secure , encrypt, and digitally sign messages. WEBridge also adopts the X.509 digital certificate technology. Each controller should apply for an X.509 certificate from the certification authority and can be verified by the public. C. Network view We describe all the network entities with a network view and further divide it into two kinds: physical network view and virtual network view. () Network view abstraction Furthermore, a network view mainly includes two aspects: the network static information and dynamic information. The network static aspect includes the following information. (i) Reachability: in carrier network, reachability refers to the IP address prefixes; in DC and enterprise network, it also includes the server/host addresses. (ii) Topology: node (e.g., switch, server, host, controller, firewall, balancer, and others), link, link attributes, port throughput, and link connection. (iii) Network service capabilities: such as service level agreement (SLA), generic routing encapsulation, and SSL. (iv) Quality of service parameters, such as latency, reliability, packet loss rate, availability, throughput, time delay variation, and cost. The network dynamic information mainly includes the network status, such as FlowTable entries information in each switch, real-time bandwidth utilization in the topology, and all the flow paths in the network. () Network view storage The network view can be expressed by undirected graph with entity (node, virtual node, link, virtual link) and entity attributes. Thus, full network view can be formalized accordingly. With WEBridge, different domains could adopt different storage systems, and the data structures could be defined differently, as long as the network view messages delivered on the WEBridge have uniform labels and are in a specific format that will be showed later in this section. Considering the scalability, availability, and data I/O (input/output) access speed of the network storage system, WEBridge suggests the key-value database [0] [3] plus caching systems. Furthermore, databases with transactional function should be adopted to guarantee the data integrity. One possible key-value storage design is described in Table I. The edge in the aforementioned table refers whether this entity is the edge of a domain. Each domain uses the edge nodes and the cross domain links to construct a global network view. The node, link, and port can be physical or virtual that will be showed later in this section. (3) Network view learning Currently, Link Layer Discovery Protocol (LLDP) is used by controllers to discover the intra-domain topology. Usually, the controller in each SDN domain instructs each of the connected OpenFlow switches to send out LLDP packets in all the ports (the LLDP packet carries the source switch identity, out-port, and other capabilities). Because OpenFlow switches rely on the flow entries in FlowTable to match and forward the packets, for a new flow the Security Comm. Networks 05; 8: John Wiley & Sons, Ltd. 99

5 P. Lin, J. Bi and Y. Wang Table I. Key-value storage design. Key Columns Node_ID (physical/virtual) is_virtual (first column) IP_addresses, OF_version, port_numbers, is_edge_node, Vendor_name, MTU, Device_type, Device_function Link_ID (physical/virtual) is_virtual (first column) Node_ID_src, Port_ID_src, Node_ID_dst, Port_ID_dst, Bandwidth, is_crossdomain_link Port_ID (physical/virtual) is_virtual (first column) Node_ID, Port_MAC, is_active, is_edge_port, VLAN_ID, throughput Node_capbility protocol_name, version, port Reachability IP_prefixes, length Node_table_ID (Flow entity) Column names are the same as the fields defined in the FlowTable Link_Utilities Link_ID, link utilities Flow_path (Node_ID_src_Node_ID_dst) Port_ID (in), Node_ID_src, Port_ID (out), node series with ingress and egress ports, Port_ID (in), Node_ID_dst, Port_ID (out) OpenFlow switches normally send the first packet (packetin) to the controller. Thus, once the neighbors receive the LLDP packets, they will send it directly to the controller. Then the controller will abstract and analyze the information from the LLDP packet: (i) if the source switch identity belongs to its domain and the LLDP packet received by a neighbor is the same with the one sent out from the source OpenFlow switch. If this is true, the controller then will create a direct link between the source switch and this neighbor. (ii) For the inter-domain link, we extended the NOS by adding a network view driver module named LLDP extension shown in Figure : if the source switch identity does not belong to its domain, then the controller can infer that this packet is from other domains and will create an inter-domain link like (S6, S7) in Figure 3 according to the source switch identity, source switch out-port, and the neighbor switch (who received the LLDP packet in its SDN domain) identity with the in-port. The inter-domain links should be stored in both the neighbor domains local network views before exchanging the local network view. To learn more network view information such as OpenFlow version and number of the FlowTables on each node, link utilities, and flow entries, we extended more to the LLDP in LLDP extension. All the network view information is provided to network APPs by Representational State Transfer (REST) Application Programming Interface (API). By counting the total number of packets related to a certain port in all the FlowTables in an OpenFlow switch, the driver can learn the link utilities. By sending commands (normally, a switch provides those commands) to the OpenFlow switches, LLDP extension module can also learn the OpenFlow version, number of FlowTables, and flow entries in each switch. In addition, the LLDP extension can make a call to the OpenFlow switch functions or commands by the SSL control channel between controller and switch. We can enable the WEBridge in all kinds of NOSes by adding the three modules in red color shown in Figure. We will give a detailed explanation to network virtualization module and west east bridge module later in this section. Figure. Enable WEBridge in all kinds of NOSes by adding three modules (network virtualiztion, west east bridge, and LLDP extension modules). 930 Security Comm. Networks 05; 8: John Wiley & Sons, Ltd.

6 P. Lin, J. Bi and Y. Wang Figure 3. Physical view to virtual view abstraction (PP: physical path; VP: virtual path; OF: openflow; S: switch; bd: bandwidth; t: time; bps: bits per second). (4) Network view transfer format JavaScript Object Notation (JSON) is a text-based, open standard designed, mature, and human-readable metamarkup language described in RFC 467, which defines a set of rules for encoding documents. Other similar languages are XML (extensible Markup Language), YANG [6], and YAML [7]. This paper suggests JSON as a basic implementation, and XML, YANG, and YAML as alternatives. Those languages have the ability to enable WEBridge with the following advantages. (i) They are vendor independent and application independent. Thus, the network view message transfer format is independent to different storage systems in different SDN domains. (ii) They allow explicit definition of the inherent structure according to the requirements. Such feature makes the network view message format flexible to extend. (iii) They express the network view with file format and not data packet format. The elements of those languages are easy to extend, which means that the labels for network view messages are easy to extend. (5) Virtual network view for privacy and scalability Usually the network view refers to the entire network. However, some domains may be only willing to expose a part of its domain, rather than the entire network view, because of privacy concerns. WEBridge supports abstracting a physical network to a virtual network in such situations. As showed in Figure 3, we design two kinds of virtual network views. (i) abstract a physical network into a virtual network with only the edge switches, like network. Route path segments (such as VP, VP, and VP 3) from the ingress switch to the egress switch in the virtual network can have SLA-level path attributes such as time latency, reliability, bandwidth, and packet loss rate. The simplest way to estimate the cost of each path between each ingress and egress switch pair is by the hops of the switches, which is similar to the cost in widely deployed Open Shortest Path First protocol. (ii) Abstract a physical network into a virtual node, like network. The virtual node only retains three physical inter-domain (crossdomain) links: link, link3, and link4. After network abstraction, each NOS should store a mapping table between the physical network and the virtual network that is showed in the table of Figure 3. Without exchanging local network views, the path computing application on NOS can only compute the local routing path (just a path segment such as VP or VP) based on the local network view. When a cross-domain packet with destination IP address located in another SDN domain comes, the controller has to instruct the edge OpenFlow switches to flood the packet to all the ports on inter-domain links except the incoming one. Such kind of flooding will be repeated by other SDN domains until reaching the destination. Security Comm. Networks 05; 8: John Wiley & Sons, Ltd. 93

7 P. Lin, J. Bi and Y. Wang To compute an end-to-end or global routing path, the path computing application on NOS needs the network views in other domains or should at least know the abstracted virtual network views of other domains as shown in Figure 3. With WEBridge, after exchanging the local network views, each NOS can construct the global network view based on all the local virtual network views plus the inter-domain links and their attributes, and provide it to network APPs earlier. And then, the path computing application can compute an end-to-end path and send the cross-domain packet directly to the edge switch output port along the routing path, which is more effective than flooding. The work flow for routing a packet with WEBridge is as follows: after the ingress switch receives a packet, it will find whether there is a flow entry in its FlowTable that can be matched. If there is one, it will forward the packet according to the flow entry; otherwise, it will forward the packet to the controller. The controller will deliver it to the path computing application earlier (the path computing application is triggered off). The path computing application then according to the global network view checks whether the destination address belongs to this domain. If this is true, the path computing application will compute all the flow entries and install them into the relevant switches within its domain. Otherwise (the packet is a cross domain packet), it will compute an end-to-end path and instruct all the controllers in relevant domains along the routing path to install the corresponding path segments. After successfully installing all the path segments, the packet at last can be routed. The path segment is designed as follows: Ingress switch In-put port Egress switch Out-put port Match fields SLA attributes There are two methods for the path computing application to install the corresponding flow entries in other SDN domains. (i) the path computing application computes and generates all the corresponding flow entries for the corresponding switches in all the relevant domains along the routing path, and installs them into those switches by the corresponding REST APIs provided by each controller. The main disadvantage of this method is that those flow entries will be transmitted between SDN domains and will occupy bandwidth. (ii) The path computing application only sends a path segment installation command to each NOS in each domain and lets the NOS to install flow entries into its corresponding switches within its domain. The command should include packet match fields, ingress switch ID, in-port of the ingress switch ID, egress switch ID, out-port of the egress switch ID, actions, and SLA attributes. The second method is a lightweight solution, which is strongly recommended. To achieve large-scale scalability, each SDN domain can be abstracted to a node with multi-ports and multiple bandwidths like network. Other NOSes treat the network domain as an abstract node with three ports and three links. The type of the network view (abstracted virtual network view, full network view) that should be exchanged depends on how heavy the busy traffic is and the domain privacy policies. But the abstracted virtual network view is the minimum information for global network reachability. On the basis of the global network view, each NOS can further provide additional specific network view [] like access network view and edge network view to the network APPs earlier. D. High speed west east bridge exchange mechanism After the controller discovery process, each controller learns all the addresses of their peers. Then all the controllers can establish the network view exchange channel by Transmission Control Protocol (TCP). In OpenFlow-based networks, controllers control the SDN switches through an SSL. which is based on TCP. Similarly, WEBridge can also establish SSL connections for information transmission. In this way, the security of the information exchange process can be ensured. The virtual topology among SDN peers is shown in Figure 4. All the SDN peers are equivalent to each other and are connected as an unstructured network. We abstract all the SDN peers topology into an undirected graph G. Each peer is denoted by a vertex V, and each connection between two SDN peers is denoted by an edge E. Because of limited hardware resources such as bandwidth and computation capability, each controller might be able to handle limited connections. Here, the maximum number of connections is denoted by degree D, and the number of real-time connections is denoted by degree d. In the SDN network with (N + ) peers, if the maximum degree D of each peer equals N, then the virtual topology will be a full mesh topology. The more connections in the topology, the faster and the more reliable the data transmission will be. So the WEBridge is designed based on maximum degree connections principle, which means that in the WEBridge system, each peer should establish as many connections to its peers as it can. The workflow for a new peer joining in is shown in Figure 5. TCP/SSL connections SDN Peers with different NOSes Figure 4. TCP/SSL-based vitural topology for WEBridge. 93 Security Comm. Networks 05; 8: John Wiley & Sons, Ltd.

8 P. Lin, J. Bi and Y. Wang Each time there is a network state update, the time for synchronization among all SDN peers (system state convergence) depends on the maximum hops between each two peers: Function Sync ðþ¼max t hops V i ; V j j i ½; NŠ; j ½; NŠg On the basis of this principle, we design a new algorithm for the (N + ) peer joining in the following: Vertex selection algorithm Take the topology in Figure 4 as an example. We assume that the rest degree of each peer (A, B, C, D, E, F, G) is bigger than or at least equals. If the maximum connection degree of peer (N + ) is an even number such as, then triangular matrix X will be constructed as shown in Figure 6. Each number in the matrix means the hops of the shortest path between each pair. The maximum value in the matrix X is hops (V D,V G ) = 4. So the new peer V N+ should directly connect to V D and V G as shown in Figure 4 (the two red lines). If the maximum connection degree of peer (N +)isan odd number, it will construct a symmetric matrix Y as shown in Figure 6. The target peer in matrix Y can be found by the following formula: Max hops V x ; V j j j ½; NŠ ¼ Min Max hops V i ; V j j j ½; NŠg j i ½; NŠg From matrix Y, we can see the maximum hops of V B is, which is the smallest number compared with that in other columns. So the target is B. In such situation, the new peer V N+ should connect to peer V B as shown in Figure 4 (the blue line). The time complexity of the Vertex selection algorithm is O(n). For network events such as link failure, adding/ deleting switch, and adding/deleting IP prefixes, each controller can subscribe to other controllers database events. WEBridge adopts the published/subscribed mechanism to deliver update messages. Once an event is triggered, the corresponding controller should push the event to all the subscribers simultaneously. Each controller can obtain the update messages directly from the controllers it cares. In which, the Update message is delivered by the format of JSON file, and the rest of the messages are delivered by the format of data packet. Compared with the BGP, the WEBridge system mainly changes the Update message into JSON file format and simplifies the finite state machine. NOS can also use the View-Refresh message to fetch the entire network view information from its peers. As shown in Figure 7, from the finite state machine, NOS peers use TCP to setup connections. After a TCP connection is established, the first message sent is the Open message. Some parameters such as timers should be negotiated by the Open message. After the Open message is confirmed, NOS will keep on sending out Keepalive message. WEBridge defines a timer for the Keepalive message. If one peer has not been receiving Keepalive packet from a certain NOS for a while, this peer will think that the peer failed and will release this connection. All the errors will lead to the idle state and Notification message will be sent to all the peers. Under normal circumstances, each NOS keeps sending out Keepalive messages to its peers in parallel so that it can remain in connection with all its peers. Once network view updates within its domain appear, it can directly send Update file out to all its peers in parallel without resetting Security Comm. Networks 05; 8: John Wiley & Sons, Ltd. 933

9 P. Lin, J. Bi and Y. Wang New Peer Register itself & Fetch peer list Traverse all peers, Get neighbors and D num > DN+ Calculate num of peers with R(D)>= num <= DN+ Connect to peers whose R(D) >= Vertex Selection Algorithm Find out DN+ peers & Connect Calculate Rest(D) R(D)=(D-current(d)) Stop Figure 5. Workflow for a new peer joining in. G F E D C B A X X 0 X X X X X X X X X X X X X X X X X X X G F E D C B A A B C D E F G Matrix X A B C D E F G Matrix Y Figure 6. Two kinds of matrixes for finding out the target peers. Figure 7. Finite state machine for the network view exchange protocol. the TCP connections. Furthermore, each Update file can carry multiple Update messages. In such a way, WEBridge can achieve highly efficient update message exchange. 3.. Recovery strategy During network failure, usually, peers detect the connectivity by the Keepalive message. Once there is no Keepalive messages from a peer for a while, then this peer can be judged to be unreachable. There are two main kinds of network failures: Single Node/NOS/controller failure (SNF); Single virtual link failure (SVLF) (such as network unreachable or a peer decreases the maximum degree D and drop connections when the network updates are too much). 934 Security Comm. Networks 05; 8: John Wiley & Sons, Ltd.

10 P. Lin, J. Bi and Y. Wang Therefore, we propose a recovery strategy for such network failures: when there is a network failure or the network connection degree of a peer drops, then the affected peers should find new peers to connect. It should still use the Vertex selection algorithm on the basis of the new network graph after the failure. Furthermore, we suggest the network connection degree to have a minimum value of 6 for a 0-peer network and 7 for a 00-peer network, which will be explained in Section 5. degree value as 6 in the configuration file for the maximum connection degree connection algorithm. The network view and the reachability information are transferred in JSON format as follows: 4. IMPLEMENTATION AND DEPLOYMENT A. Implementation and experiment results WEBridge has been implemented in two open-source NOSes: Floodlight [8] and NOX [0]. We use Mininet [5] to emulate two DCs. One is located in Beijing, China, and the other is located in California, USA, as shown in Figure 8. The OpenFlow switches in network are Open vswitches [].We divide the entire network of Beijing DC into four sub-networks and divide the California DC into three sub-networks. Each sub-network runs one NOS and one database. We run a centralized register center server for WEBridge in sub-network 6 in California. We also run a topology viewer application, a video server process on this server. Each controller (Floodlight or NOX) has a WEBridge module, and each WEBridge module boots together with its controller. Each WEBridge registers itself in the register center server and fetches other WEBridges information (WEBridge, which are located in other subnets). Then all the WEBridges from all the subnets setup connections based on the maximum connection degree algorithm and exchange the reachability (IP prefixes) and network view information with each other. Because there are only seven sub-networks, we set the maximum connection Furthermore, in control plane, the peer-to-peer protocol between all WEBridges is based on the maximum connection degree algorithm with TCP plus the finite state machine that was introduced in Figure 7. The communications for network view and reachability information exchange are based on the REST API (Application Programming Interface). Through analysis to the related work, we observed when other researchers design the distributed control plane, the most successful and well-received technology was DHT, which is a distributed storage system used in distributed controller at present. Similarly to WEBridge, DHT supports multiple controller instances to work together and construct a distributed control plane: A controller instance in distributed control plane writes its corresponding sub-network information into the DHT, and then other controller instances can read this network information from DHT because DHT is a distributed database. This section focuses on the performance experiment comparison between WEBridge and DHT. We will give a detailed analysis to them in the evaluation section. In the first experiment, all the NOSes are Floodlight software, and all databases are Cassandra [] DHT databases. We use the topology viewer application to fetch Figure 8. Experiment topology. Security Comm. Networks 05; 8: John Wiley & Sons, Ltd. 935

11 P. Lin, J. Bi and Y. Wang the entire topology of all the sub-networks. Then in the second experiment, the Controllers in sub-network, 3, 4, 5, 7 run Floodlight and the controllers in sub-network, 6-run NOX. In addition, we use the Cassandra single PC (personal computer) mode, and all the SDN domains or sub-networks are connected by WEBridge. We use the topology viewer application to fetch the entire topology of all the sub-networks again. The results are shown in Figure 9. As shown in Figure 9, the heterogeneous WEBridge system with maximum degree connections has two obvious advantages: () Low time delay. Once the requests are received from APPs, the entire system can respond very quickly. Because in WEBridge, each database stores the entire network topology locally, while the DHT mode database stores the entire network topology in distributed database located in different places. When a DHT database node receives a data request, it has to fetch other data from different places and then assemble all the data and then return it to the application. () For the same size topology, the WEBridge system can finish all the data transmission earlier than the DHT system. That is because the DHT system is a little slower at connection setup. So it needs longer time to finish transmission. Local network view in sub-networks, 3, 4, 6, 7 is bytes, and in sub-networks, 5 is 337 bytes. The global network view is.4 Kbytes. The total size transmitted is.4 KBytes 6 = 7.44 KBytes (6 is the connection degree). So WEBridge is a lightweight solution and can enable different NOSes to cooperate. B. Two use cases of WEBridge We carried out two use cases to prove the feasibility of WEBridge: (i) a deployable inter-domain IP source address anti-spoofing approach [9] and (ii) a new policy routing approach named pathlet routing [3]. We redesigned and implemented both of them to the SDN network with WEBridge. Both of them are implemented as network APPs running on controller. The former is called DIA- APP, and the latter is called Pathlet-APP. () Case: source address verification application 4.. Overview of DIA DIA is a deployable approach for inter-as IP source address anti-spoofing and is designed for traditional IP networks. The high level architecture of DIA is shown in Figure 0. DIA includes two planes: data plane and control plane. In data plane, the border routers in source domain add tags (signatures) to packets, and the border routers in destination domain verify and remove the tags. In control plane, DIA submits and distributes the registration information from federation members: the announcement of deployment of the address prefixes, negotiation, update, and synchronization of the infinite state machine, synchronization of time, formulation, and deployment of strategies. The whole process is transparent to the end users. In this paper, the DC/enterprise/ intra-as sub-domains all belong to the same operator. The DIA-APP can be designed as a central working mode or a distributed working mode. We adopt the latter one. The redesign of DIA in SDN with WEBridge is as follows: () In DIA, each AS in the alliance has a central Control Server (ACS). In SDN, we use the DIA- APP to substitute the ACS server. For the registration center of DIA, we reuse the centralized WEBridge registration center server. We write and run a DIA registration process on this server. () In data plane, we use the border OpenFlow switches in the test environment as the AS Edge Routers (AERs), and by instructions in OpenFlow FlowTable to modify packets: adding, verifying and removing signature tags. (3) In control plane, the specification and the design document of DIA formulate a series of new packets and fields. Because WEBridge uses the JSON file to Figure 9. Performance of DHT and WEBridge for data synchronization. 936 Security Comm. Networks 05; 8: John Wiley & Sons, Ltd.

12 P. Lin, J. Bi and Y. Wang Figure 0. Architecture of DIA, REG (registration center), ACS (AS central control server), AER (AS edge router). express the message during communications. Thus, we transformed all the new packets and fields to the JSON format and defined the corresponding labels named DIA-label. (4) The message (packets modification instructions) delivery between data plane and control plane is achieved by communication between the DIA- APP, and the border OpenFlow switches with NOS s REST API. In detail, DIA signatures are stored in the identification field in the IP header. So we extended the match fields in Open vswitch by adding a nw_identification field. Also we extended the actions in Open vswitch and added a mod_nw_identification action. (5) For this DIA application, different controllers need to exchange a specific network view through the WEBridge: the edge network topology with the border switches information. (6) Furthermore, to confirm that the DIA-APP works correctly, DIA-APP registers the flow table exchange events for all the border switches and analyzes whether all the filter flow entries installed on the edge switches take effect. After those steps, we observed that when the user uses the IP addresses belonging to the corresponding subnetworks to visit the video server located in California, he or she was successful. Otherwise, he would fail. By analyzing the flow tables in the border switches located in California, we can see that all the filter flow entries are successfully installed. Then we can conclude that DIA works well in SDN network with the WEBridge. () Case : pathlet routing application. 4.. Overview of pathlet routing In pathlet routing, networks can advertise fragments of end-to-end paths (ingress egress pairs) with local policies, and then a source can assemble an end-to-end route to achieve the source-controlled routing. Similar to DIA-APP, the pathlet-app can be designed as a central working mode or a distributed working mode. In this case, we also adopt the latter one. The redesign of pathlet routing in SDN network with WEBridge is as follows: () As described in Section 3.C, WEBridge supports virtual network view for privacy or scalability. For the pathlet-app, different controllers need to exchange a virtual network view with SLA path attributes through the WEBridge. () Then pathlet-app in each domain learns the pathlets information for the local domain and other external domains from the WEBridge in local controller. (3) Pathlet-APPs in different domains can set up connections and negotiate how data flows transit each other. For example, for data flow with source IP address , pathlet-app A in SDN domain A hopes SDN domain B to route it with flow path whose transit time delay is less than s. For data flow with destination IP address , pathlet-app A hopes domain B to route it with bandwidth larger than Mbyte. So Security Comm. Networks 05; 8: John Wiley & Sons, Ltd. 937

13 P. Lin, J. Bi and Y. Wang pathlet-app A will tell such pathlets decision information to Pathlet-APP B. (4) After the negotiation, pathlet-app B will insert such flow path entries into relevant OpenFlow switches in domain B. SLA attributes through WEBridge. At last, the two pathletapps reached an agreement to the following XML file by the SSL channel: We carry out the measurement in sub-network 5 and sub-network 6 in California, and the simplified topology and configuration are shown in Figure. Before starting the WEBridge and pathlet-app, we first inserted a static OpenFlow entry into the FlowTable of S4 and told S4 that all the flows from the port connected with the video server were directly sent to the edge OpenFlow switch S7. In this way, the video stream data were sent by the path: video server S4 S7 S6 S user. As the bandwidth between S7 and S4 is just 00 Kbytes, the end user could not watch the movie at a smooth speed. Then, we deleted the entry in the FlowTable of S4, defined the following tags for pathlet-app: video_requirement, from_ip, to_ip, min_bandwidth, and configured the address of user as Then we started the WEBridge and pathlet-app. Controller 5 in OpenFlow sub-network 5 sets up an SSL connection with the controller 6 in OpenFlow sub-network 6 and exchanges the virtual network view with After this, the user visited the video server again, and then the Controller 6 used the path video server S4 S3 S S7 S6 S user to transfer the video data, and this time, the user could enjoy the high definition movie. Meanwhile, we used a traffic monitor tool to sample the video file downloading speed. The speeds before and after running WEBridge/pathlet-APP are shown in Figure. The WEBridge/pathlet-APP use case demonstrates the feasibility and the high performance of the whole OpenFlow Sub 6 Controller 6 User 00M S4 G S6 00M OpenFlow Sub 5 S7 00M 00KB S3 S 80M 00M 50M S Controller 5 Video Server Figure. Simplified topology and configuration in sub-network 5 and 6, S: open flow switch. Figure. File download speeds before and after WEBridge/pathlet-APP. 938 Security Comm. Networks 05; 8: John Wiley & Sons, Ltd.

14 P. Lin, J. Bi and Y. Wang WEBridge system again. With WEBridge, it is easy to achieve the new network features (such as policy routing in this example), which are hard to achieve in traditional IP networks. C. Deployment As shown in Figure 3, we deployed WEBridge in three SDN networks: SDN networks in CERNET, INTERNET, and CSTNET. WEBridge successfully connected the three SDN networks: NOS in each network exchanges the reachability and topology information on the WEBridge with its peers, and the global network view is constructed. Then, path computing APPs corporate to compute and setup end-to-end paths. And then, the clients and servers can ping each other. Such deployment to the real SDN network proves the feasibility of WEBridge. 5. EVALUATION WEBridge uses X.509 for NOS identity authentication and uses SSL channel for JSON-based network view transfer. Thus, the security of network view from each domain can be ensured. WEBridge is designed for enterprise/dc/ AS networks, so that it should have high performance and high scalability in large AS. In the following sections, we evaluate it in different aspects. A. Innovativeness According to the best of our knowledge, this is the first time to propose heterogeneous controllers from different vendors working together, in the open source community, to encourage third parties to engage in contributing to the NOS and promote competition and coexistence in the NOS s environment. B. Performance analysis Besides the performance experiment comparison in Section 4 (Figure 9), in this section, we analyze the performance of WEBridge by further comparing it with DHT. The metrics for comparisons are as follows Speed of network view updates. WEBridge adopts the subscribed/published model. Usually, all the peers are in the established TCP connection state. Once a network view in a domain changes, the controller directly pushes the change to its peers with time return(t). Each controller stores the global network view. Figure 3. Current deployment. Security Comm. Networks 05; 8: John Wiley & Sons, Ltd. 939

15 P. Lin, J. Bi and Y. Wang The APPs can directly access the data in local storage. For storage in DHT, entire network view is in distributed locations. DHT needs to fetch the data and assemble them. So the time is {request(t) + return(t)}. Thus, WEBridge is request(t) faster than DHT Network view updates bandwidth cost. If we define the update frequency as f per second, and the number of NOSes is n, the amount of network view change in each domain is f change (view). Then, in time /f, WEBridge needs to transfer {n (n ) f change (view)} data, while DHT has to transfer {n (n ) [f change (view)+f size (request packet)]}. Thus, WEBridge transfers n (n ) f size (request packet) less than that of DHT Size of network view storage. We assume the size of the global SDN network view information is X. In DHT, each domain just stores X = n.in reality, the controller needs the entire view of the network when computing path. It stores the rest size ofðx X = nþ outside the DHT table and in memory. So the total storage sizes in DHT and WEBridge are the same in each domain. WEBridge can also store the local network view of X = n in high speed database and store the rest in relative lower Symbol x y z N Table II. Symbol definition. Meaning Requests a controller can deal in one second Requests a computer generates in one second Number of computers one person has Number of people in an AS/DC/enterprise speed database if there is a resource limitation in high speed database Data query speed. The WEBridge and DHT are both based on the keyvalue storage system, but DHT has routing issues (multi-hop routing), while WEBridge is single-hop routing in more than 99.5% scenarios (will be shown in Section 5.C). Here a hop may really mean multiple physical hops in the underlying network, so WEBridge is faster than DHT in the data query speed. In summary, WEBridge and DHT are designed for different deployment scenarios. For example, Onix adopts the DHT storage and focuses on large-scale network, while WEBridge focuses on enterprise/dc/intra-domain networks. DHT is more scalable than WEBridge for large-scale networks. WEBridge is more efficient than DHT when they are working in enterprise/dc/intra-domain networks. C. Scalability WEBridge supports the network abstraction strategy to improve scalability. As shown in Section 3.C, WEBridge supports network view abstraction. A large network can be abstracted into a small virtual network only with the edge switches, or even into a virtual node. To achieve large-scale scalability, each NOS should treat other SDN networks as an abstract node with multiple-ports. So the global network view to a controller is a full local network view plus all the abstracted network views from all the peers. We use a mathematical model to evaluate the Table III. Number of domain and connection degree. People/customer (online) Domain number Connection degree Campus 00, Enterprise 500, DC or AS 3,000, Table IV. Minimum degree suggestions for different N. Value of N Minimum degree suggestion 0 6 ~ ~ ~ 0,000 9 >0,00 log N + 5 Figure 4. Relationship between N and k. 940 Security Comm. Networks 05; 8: John Wiley & Sons, Ltd.

16 P. Lin, J. Bi and Y. Wang Figure 5. Simulation results of SNF or SVLF, N = 00 (we choose 00 because 99.5% of all the ASes have less than 00 peers). intra-domain level scalability, and the symbols are defined in Table II. Nzy Then there will be about = x domains, and the total number of connections (with maximum degree ð Nzy = x Þ)is Nzy x Nzy = x Þ Normally, one client uses one computer at a time, and we assume each computer generate request per second ( rps). We use the well-known NOX as controller, and NOX could process about 30 K rps [3]. The number of online people or customer on campus are usually less than 00,000, while in enterprise, it is usually less than 500,000; in DC/AS, it is usually less than 3,000,000 [4]. The 6-bit TCP port can support 6 = connections in theory, and it is very easy to support 3,000,000 (the number of online users in 99.5% AS is less than 3,000,000 [4]) online users with 99 connections. Thus, the estimate of Table III is feasible. With the WEBridge, about 99.5% of all the AS can use the connection degree ð Nzy = x Þ, which means full mesh topology, and about 0.5% AS need to use connection degree less than ð Nzy = x Þ. D. Availability WEBridge stores the entire network view in every domain. If a WEBridge instance in one domain fails, it can quickly recover by using the network view from others. WEBridge uses the finite state machine to start connection; once there is a failure, it will repeat the connection set-up process until it succeeds. Unlike the physical network, the topology of SDN peers is unstructured, and the neighbor can be changed dynamically. According to the reliability analysis result of the random multicast protocol Gossip [30], if there are N nodes, and each node randomly delivers message to other (log N + k) nodes, then the probability of everyone receiving the message converges to exp( exp( k)). As shown in Figure 4, when k = 5, the probability of everyone receiving the message converges to 99.3%. So we suggest the minimum value of k to be 5. This means that when k 5, the probability of keeping the SDN peer network connected is more than 99.3%, independent of SNF or SVLF. So the minimum degree we suggest for a N peer SDN network is (log N + 5). According to the analysis in Section 5.C, 99.5% of all the AS have less than 00 SDN peers. So the minimum degree we suggest is log = 7. More suggestions are listed in Table IV. So according to the hardware resource, the network operator can configure the connections from the minimum degree to (N ). However, once the hardware resource is enough, we suggest the connection degree to be (N ). The bigger the degree is, the more reliable the SDN peer network. 5.. Connection simulation with two kinds of network failure (SNF, SVLF) According to the Sharp threshold theory in random graph, If G is a k-connected graph of n vertices and the edge probability p(n) satisfies p(n) c log(n/k) for large enough absolute constant c > 0, then almost surely the random sub-graph G p is connected [3]. We simulate SVLF for the graph connection on the basis of this theory and SNF on the basis of the proposed vertex selection algorithm. From Figure 5, we can see that when the connection degree is 3, the probability of network disconnection is already near 0. So the degree value we suggest is 7 with N = 00, which is a very conservative value for 00 peers. 6. CONCLUSION AND CURRENT DEPLOYMENT This paper designs a high-performance mechanism for distributed heterogeneous NOSes to construct a resilient peer-to-peer control plane, to exchange network view in enterprise/dc/intra-as networks. It also defines what Security Comm. Networks 05; 8: John Wiley & Sons, Ltd. 94

17 P. Lin, J. Bi and Y. Wang information should be shared and how it should be shared. The implementation and deployment on SDN networks in CERNET (China Education and Research Network), Internet in USA, and CSTNET (China Science and Technology Network) proves the feasibility of WEBridge. The evaluation proves that WEBridge is a high-performance mechanism. ACKNOWLEDGEMENTS This work is supported by the National High-tech R&D Program ( 863 Program) of China(No.03AA00605), National Science and Technology Pillar Program of China (No.0BAH0B0). REFERENCES. McKeown N. Keynote talk: Software-defined networking. IEEE INFOCOM 09, Open Networking Laboratory: Open Networking Foundation: Beacon: Yin H, Xie H, Tsou T, Lopez D, Aranda PA, Sidi R. SDNi: a message exchange protocol for software defined networks (SDNs) across multiple domains. IETF Internet-Draft, SDN at Google, : proceedings/84/slides/slides-84-sdnrg-4.pdf, 7. Koponen T, Casado M, Gude N, et al. Onix: a distributed control platform for large-scale production networks. InOSDI,00;0: Tootoocian A, Ganjali Y. HyperFlow: a distributed control plane for OpenFlow. In INM/WREN workshop, McKeown N, Anderson T, Balakrishnan H, et al. OpenFlow: enabling innovation in campus networks. ACM SIGCOMM Computer Communication Review 008; 38(): DeCandia G, Hastorun D, Jampani M, et al. Dynamo: amazon s highly available key value store. Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, 007. DOI: 0.45/ Open vswitch: Cassandra: Schütt T, Schintke F, Reinefeld A. Scalaris: reliable transactional pp key/value store. Proceedings of the 7th ACM SIGPLAN workshop on ERLANG, 008. DOI: 0.45/ Caida: Stribling J, Sovran Y, Zhang I, et al. Flexible, widearea storage for distributed systems with wheelfs. In Proceedings of the 6th USENIX Symposium on Networked Systems Design and Implementation (NSDI 09), ONOS: Yu M, Rexford J, Freedman MJ, Wang J. Scalable flow-based networking with DIFANE. In Proceedings of SIGCOMM, 00. DOI: 0.45/ Floodlight: Xie H, Tsou T, Lopez D, Yin H, Gurbani V. Use cases for alto with software defined networks. IETF Internet- Draft, Gude N, Koponen T, Pettit J, et al. NOX: towards an operating system for networks. In ACM SIGCOMM Computer Communication Review, 008; 38(3): Cai Z, Cox AL, Ng TSE. Maestro: a system for scalable openflow control. Tech. Rep. TR0-08, Rice University, 00.. Simple network access control, : Tavakoli A, Casado M, Koponen T, Shenker S. Applying NOX to the datacenter. In Proceedings of workshop on Hot Topics in Networks (HotNets-VIII), Kandula S, Sengupta S, Greenberg A, Patel P. The nature of datacenter traffic: measurements & analysis. In Proceedings of IMC, Mininet, : bin/view/openflow/mininet. 6. YANG: YAML: TREMA: Liu B, Bi J, Zhu Y. A deployable approach for inter- AS anti-spoofing. In Proceedings of the 9th IEEE International Conference on Network Protocols, Kermarrec A-M, Massouli e L. Probabilistic reliable dissemination in large-scale systems. IEEE Transactions on Parallel and Distributed Systems, 003; 4(3): Krivelevich M, Sudakov B, Vu H. A sharp threshold for network reliability. Combinatorics, Probability and Computing, 00; (5): Brighten Godfrey P, Ganichev I, Shenker S, Stoica I. Pathlet routing. In Proceedings of the ACM SIGCOMM 009 conference on Data communication, 009. DOI: 0.45/ BGP [RFC 47], : rfc47.txt. 94 Security Comm. Networks 05; 8: John Wiley & Sons, Ltd.