Floodless in SEATTLE: A Scalable Ethernet Architecture for Large Enterprises

Transcription

1 Floodless in SEATTLE: A Scalable Eternet Arcitecture for Large Enterprises Cangoon Kim Mattew Caesar Jennifer Rexford Princeton University Princeton University Princeton University Abstract IP networks today require massive effort to configure and manage. Eternet is vastly simpler to manage, but does not scale beyond small local area networks. Tis paper describes an alternative network arcitecture called SEATTLE tat acieves te best of bot worlds: Te scalability of IP combined wit te simplicity of Eternet. SEATTLE provides plug-and-play functionality via flat addressing, wile ensuring scalability and efficiency troug sortest-pat routing and as-based resolution of ost information. In contrast to previous work on identity-based routing, SEATTLE ensures pat predictability and stability, and simplifies network management. We performed a simulation study driven by realworld traffic traces and network topologies, and used Emulab to evaluate a prototype of our design based on te Click and XORP open-source routing platforms. Our experiments sow tat SEATTLE efficiently andles network failures and ost mobility, wile reducing control overead and state requirements by rougly two orders of magnitude compared wit Eternet bridging. 1. Introduction Eternet stands as one of te most widely used networking tecnologies today. Due to its simplicity and ease of configuration, many enterprise and access provider networks utilize Eternet as an elementary building block. Eac ost in an Eternet is assigned a persistent MAC address, and Eternet bridges automatically learn ost addresses and locations. Tese plug-and-play semantics simplify many aspects of network configuration. Flat addressing simplifies te andling of topology canges and ost mobility, witout requiring administrators to perform address reassignment. However, Eternet is facing revolutionary callenges. Today s layer-2 networks are being built on an unprecedented scale and wit igly demanding requirements in terms of efficiency and availability. Large data centers are being built, comprising undreds of tousands of computers witin a single facility [1], and maintained by undreds of network operators. To reduce energy costs, tese data centers employ virtual macine migration and adapt to varying workloads, placing additional requirements on agility (e.g., ost mobility, fast topology canges). Additionally, large metro Eternet deployments contain over a million osts and tens of tousands of bridges [2]. Eternet is also being increasingly deployed in igly dynamic networks, for example as backaul for wireless campus networks, and in transport networks for developing regions [3]. Wile an Eternet-based solution becomes all te more important in tese environments because it ensures service continuity and simplifies configuration, conventional Eternet as some critical limitations. First, Eternet bridging relies on network-wide flooding to locate end osts. Tis results in large state requirements and control message overead tat grows wit te size of te network. Second, Eternet forces pats to comprise a spanning tree. Spanning trees perform well for small networks wic often do not ave many redundant pats anyway, but introduce substantial inefficiencies on larger networks tat ave more demanding requirements for low latency, ig availability, and traffic engineering. Finally, critical bootstrapping protocols used frequently by end osts, suc as Address Resolution Protocol (ARP) and Dynamic Host Configuration Protocol (DHCP), rely on broadcasting. Tis not only consumes excessive resources, but also introduces security vulnerabilities and privacy concerns. Network administrators sidestep Eternet s inefficiencies today by interconnecting small Eternet LANs using routers running te Internet Protocol (IP). IP routing ensures efficient and flexible use of networking resources via sortestpat routing. It also as control overead and forwardingtable sizes tat are proportional to te number of subnets (i.e., prefixes), rater tan te number of osts. However, introducing IP routing breaks many of te desirable properties of Eternet. For example, network administrators must now subdivide teir address space to assign IP prefixes across te topology, and update tese configurations wen te network design canges. Subnetting leads to wasted address space, and laborious configuration tasks. Altoug DHCP automates ost address configuration, maintaining consistency between DHCP servers and routers still remains callenging. Moreover, since IP addresses are not persistent identifiers, ensuring service continuity across location canges (e.g., due to virtual macine migration or pysical mobility) becomes more callenging. Additionally, access-control policies must be specified based on te ost s current position, and updated wen te ost moves. Alternatively, operators may use Virtual LANs (VLANs) to build IP subnets independently of ost location. Wile te overead of address configuration and IP routing may be reduced by provisioning VLANs over a large number of, if not all, bridges, doing so reduces benefits of broadcast scoping, and worsens data-plane efficiency due to larger spanning trees. Efficiently assigning VLANs over bridges and links must also consider osts communication and mobility pat-

2 terns, and ence is ard to automate. Moreover, since osts in different VLANs still require IP to communicate wit one anoter, tis arcitecture still inerits many of te callenges of IP mentioned above. In tis paper, we address te following question: Is it possible to build a protocol tat maintains te same configuration-free properties as Eternet bridging, yet scales to large networks? To answer tis question, we present a Scalable Eternet Arcitecture for Large Enterprises (SEATTLE). Specifically, SEATTLE offers te following novel features: A one-op, network-layer DHT: SEATTLE forwards packets based on end-ost MAC addresses. However, SEATTLE does not require eac switc to maintain state for every ost, nor does it require network-wide floods to disseminate ost locations. Instead, SEATTLE uses te global switc-level view provided by a link-state routing protocol to form a oneop DHT [4], wic stores te location of eac ost. We use tis network-layer DHT to build a flexible directory service wic also performs address resolution (e.g., storing te MAC address associated wit an IP address), and more flexible service discovery (e.g., storing te least loaded DNS server or printer witin te domain). In addition, to provide stronger fault isolation and to support delegation of administrative control, we present te design of a ierarcical, multi-level one-op DHT. Traffic-driven location resolution and cacing: To forward packets along sortest pats and to avoid excessive load on te directory service, switces cace responses to queries. In enterprise networks, osts typically communicate wit a small number of oter osts [5], making cacing igly effective. Furtermore, SEATTLE also provides a way to piggyback location information on ARP replies, wic eliminates te need for location resolution wen forwarding data packets. Tis allows data packets to directly traverse te sortest pat, making te network s forwarding beavior predictable and stable. A scalable, prompt cace-update protocol: Unlike Eternet wic relies on timeouts or broadcasts to keep forwarding tables up-to-date, SEATTLE proposes an explicit and reliable cace update protocol based on unicast. Tis ensures tat all packets are delivered based on up-to-date state wile keeping control overead low. In contrast to conventional DHTs, tis update process is directly triggered by networklayer canges, providing fast reaction times. For example, by observing link-state advertisements, switces determine wen a ost s location is no longer reacable, and evict tose invalid entries. Troug tese approaces, SEATTLE seamlessly supports ost mobility and oter dynamics. Despite tese features, our design remains backwardscompatible wit existing applications and protocols running at end osts. For example, SEATTLE allows osts to generate broadcast ARP and DHCP messages, and internally converts tem into unicast-based queries to a directory service. SEATTLE switces can also andle general (i.e., non- ARP and non-dhcp) broadcast traffic troug loop-free multicasting. To offer broadcast scoping and access control, SEATTLE also provides a more scalable and flexible way to create VLANs tat reduces manual configuration overead. 1.1 Related work Our quest is to design, implement, and evaluate a practical replacement for Eternet tat scales to large and dynamic networks. Altoug tere are many approaces to enance Eternet bridging, none of tese are suitable for our purposes. SmartBridges [6] and RBridges [7, 8] leverage a link-state protocol to disseminate information about bot bridge connectivity and ost state. Tis eliminates te need to maintain a spanning tree and improves forwarding pats. CMU-Eternet [9] also leverages link-state, but eliminates per-ost broadcasting by propagating ost information in link-state updates. Viking [10] uses multiple spanning trees for faster fault recovery, wic can be dynamically adjusted to conform to canging load. Toug SEATTLE was inspired by te problems addressed in tese works, it takes a radically different approac tat eliminates network-wide dissemination of per-ost information. Tis results in substantially improved control-plane scalability and data-plane efficiency. Wile tere ave been works on using asing to support flat addressing conducted in parallel wit our work [11, 12, 13], tese works do not promptly andle ost dynamics, require some packets to be detoured away from te sortest pat or be forwarded along a spanning tree, and do not support ierarcical configurations to ensure fault/pat isolation and te delegation of administrative control necessary for large networks. Te design we propose is also substantially different from recent work on identity-based routing (ROFL [14], UIP [15], and VRR [16]). Our solution is suitable for building a practical and easy-to-manage network for several reasons. First, tese previous approaces determine pats based on a as of te destination s identifier (or te identifier itself), incurring a stretc penalty (wic is unbounded in te worst case). In contrast, SEATTLE does not perform identity-based routing. Instead, SEATTLE uses resolution to map a MAC address to a ost s location, and ten uses te location to deliver packets along te sortest pat to te ost. Tis reduces latency and makes it easier to control and predict network beavior. Predictability and controllability are extremely important in real networks, because tey make essential management tasks (e.g., capacity planning, troublesooting, traffic engineering) possible. Second, te pat between two osts in a SEATTLE network does not cange as oter osts join and leave te network. Tis substantially reduces packet reordering and improves constancy of pat performance. Finally, SEATTLE employs trafficdriven cacing of ost-information, as opposed to te trafficagnostic cacing (e.g., finger caces in ROFL) used in previous works. By only cacing information tat is needed to forward packets, SEATTLE significantly reduces te amount of state required to deliver packets. However, our design also consists of several generic components, suc as te multilevel one-op DHT and service discovery mecanisms, tat 2

3 could be reused and adapted to te work in [14, 15, 16]. Roadmap: We summarize ow conventional enterprise networks are built and motivate our work in Section 2. Ten we describe our main contributions in Sections 3 and 4 were we introduce a very simple yet igly scalable mecanism tat enables sortest-pat forwarding wile maintaining te same semantics as Eternet. In Section 5, we enance existing Eternet mecanisms to make our design backwardscompatible wit conventional Eternet. We ten evaluate our protocol using simulations in Section 6 and an implementation in Section 7. Our results sow tat SEATTLE scales to networks containing two orders of magnitude more osts tan a traditional Eternet network. As compared wit ROFL, SEATTLE reduces state requirements required to acieve reasonably low stretc by a factor of ten, and improves pat stability by more tan tree orders of magnitude under typical workloads. SEATTLE also andles network topology canges and ost mobility witout significantly increasing control overead. 2. Today s Enterprise and Access Networks To provide background for te remainder of te paper, and to motivate SEATTLE, tis section explains wy Eternet bridging is limited to small LANs. Ten we describe ybrid IP/Eternet networks and VLANs, two widely-used approaces wic improve scalability over conventional Eternet, but introduce management complexity, eliminating many of te plug-and-play advantages of Eternet. 2.1 Eternet bridging An Eternet network is composed of segments, eac comprising a single pysical layer 1. Eternet bridges are used to interconnect multiple segments into a multi-op network, namely a LAN, forming a single broadcast domain. Eac ost is assigned a unique 48-bit MAC (Media Access Control) address. A bridge learns ow to reac osts by inspecting te incoming frames, and associating te source MAC address wit te incoming port. A bridge stores tis information in a forwarding table tat it uses to forward frames toward teir destinations. If te destination MAC address is not present in te forwarding table, te bridge sends te frame on all outgoing ports, initiating a domain-wide flood. Bridges also flood frames tat are destined to a broadcast MAC address. Since Eternet frames do not carry a TTL (Time-To-Live) value, te existence of multiple pats in te topology can lead to broadcast storms, were frames are repeatedly replicated and forwarded along a loop. To avoid tis, bridges in a broadcast domain coordinate to compute a spanning tree tat is used to forward frames [17]. Administrators first select and configure a single root bridge; ten, te bridges collectively compute a spanning tree based on distances to te root. Links not present in te tree are not used to carry traffic, causing longer pats and inefficient use of resources. Unfortunately, Eternet-bridged networks cannot grow to a large scale due to following reasons. 1 In modern switced Eternet networks, a segment is just a point-to-point link connecting an end ost and a bridge, or a pair of bridges. Globally disseminating every ost s location: Flooding and source-learning introduce two problems in a large broadcast domain. First, te forwarding table at a bridge can grow very large because flat addressing increases te table size proportionally to te total number of osts in te network. Second, te control overead required to disseminate eac ost s information via flooding can be very large, wasting link bandwidt and processing resources. Since osts (or teir network interfaces) power up/down (manually, or dynamically to reduce power consumption), and cange location relatively frequently, flooding is an expensive way to keep per-ost information up-to-date. Moreover, malicious osts can intentionally trigger repeated network-wide floods troug, for example, MAC address scanning attacks [18]. Inflexible route selection: Forcing all traffic to traverse a single spanning tree makes forwarding more failure-prone and leads to suboptimal pats and uneven link loads. Load is especially ig on links near te root bridge. Tus, coosing te rigt root bridge is extremely important, imposing an additional administrative burden. Moreover, using a single tree for all communicating pairs, rater tan sortest pats, significantly reduces te aggregate trougput of a network. Dependence on broadcasting for basic operations: DHCP and ARP are used to assign IP addresses and manage mappings between MAC and IP addresses, respectively. A ost broadcasts a DHCP-discovery message wenever it believes its network attacment point as canged. Broadcast ARP requests are generated more frequently, wenever a ost needs to know te MAC address associated wit te IP address of anoter ost in te same broadcast domain. Relying on broadcast for tese operations degrades network performance. Moreover, every broadcast message must be processed by every end ost; since andling of broadcast frames is often application or OS-specific, tese frames are not andled by te network interface card, and instead must interrupt te CPU [19]. For portable devices on low-bandwidt wireless links, receiving ARP packets can consume a significant fraction of te available bandwidt, processing, and power resources. Moreover, te use of broadcasting for ARP and DHCP opens vulnerabilities for malicious osts as tey can easily launc network-wide ARP or DHCP floods [9]. 2.2 Hybrid IP/Eternet arcitecture One way of dealing wit Eternet s limited scalability is to build enterprise and access provider networks out of multiple LANs interconnected by IP routing. In tese ybrid networks, eac LAN contains at most a few undred osts tat collectively form an IP subnet. An IP subnet is given an IP prefix representing te subnet. Eac ost in te subnet is ten assigned an IP address from te subnet s prefix. Assigning IP prefixes to subnets, and associating subnets wit router interfaces is typically a manual process, as te assignment must follow te addressing ierarcy, yet must reduce wasted namespace, and must consider future use of addresses to minimize later reassignment. Unlike a MAC address, wic functions as a ost identifier, an IP address denotes te ost s current location in te network. Since 3

4 IP routing protocols use a different addressing and patselection mecanism from Eternet, it is impossible to sare routing information across te two protocols. Tis forces te two protocols to be deployed independently, and be connected only at certain fixed nodes called default gateways. Te biggest problem of te ybrid arcitecture is its massive configuration overead. Configuring ybrid networks today represents an enormous callenge. Some estimates put 70% of an enterprise network s operating cost as maintenance and configuration, as opposed to equipment costs or power usage [20]. In addition, involving uman administrators in te loop increases reaction time to faults and increases potential for misconfiguration. Configuration overead due to ierarcical addressing: An IP router cannot function correctly until administrators specify subnets on router interfaces, and direct routing protocols to advertise te subnets. Similarly, an end ost cannot access te network until it is configured wit an IP address corresponding to te subnet were te ost is currently located. DHCP automates end-ost configuration, but introduces substantial configuration overead for managing te DHCP servers. In particular, maintaining consistency between subnet router configuration and DHCP address allocation configuration, or coordinating policies across distributed DHCP servers, are not simple matters. Finally, network administrators must continually revise tis configuration to andle network canges. Complexity in implementing networking policies: Administrators today use a collection of access controls, QoS (Quality of Service) controls [21], and oter policies to control te way packets flow troug teir networks. Tese policies are typically defined based on IP prefixes. However, since prefixes are assigned based on te topology, canges to te network design require tese policies to be rewritten. More significantly, rewriting networking policies must appen immediately after network design canges to prevent reacability problems and to avoid vulnerabilities. Ideally, administrators sould only need to update policy configurations wen te policy itself, not te network, canges. Limited mobility support: Supporting seamless ost mobility is becoming increasingly important. In data centers, migratable virtual macines are being widely deployed to improve power efficiency by adapting to workload, and to minimize service disruption during maintenance operations. Large universities or enterprises often build campus-wide wireless networks, using a wired backaul to support ost mobility across access points. To ensure service continuity and minimize policy update overead, it is igly desirable for a ost to retain its IP address regardless of its location in tese networks. Unfortunately, ybrid networks constrain ost mobility only witin a single, usually small, subnet. In a data center, tis can interfere wit te ability to andle load spikes; in wireless backaul networks, tis can cause service disruptions. One way to deal wit tis is to increase te size of subnets by increasing broadcast domains, wic introduces te scaling problems mentioned in Section Virtual LANs VLANs address some of te problems of Eternet and IP networks. VLANs allow administrators to group multiple osts saring te same networking requirements into a single broadcast domain. Unlike a pysical LAN, a VLAN can be defined logically, regardless of individual osts locations in a network. VLANs can also be overlapped by allowing bridges (not osts) to be configured wit multiple VLANs. By dividing a large bridged network into several appropriately-sized VLANs, administrators can reduce te broadcast overead imposed on osts in eac VLAN, and also ensure isolation among different ost groups. Compared wit IP, VLANs simplify mobility, as osts may retain teir IP addresses wile moving between bridges in te same VLAN. Tis also reduces policy reconfiguration overead. Unfortunately, VLANs introduces several problems: Trunk configuration overead: Extending a VLAN across multiple bridges requires te VLAN to be trunked (provisioned) at eac of te bridges participating in te VLAN. Deciding wic bridges sould be in a given VLAN must take into account traffic and mobility patterns to ensure efficient operation, and ence is often done manually. Limited control-plane scalability: Altoug VLANs reduce te broadcast overead imposed on a particular end ost, bridges provisioned wit multiple VLANs must maintain forwarding-table entries and process broadcast traffic for every active ost in every VLAN visible to temselves. Unfortunately, to enance resource utilization and ost mobility, and to reduce trunk configuration overead, VLANs are often provisioned larger tan necessary, worsening tis problem. A large forwarding table complicates bridge design, since forwarding tables in Eternet bridges are typically implemented using Content-Addressable Memory (CAM), an expensive and power-intensive tecnology. Insufficient data-plane efficiency: Larger enterprises and data centers often ave ricer topologies, for greater reliability and performance. Unfortunately, a single spanning tree is used in eac VLAN to forward packets, wic prevents certain links from being used. Altoug configuring a disjoint spanning tree for eac VLAN [10, 22] may improve load balance and increase aggregate trougput, effective use of per-vlan trees requires periodically moving te roots and rebalancing te trees, wic must be manually updated as traffic sifts. Moreover, inter-vlan traffic must be routed via IP gateways, rater tan sortest pysical pats. 3. Network-Layer One-op DHT Te goal of a conventional Eternet is to route packets to a destination specified by a MAC address. To do tis, Eternet bridges collectively provide end osts wit a service tat maps MAC addresses to pysical locations. Eac bridge implements tis service by maintaining next-op pointers associated wit MAC addresses in its forwarding table, and relies on domain-wide flooding to keep tese pointers up to date. Additionally, Eternet also allows osts to look up te MAC address associated wit a given IP address by broadcasting 4

5 Address Resolution Protocol (ARP) messages. In order to provide te same interfaces to end osts as conventional Eternet, SEATTLE also needs a mecanism tat maintains mappings between MAC/IP addresses and locations. To scale to large networks, SEATTLE operates a distributed directory service built using a one-op, networklevel DHT. We use a one-op DHT to reduce lookup complexity and simplify certain aspects of network administration suc as traffic engineering and troublesooting. We use a network-level approac tat stores mappings at switces, so as to ensure fast and efficient reaction to network failures and recoveries, and avoid te control overead of a separate directory infrastructure. Moreover, our network-level approac allows storage capability to increase naturally wit network size, and exploits cacing to forward data packets directly to te destination witout needing to traverse any intermediate DHT ops [23, 24]. 3.1 Scalable key-value management wit a one-op DHT Our distributed directory as two main parts. First, running a link-state protocol ensures eac switc can observe all oter switces present in te network, and allows any switc to route any oter switc along sortest pats. Second, SEATTLE uses a as function to map ost information to a switc. Tis ost information is maintained in te form of (key,value). Specific examples of tese key-value pairs are (MAC address, location), and (IP address, MAC address) Link-state protocol to maintain switc topology SEATTLE enables sortest-pat forwarding by running a link-state protocol. However, distributing end-ost information in link-state advertisements, as advocated in previous proposals [9, 7, 6, 8], would lead to serious scaling problems in te large networks we consider ere. Instead, SEATTLE s link-state protocol maintains only te switc-level topology, wic is muc more compact and stable. SEATTLE switces use te link-state information to compute sortest pats for unicasting, and multicast trees for broadcasting. To automate configuration of te link-state protocol, SEATTLE switces run a discovery protocol to determine wic of teir links are attaced to osts, and wic are attaced to oter switces. Distinguising between tese different kinds of links is done by sending control messages tat Eternet osts do not respond to. Tis process is similar to ow Eternet distinguises switces from osts wen building its spanning tree. To identify temselves in te link-state protocol, SEATTLE switces determine teir own unique switc IDs witout administrator involvement. For example, eac switc does tis by coosing te MAC address of one of its interfaces as its switc ID Hasing key-value pairs onto switces Instead of disseminating per-ost information in link-state advertisements, SEATTLE switces learn tis information in an on-demand fasion, via a simple asing mecanism. Tis information is stored in te form of (key= k,value= v) pairs. A publiser switc s a wising to publis a (k, v) pair Figure 1: Keys are consistently ased onto resolver switces (s i ). via te directory service uses a as function F to map k to a switc identifier F(k) = r k, and instructs switc r k to store te mapping (k, v). We refer to r k as te resolver for k. A different switc s b may ten look up te value associated wit k by using te same as function to identify wic switc is k s resolver. Tis works because eac switc knows all te oter switces identifiers via link-state advertisements from te routing protocol, and ence F works identically across all switces. Switc s b may ten forward a lookup request to r k to retrieve te value v. Switc s b may optionally cace te result of its lookup, to reduce redundant resolutions. All control messages, including lookup and publis messages, are unicast wit reliable delivery. Reducing control overead wit consistent asing: Wen te set of switces canges due to a network failure or recovery, some keys ave to be re-ased to different resolver switces. To minimize tis re-asing overead, SEATTLE utilizes Consistent Hasing [25] for F. Tis mecanism is illustrated in Figure 1. A consistent asing function maps keys to bins suc tat te cange of te bin set causes minimal curn in te mapping of keys to bins. In SEATTLE, eac switc corresponds a bin, and a ost s information corresponds to a key. Formally, given a set S = {s 1, s 2,..., s n } of switc identifiers, and a key k, F(k) = argmin si S{D(H(k), H(s i ))} were H is a regular as function, and D(x, y) is a simple metric function computing te counter-clockwise distance from x to y on te circular as-space of H. Tis means F maps a key to te switc wit te closest identifier not exceeding tat of te key on te as space of H. As an optimization, a key may be additionally mapped to te next m closest switces along te as ring, to improve resilience to multiple failures. However, in our evaluation, we will assume tis optimization is disabled by default. Balancing load wit virtual switces: Te sceme described so far assumes tat all switces are equally powerful, and ence low-end switces will need to service te same load as more powerful switces. To deal wit tis, we propose a new sceme based on running multiple virtual switces on eac pysical switc. A single switc locally creates one or more virtual switces. Te switc may ten increase or decrease its load by spawning/destroying tese virtual switces. Unlike tecniques used in traditional DHTs for load balancing [24, 26], it is not necessary for our virtual switces to be advertised to oter pysical switces. Instead of advertising every virtual switc in te link-state protocol, switces only advertise te number of virtual switces tey are currently running. Eac switc ten locally com- 5

6 Figure 2: Hierarcical SEATTLE ases keys onto regions. putes virtual switc IDs using te following tecnique. All switces use te same function R(s, i) tat takes as input a switc identifier s and a number i, and outputs a new identifier unique to te inputs. A pysical switc w only advertises in link-state its own pysical switc identifier s w and te number L of virtual switces it is currently running. Every switc can ten determine te virtual identifiers of w by computing R(s w, i) for 1 i L. Note tat it is possible to automate te process of determining a desirable number of virtual switces per pysical switc [26]. Enabling flexible service discovery: Tis design also enables more flexible service discovery mecanisms witout te need to perform network-wide broadcasts. Tis is done by utilizing te as function F to map a string defining te service to a switc. For example, a printer may as te string PRINTER to a switc, at wic it may store its location or address information. Oter switces can ten reac te printer using te as of te string. Services may also encode additional attributes, suc as load or network location, as simple extensions to te as. Specific ways to describe and name services on ave been widely studied in previous work and are out of scope of tis paper [27]. 3.2 Responding to topology canges Te switc-level topology may cange if a new switc/link is added to te network, an existing switc/link fails, or a previously failed switc/link recovers. Tese failures may or may not partition te network into multiple disconnected components. Link failures are typically more common tan switc failures, and partitions are very rare if te network as sufficient redundancy. In te case of a link failure/recovery tat does not partition a network, te set of switces appearing in te link-state map does not cange. Since te as function F is defined wit te set of switces in te network, te resolver a particular key maps to will not cange. Hence all tat needs to be done is to update te link-state map to ensure packets continue to traverse new sortest pats. In SEATTLE, tis is simply andled by te link-state routing protocol. However, if a switc fails or recovers, te set of switces in te link-state map canges. Hence tere may be some keys k wose old resolver rk old differs from a new resolver rk new. To deal wit tis, te value (k, v) must be moved from rk old to rk new. Tis is andled by aving te switc s k tat originally publised k monitor te liveness of k s resolver troug link-state advertisements, and republising (k, v) to rk new wen rk new differs from rk old. Te value (k, v) is eventually removed from rk old after a timeout. Additionally, wen a value v denotes a location, suc as a switc id s, and s goes down, eac switc scans te list of locally-stored (k, v) pairs, and remove all entries wose value v equals s. Note tis procedure correctly andles network partitions because te link-state protocol ensures tat eac switc will be able to see only switces present in its partition. 3.3 Supporting ierarcy wit a multi-level, one-op DHT Te SEATTLE design presented so far scales to large, dynamic networks [28]. However, since tis design runs a single, network-wide link-state routing protocol, it may be inappropriate for networks wit igly dynamic infrastructure, suc as networks in developing regions [3]. A single network-wide protocol may also be inappropriate if network operators wis to provide stronger fault isolation across geograpic regions, or to divide up administrative control across smaller routing domains. Moreover, wen a SEATTLE network is deployed over a wide area, te resolver could lie far bot from te source and destination. Forwarding lookups over long distances increases latency and makes te lookup more prone to failure. To deal wit tis, SEATTLE may be configured ierarcically, by leveraging a multi-level, oneop DHT. Tis mecanism is illustrated in Figure 2. A ierarcical network is divided into several regions, and a backbone providing connectivity across regions. Eac region is connected to te backbone via its own border switc, and te backbone is composed of te border switces of all regions. Every switc in a region knows te identifier of te region s border switc, because te border switc advertises its role troug te link-state protocol. In suc an environment, SEATTLE ensures tat only inter-region lookups are forwarded via te backbone wile all regional lookups are andled witin teir own regions, and link-state advertisements are only propagated locally witin regions. SEATTLE ensures tis by defining a separate regional and backbone as ring. Wen a (k, v) is inserted into a region P and is publised to a regional resolver rk P (i.e., a resolver for k in region P ), rk P additionally forwards (k, v) to one of te region P s border switces b P. Ten b P ases k again onto te backbone ring and publises (k, v) to anoter backbone switc b Q k, wic is a backbone resolver for k and a border switc of region Q at te same time. Switc b Q k stores k s information. If a switc in region R wises to lookup (k, v), it forwards te lookup first to its local resolver rk R, wic in turn forwards it to b R, and b R forwards it to b Q k. As an optimization to reduce load on border switces, b Q k may as k and store (k, v) at a switc witin its own region Q, rater tan storing (k, v) locally. Since switc failures are not propagated across regions, eac publiser switc periodically sends probes to backbone resolvers tat lie outside of its region. To improve availability, (k, v) may be stored at multiple backbone resolvers (as described in Section 3.1.2), and multiple simultaneous lookups may be sent in parallel. 4. Scaling Eternet wit a One-op DHT Te previous section described te design of a distributed 6

7 Unlike Eternet bridging, cace misses in SEATTLE do not lead to flooding, making te network resistant to cace poisoning attacks (e.g., forwarding table overflow attack) or a sudden sift in traffic. Moreover, tose switces tat are not directly connected to end osts (i.e., aggregation or core switces) do not need to maintain any caced entries. Figure 3: Packet forwarding and lookup in SEATTLE. network-level directory service based on a one-op DHT. In tis section, we describe ow te directory service is used to provide efficient packet delivery and scalable address resolution. We first briefly describe ow to forward data packets to MAC addresses in Section 4.1. We ten describe our remaining contributions: an optimization tat eliminate te need to look up ost location in te DHT by piggy-backing tat information on ARP requests in Section 4.2, and a scalable dynamic cace-update protocol in Section Host location resolution Hosts use te directory service described in Section 3 to publis and maintain mappings between teir MAC addresses and teir current locations. Tese mappings are used to forward data packets, using te procedure sown in Figure 3. Wen a ost a wit MAC address mac a first arrives at its access switc s a, te switc must publis a s MAC-tolocation mapping in te directory service. Switc s a does tis by computing F(mac a ) = r a, and instructing r a to store (mac a, s a ). We refer to r a as te location resolver for a. Ten, if some ost b connected to switc s b wants to send a data packet to mac a, b forwards te data packet to s b, wic in turn computes F(mac a ) = r a. Switc s b ten and forwards te packet to r a. Since r a may be several ops away, s b encapsulates te packet wit an outer eader wit r a s address as te destination. Switc r a ten looks up a s location s a, and forwards te packet on towards s a. In order to limit te number of data packets traversing te resolver, r a also notifies s b tat a s current location is s a. Switc s b ten caces tis information. Wile forwarding te first few packets of a flow via a resolver switc increases pat lengts, in te next section we describe an optimization tat allows data packets to traverse only sortest pats, by piggy-backing location information on ARP replies. Note SEATTLE manages per-ost information via reactive resolution, as opposed to te proactive dissemination sceme used in previous approaces [9, 7, 6]. Te scaling benefits of tis reactive resolution increase in enterprise/data-center/access provider networks because most osts communicate wit a small number of popular osts, suc as mail/file/web servers, printers, VoIP gateways, and Internet gateways [5]. To prevent forwarding tables from growing unnecessarily large, te access switces can apply various cace-management policies. For correctness, owever, te cace-management sceme must not evict te ost information of te osts tat are directly connected to te switc or are registered wit te switc for resolution. 4.2 Host address resolution In conventional Eternet, a ost wit an IP packet first broadcasts an ARP request to look up te MAC address of te ost owning te destination IP address contained in te request. To enance scalability, SEATTLE avoids broadcastbased ARP operations. In addition, we extend ARP to return bot te location and te MAC address of te end ost to te requesting switc. Tis allows data packets following an ARP query to directly traverse sortest pats. SEATTLE replaces te traditional broadcast-based ARP wit an extension to te one-op DHT directory service. In particular, switces use F wit an IP address as te key. Specifically, wen ost a arrives at access switc s a, te switc learns a s IP address ip a (using tecniques described in Section 5.1), and computes F(ip a ) = v a. Te result of tis computation is te identifier of anoter switc v a. Finally, s a informs v a of (ip a, mac a ). Switc v a, te address resolver for ost a, ten uses te tuple to andle future ARP requests for ip a redirected by oter remote switces. Note tat ost a s location resolver (i.e., F(mac a )) may differ from a s address resolver (i.e., F(ip a )). Optimizing forwarding pats via ARP: For osts tat issue an ARP request, SEATTLE eliminates te need to perform forwarding via te location resolver as mentioned in Section 4.1. Tis is done by aving te address resolver switc v a also maintain te location of a (i.e., s a ) in addition to mac a. Upon receiving an ARP request from some ost b, te address resolver v a returns bot mac a and s a back to b s access switc s b. Switc s b ten caces s a for future packet delivery, and returns mac a to ost b. Any packets sent by b to a are ten sent directly along te sortest pat to a. It is, owever, possible tat ost b already as mac a in its ARP cace and immediately sends data frames destined to mac a witout issuing an ARP request in advance. Even in suc a case, as long as te s b also maintains a s location associated wit mac a, s b can forward tose frames correctly. To ensure access switces cace te same entries as osts, te timeout value tat an access switc applies to te caced location information must be larger tan te ARP cace timeout used by end osts 2. Note tat, even if te cace and te ost become out of sync (due to switc reboot, etc.), SEATTLE continues to operate correctly because switces can resolve a ost s location by asing te ost s MAC address to te ost s location resolver. 4.3 Handling ost dynamics Hosts can undergo tree different kinds of canges in a SEATTLE network. First, a ost may cange location, for 2 Te default setting of te ARP cace timeout in most common operating systems ranges 10 to 20 minutes. 7

8 example if it as pysically moved to a new location (e.g., wireless andoff), if its link as been plugged into a different access switc, or if it is a virtual macine and as migrated to a new osting system tat allows te VM to retain its MAC address. Second, a ost may cange its MAC address, for example if its NIC card is replaced, if it is a VM and as migrated to a new osting system tat requires te VM to use te ost s MAC address, or if multiple pysical macines collectively acting as a single server or router (to ensure ig availability) experience a fail-over event [29]. Tird, a ost may cange its IP address, for example if a DHCP lease expires, or if te ost is manually reconfigured. In practice, multiple of tese canges may occur simultaneously. Wen tese canges occur, we need to keep te directory service up-to-date, to ensure correct packet delivery. SEATTLE andles tese canges by modifying te contents of te directory service via insert, delete, and update operations. An insert operation adds a new (k, v) pair to te DHT, a delete operation removes a (k, v) pair from te DHT, and te update operation updates te value v associated wit a given key k. First, in te case of a location cange, te ost moves from one access switc s old to anoter snew. In tis case, s new inserts a new MAC-to-location entry. Since s MAC address already exists in te DHT, tis action will update s old location wit its new location. Second, in te case of a MAC address cange, s access switc s inserts an IP-to-MAC entry containing s new MAC address, causing s old IP-to-MAC mapping to be updated. Since a MAC address is also used as a key of a MAC-to-location mapping, s deletes s old MAC-to-location mapping and inserts a new mapping, respectively wit te old and new MAC addresses as keys. Tird, in te case of an IP address cange, we need to ensure tat future ARP requests for s old IP address are no longer resolved to s MAC address. To ensure tis, s deletes s old IP-to-MAC mapping and insert te new one. Finally, if multiple canges appen at once, te above steps occur simultaneously. Ensuring seamless mobility: As an example, consider te case of a mobile ost moving between two access switces, s old and s new. To andle tis, we need to update s MAC-to-location mapping to point to its new location. As described in Section 4.1, s new inserts (mac, s new ) into r upon arrival of. Note tat te location resolver r selected by F(mac ) does not cange wen s location canges. Meanwile, s old deletes (mac, s old ) wen it detects is unreacable (eiter via timeout or active polling). Additionally, to enable prompt removal of stale information, te location resolver r informs s old tat (mac, s old ) is obsoleted by (mac, s new ). However, ost locations caced at oter access switces must be kept up-to-date as osts move. SEATTLE takes advantage of te fact tat, even after updating te information at r, s old may receive packets destined to because oter access switces in te network migt ave te stale information in teir forwarding tables. Hence, wen s old receives packets destined to, it explicitly notifies ingress switces tat sent te misdelivered packets of s new location s new. To minimize service disruption, s old also forwards tose misdelivered packets s new. Updating remote osts caces: In addition to updating contents of te directory service, some ost canges require informing oter osts in te system about te cange. For example, if a ost canges its MAC address, te new mapping (ip, mac new ) must be immediately known to oter osts wo appened to store (ip, mac old ) in teir local ARP caces. In conventional Eternet, tis is acieved by broadcasting a gratuitous ARP request originated by [30]. A gratuitous ARP is an ARP request containing te MAC and IP address of te ost sending it. Tis request is not a query for a reply, but is instead a notification to update oter end osts ARP tables and to detect IP address conflicts on te subnet. Relying on broadcast to update oter osts clearly does not scale to large networks. SEATTLE avoids tis problem by unicasting gratuitous ARP packets only to osts wit invalid mappings. Tis is done by aving s maintain a MAC revocation list. Upon detecting s MAC address cange, switc s inserts (ip, mac old, macnew ) in its revocation list. From ten on, wenever s receives a packet wose source or destination (IP, M AC) address pair equals (ip, mac old ), it sends a unicast gratuitous ARP request containing (ip, mac new ) to te source ost wic sent tose packets. Note tat, wen bot s MAC address and location cange at te same time, te revocation information is created at s old access switc by s address resolver v = F(ip ). To minimize service disruption, s also delivers te misaddressed packets to by rewriting te destination to mac new. Additionally, s also informs te source ost s ingress switc of (mac new, s ) so tat te packets destined to mac new can ten be directly delivered to s, avoiding an additional location lookup. Note tis approac to updating remote ARP caces does not require s to look up eac packet s IP and MAC address pair from te revocation list because s can skip te lookup in te common case (i.e., wen its revocation list is empty). Entries from te revocation list are removed after a timeout set equal to te ARP cace timeout of end osts. 5. Providing Eternet-like Semantics To be fully backwards-compatible wit conventional Eternet, SEATTLE must act like a conventional Eternet from te perspective of end osts. First, te way tat osts interact wit te network to bootstrap temselves (e.g., acquire addresses, allow switces to discover teir presence) must be te same as Eternet. Second, switces ave to support traffic tat uses broadcast/multicast Eternet addresses as destinations. In tis section, we describe ow to perform tese actions witout incurring te scalability callenges of traditional Eternet. For example, we propose to eliminate broadcasting from te two most popular sources of broadcast traffic: ARP and DHCP. Since we described ow SEATTLE switces andle ARP witout broadcasting in Section 4.2, we discuss only DHCP in tis section. 8

9 5.1 Bootstrapping osts Host discovery by access switces: Wen an end ost arrives at a SEATTLE network, its access switc needs to discover te ost s MAC and IP addresses. To discover a new ost s MAC address, SEATTLE switces use te same MAC learning mecanism as conventional Eternet, except tat MAC learning is enabled only on te ports connected to end osts. To learn a new ost s IP address or detect an existing ost s IP address cange, SEATTLE switces snoop on gratuitous ARP requests. Most operating systems generate a gratuitous ARP request wen te ost boots up, te ost s network interface or links comes up, or an address assigned to te interface canges [30]. If a ost does not generate a gratuitous ARP, te switc can still learn of te ost s IP address via snooping on DHCP messages, or sending out an ARP request only on te port connected to te ost. Similarly, wen an end ost fails or disconnects from te network, te access switc is responsible for detecting tat te ost as left, and deleting te ost s information from te network. Host configuration witout broadcasting: For scalability, SEATTLE resolves DHCP messages witout broadcasting. Wen an access switc receives a broadcast DHCP discovery message from an end ost, te switc delivers te message directly to a DHCP server via unicast, instead of broadcasting it network-wide. SEATTLE implements tis mecanism using te existing DHCP relay agent standard [31]. Tis standard is used wen an end ost needs to communicate wit a DHCP server outside te ost s broadcast domain. Te standard proposes tat a ost s IP gateway forward a DHCP discovery to a DHCP server via IP routing. In SEATTLE, a ost s access switc can perform te same function wit Eternet encapsulation. Access switces can discover a DHCP server using te service discovery mecanism described in Section For example, te DHCP server ases te string DHCP SERVER to a switc, and ten stores its location at tat switc. Oter switces ten forward DHCP requests using te as of te string. 5.2 Scalable and flexible VLANs SEATTLE completely eliminates flooding of unicast packets. However, to offer te same semantics as Eternet bridging, SEATTLE needs to support transmission of packets sent to a broadcast address. Supporting broadcasting is important because some applications (e.g., IP multicast, peer-to-peer file saring programs, etc.) rely on subnetwide broadcasting. However, in large networks to wic our design is targeted, performing broadcasts in te same style as Eternet may significantly overload switces and reduce data plane efficiency. Instead, SEATTLE provides a mecanism wic is similar to, but more flexible tan, VLANs. In particular, SEATTLE introduces a notion of group. Similar to a VLAN, a group is defined as a set of osts wo sare te same broadcast domain regardless of teir location. Unlike Eternet bridging, owever, a broadcast domain in SEATTLE does not limit unicast layer-2 reacability between osts because a SEATTLE switc can resolve any ost s address or location witout relying on broadcasting. Tus, groups provide several additional benefits over VLANs. First, groups do not need to be manually assigned to switces. A group is automatically extended to cover a switc as soon as a member of tat group arrives at te switc 3. Second, a group is not forced to correspond to a single IP subnet, and ence may span multiple subnets or a portion of a subnet, if desired. Tird, unicast reacability in layer-2 between two different groups may be allowed (or restricted) depending on te access-control policy a rule set defining wic groups can communicate wit wic between te groups. Te flexibility of groups can ensure several benefits tat are quite ard to acieve wit conventional Eternet bridging and VLANs. For example, wen a group is aligned wit a subnet, and unicast reacability between two different groups is not permitted by default, groups provide exactly te same functionality as VLANs. However, groups can include a large number of end osts and can be extended to anywere in te network witout arming controlplane scalability and data-plane efficiency. Moreover, wen groups are defined as subsets of an IP subnet, and intergroup reacability is proibited, eac group is equivalent to a private VLAN (PVLAN), wic are popularly used in otel/motel networks [33]. Unlike PVLANs, owever, groups can be extended over multiple bridges. Finally, wen unicast reacability between two groups is permitted, traffic between te groups takes te sortest pat, witout needing to traverse an IP gateway. Multicast-based group-wide broadcasting: All broadcast packets witin a group are delivered troug a multicast tree sourced at a dedicated switc, namely a broadcast root, of te group. Te mapping between a group and its broadcast root is determined by using F to as te group s identifier to a switc. Construction of te multicast tree is done in a manner similar to IP multicast, ineriting its safety (i.e., loop freedom) and efficiency (i.e., to receive broadcast only wen necessary). Wen a switc first detects an end ost tat is a member of group g, te switc issues a join message tat is carried up to te nearest graft point on te tree toward g s broadcast root. Wen a ost departs, its access switc prunes a branc if necessary. Finally, wen an end ost in g sends a broadcast packet, its access switc marks te packet wit g and forwards it along g s multicast tree. Separating unicast reacability from broadcast domains: In addition to andling broadcast traffic, groups in SEAT- TLE also provide a namespace upon wic reacability policies for unicast traffic are defined. Wen a ost arrives at an access switc, te ost s group membersip is determined by its access switc and publised to te ost s resolvers along wit its location information. Access control policies are ten applied by a resolver wen a ost attempts to look up a destination ost s information. 3 Te way administrators associate a ost wit corresponding group is beyond te scope of tis paper. For Eternet, management systems tat can automate tis task (e.g., mapping an end ost or flow to a VLAN) are already available [17, 32], and SEATTLE can employ te same model. 9