Simple Efficient Load Balancing Algorithms for Peer-to-Peer Systems

Transcription

1 Simple Efficiet Load Balacig Algorithms for Peer-to-Peer Systems David R. Karger MIT Computer Sciece ad Artificial Itelligece Laboratory Cambridge, MA 0139, USA Matthias Ruhl IBM Almade Research Ceter Sa Jose, CA 9510, USA ABSTRACT Load balacig is a critical issue for the efficiet operatio of peerto-peer etworks. We give two ew load-balacig protocols whose provable performace guaratees are withi a costat factor of optimal. Our protocols refie the cosistet hashig data structure that uderlies the Chord ad Koorde PP etwork. Both preserve Chord s logarithmic query time ad ear-optimal data migratio cost. Cosistet hashig is a istace of the distributed hash table DHT paradigm for assigig items to odes i a peer-to-peer system: items ad odes are mapped to a commo address space, ad odes have to store all items residig closeby i the address space. Our first protocol balaces the distributio of the key address space to odes, which yields a load-balaced system whe the DHT maps items radomly ito the address space. To our kowledge, this yields the first PP scheme simultaeously achievig Olog degree, Olog look-up cost, ad costat-factor load balace previous schemes settled for ay two of the three. Our secod protocol aims to directly balace the distributio of items amog the odes. This is useful whe the distributio of items i the address space caot be radomized. We give a simple protocol that balaces load by movig odes to arbitrary locatios where they are eeded. As a applicatio, we use the last protocol to give a optimal implemetatio of a distributed data structure for rage searches o ordered data. Categories ad Subject Descriptors C..4 [Distributed Systems]: Distributed applicatios load balacig; F.. [Aalysis of Algorithms ad Problem Complexity]: Noumerical Algorithms ad Problems Geeral Terms Algorithms, Theory Keywords Peer-to-peer systems, load balacig Permissio to make digital or hard copies of all or part of this work for persoal or classroom use is grated without fee provided that copies are ot made or distributed for profit or commercial advatage ad that copies bear this otice ad the full citatio o the first page. To copy otherwise, to republish, to post o servers or to redistribute to lists, requires prior specific permissio ad/or a fee. SPAA 04, Jue 7 30, 004, Barceloa, Spai. Copyright 004 ACM /04/ $ INTRODUCTION Peer-to-peer PP systems are a curret research focus i computer systems ad etworkig. Such systems are attractive i their potetial to haress the vast distributed computatio ad storage resources i today s etworks, without eed for complex ad sesitive cetralized cotrol. A core problem i peer-to-peer systems is the distributio of items to be stored or computatios to be carried out to the odes that make up the system. A particular paradigm for such allocatio, kow as the distributed hash table DHT, has become the stadard approach to this problem i research o peer-to-peer systems [7, 9, 1, 13, 16, 17, 1]. A distributed hash table iterface implemets a hash fuctio that maps ay give item to a particular machie bucket i the peer-to-peer etwork. For example, Chord [1] uses Cosistet Hashig [8] to assig items to odes: items ad odes are pseudo-radomly hashed to a circular address space represeted by the iterval [0,1] with addresses 0 ad 1 idetified, ad a ode stores all items whose addresses fall betwee the ode s ow address ad the address of the ode preceedig it i the address space. DHTs differ from traditioal hash tables i two key ways: First, i additio to the isertio ad deletio of items, DHTs must support the isertio ad deletio of buckets: as machies joi ad leave the etwork, items must be migrated to other machies ad the hash fuctio revised to reflect their ew locatio. Secod, some kid of routig protocol is usually ecessary: sice it is ot feasible i a PP system for every ode to maitai up-to-date kowledge of all other odes i the system, a item is looked up or iserted by followig a sequece of routig hops through the peer-to-peer etwork. A large umber of algorithms have bee proposed ad implemeted i systems [16, 1, 9, 17, 13, 7] to provide distributed hash table fuctioality. The majority of them require each ode of a -ode PP system to keep track of oly Olog eighbor odes ad allow the machie resposible for ay key to be looked up ad cotacted i Olog routig hops. Recetly some variats have bee proposed [1, 7] that support Olog-hop lookups with oly costat eighbor degree; this is theoretically optimal but may be udesirable i practice i light of fault-tolerace cosideratios. 1.1 Load Balacig A importat issue i DHTs is load-balace the eve distributio of items or other load measures to odes i the DHT. All DHTs make some effort to load-balace, geerally by i radomizig the DHT address associated with each item with a good eough hash fuctio ad ii makig each DHT ode resposible for a balaced portio of the DHT address space. Chord is

2 a prototypical example of this approach: its radom hashig of odes to a rig meas that each ode is resposible for oly a small iterval of rig address space, while the radom mappig of items meas that oly a limited umber of items lad i the small rig iterval owed by ay ode. This attempt to load-balace ca fail i two ways. First, the typical radom partitio of the address space to odes is ot completely balaced. Some odes ed up resposible for a larger portio of the addresses ad thus receive a larger portio of the radomly distributed items. Secod, some applicatios may preclude the radomizatio of data items addresses. For example, to support rage searchig i a database applicatio the items may eed to be placed i a specific order, or eve at specific addresses, o the rig. I such cases, we may fid the items uevely distributed i address space, meaig that balacig the address space amog odes does ot balace the distributio of items amog odes. We give protocols to solve both of the load balacig challeges just described Address-Space Balacig I geeral, distributed hash tables do ot offer load balace quite as good as stadard hash tables. A typical stadard hash table partitios the space of possible hash-fuctio values evely over the buckets; thus, assumig the hash fuctio is radom eough ad sufficietly may keys are iserted, those keys will be evely distributed amog the buckets. Curret distributed hash tables do ot evely partitio the address space ito which keys get mapped; some machies get a larger portio of it. Thus, eve if keys are umerous ad radom, some machies receive more tha their fair share, by as much as a factor of Olog times the average. To cope with this problem, most DHTs use virtual odes: each real machie preteds to be several distict machies, each participatig idepedetly i the DHT protocol. The machie s load is thus determied by summig over several virtual odes, creatig a tight cocetratio of total load ear the average. As a example, the Chord DHT is based upo cosistet hashig [8], which requires Olog virtual copies to be operated for every ode. Virtual odes have drawbacks. Most obviously, the real machie must allocate space for the data structures of each virtual ode; more virtual odes mea more data structure space. However, PP data structures are typically ot that space-expesive requirig oly logarithmic space per ode so multiplyig that space requiremet by a logarithmic factor is ot particularly problematic. A much more sigificat problem arises from etwork badwidth. I geeral, to maitai coectivity of the etwork, every virtual ode must frequetly pig its eighbors, make sure they are still alive, ad replace them with ew eighbors if ot. Ruig multiple virtual odes creates a multiplicative icrease i the very valuable etwork badwidth cosumed for maiteace. Below, we will solve this problem by arragig for each ode to activate oly oe of its Olog virtual odes at ay give time. The ode will occasioally check its iactive virtual odes, ad may migrate to oe of them if the distributio of load i the system has chaged. Sice oly oe virtual ode is active, the real ode eed ot pay the origial Chord protocol s multiplicative icrease i space ad badwidth costs. As i the origial Chord protocol, our scheme gives each real ode oly a small umber of legitimate addresses o the Chord rig, preservig Chord s limited protectio agaist address spoofig by malicious odes tryig to disrupt the routig layer. If each ode could choose a arbitrary address, the a malicious ode aimig to erase a certai item could take resposibility for that item s key ad the refuse to serve the item Item Balacig A secod load-balacig problem arises from certai database applicatios. A hash table radomizes the order of keys. This is problematic i domais for which order matters for example, if oe wishes to perform rage searches over the data. This is oe of the reasos biary trees are useful despite the faster lookup performace of hash tables. A order-preservig dictioary structure caot apply a radomized ad therefore load balacig hash fuctio to its keys; it must take them as they are. Thus, eve if the address space is evely distributed amog the odes, a ueve distributio of the keys e.g., all keys ear 0 may lead to all load beig placed o oe machie. I our work, we develop a load balacig solutio for this problem. Ufortuately, the limited assigmets approach discussed for key-space load balacig does ot work i this case it is easy to prove that if odes ca oly choose from a few addresses, the certai load balacig tasks are beyod them. Our solutio to this problem therefore allows odes to move to arbitrary addresses; with this freedom we show that we ca load-balace a arbitrary distributio of items, without expedig much cost i maitaiig the load balace. Our scheme works through a kid of work stealig i which uderloaded odes migrate to portios of the address space occupied by too may items. The protocol is simple ad practical, with all the complexity i its performace aalysis. Our protocol ca also be used to balace weighted items, where the weight of a item ca for example reflect its storage size, or its popularity ad the resultig badwidth requiremets. 1. Our Cotributios I this paper we give two distributed load-balacig schemes for data storage applicatios i PP etworks. First, i Sectio, we give a protocol that improves cosistet hashig i that every ode is resposible for a O1/ fractio of the address space with high probability, without usig virtual odes. The protocol is dyamic, with a isertio or deletio causig Ologlog other odes to chage their positios. Each ode has a fixed set of Olog possible positios that it chooses from. This set oly depeds o the ode itself computed e.g. as hashes of the ode IP address, impedig malicious spoofig attacks o the etwork. Aother ice property of this protocol is that the appropriate state of the system i.e., which virtual odes are active, although radom, is idepedet of the history of item ad ode arrivals ad departures. This Markovia property meas that the system ca be aalyzed as if it were static, with a fixed set of odes ad items; such aalysis is geerally much simpler tha a dyamic, history-depedet aalysis. Combiig our load-balacig scheme with the Koorde routig protocol [7], we obtai a protocol that simultaeously offers i Olog degree per real ode, ii Olog/ loglog lookup hops, ad iii costat factor load balace. Previous protocols could achieve ay two of these but ot all three geerally speakig, achievig property iii required operatig Olog virtual odes, which pushed the degree to Olog ad therefore failed to achieve property i. A secod iterpretatio of our results ca be give idepedet of PP systems. Cosistet hash fuctios [8] are useful geeralized hash fuctios assigig items to buckets i a dyamic fashio that allows both items ad buckets to be iserted ad deleted dyamically. The iitial implemetatio of cosistet hashig, however, required Olog + N space to store N items i buckets. Our ew scheme reduces the ecessary space allocatio to the optimal O + N space, at the cost of slower bucket isertios ad

3 deletios the isertio or deletio of a ode causes Ologlog buckets to chage i expectatio, compared to O1 buckets i the worst case for the origial versio of cosistet hashig. It is a iterestig ope problem to optimize space without affectig bucket isertio ad deletio time. I the secod part of our work, we cosider arbitrary distributios of keys, which forces us to allow odes to move to arbitrary addresses. I Sectio 3, we give a dyamic protocol that chages odes addresses i order to achieve load balace. The protocol is radomized, ad relies o the uderlyig PP routig framework to be able to cotact radom odes i the system. We show that the amortized rebalacig costs i terms of umber of items moved are ON/ for a ode isertio or deletio where N is the umber of items i the system, ad O1 for the isertio or deletio of a item. The protocol does ot require the kowledge of N or for operatio, ad it ca be exteded to items with differet weights or storage costs. I particular, this load balacig protocol ca be used to store ordered data, such that the items are ot hashed, but mapped to the DHT address space i a order-preservig way. Every ode the stores the items fallig ito a cotiuous segmet of that orderig. I Sectio 3..4, we describe how this ca be used to implemet a rage-search data structure, where give items a ad b, the data structure is to retur all items x stored i the system that satisfy a x b. We give the first such protocol that achieves a Olog+ K/N query time where K is the size of the output. We desig our solutios i the cotext of the Chord ad Koorde DHT [1] but our ideas seem applicable to a broader rage of DHT solutios. Chord [1] uses Cosistet Hashig to assig items to odes, achievig key-space load balace usig Olog virtual odes per real ode. O top of Cosistet Hashig, Chord layers a routig protocol i which each ode maitais a set of Olog carefully chose eighbors that it uses to route lookups i Olog hops. Our modificatios of Chord are essetially modificatios of the Cosistet Hashig protocol assigig items to odes; we ca iherit uchaged Chord s eighbor structure ad routig protocol. Thus, for the remaider of this paper, we igore issues of routig ad focus o the assigmet problem. 1.3 Related Work While much research has bee doe o routig i PP etworks, work o efficiet load balacig ad complex queries i PP is oly i its begiig stages. Most structured PP systems simply assume that items are uiformly distributed o odes. Two protocols that achieve ear-optimal load-balacig without the use of virtual odes have recetly bee give [1, 14]. Our scheme improves upo them i three respects. First, i those protocols the address assiged to a ode depeds o the rest of the etwork, i.e. the address is ot selected from a list of possible addresses that oly deped o the ode itself. This makes the protocols more vulerable to malicious attacks. Secod, i those protocols the address assigmets deped o the costructio history, makig them harder to aalyze. Third, their load-balacig guaratees are oly show for the isertios oly case, while we also hadle deletios of odes ad items. Work o load balacig by movig items ca be foud i work of Rao et al. [15]. Their algorithm is similar to ours, however it oly works whe the set of odes ad items are fixed i.e. without isertios or deletios, ad they give o provable performace guaratees, oly experimetal evaluatios. A theoretical aalysis of a similar protocol was give by Aagostopoulos, Kirsch ad Upfal []. I their settig, however, items are assumed to be jobs that are executed at a fixed rate, i.e. items disappear from odes at a fixed rate. Moreover, they aalyze the average wait time for jobs, while we are more iterested i the total umber of items moved to achieve load balace. Complex queries such as rage searches are also a emergig research topic for PP systems [5, 6]. A efficiet rage search data structure was recetly give by Aspes ad Shah [3]. However, that work does ot address the issue of load balacig the umber of items per ode, makig the simplifyig assumptio that each ode stores oly oe item. I this settig, the lookup times are OlogN i terms of the umber of items N, ad ot i terms of the umber of odes. Also, OlogN storage is used per data item, meaig a total storage of ON logn, which is typically much worse tha ON + log. I recet idepedet work, Gaesa ad Bawa [4] cosider a load balacig scheme similar to ours ad poit out applicatios to rage searches. However, their scheme relies o beig able to quickly fid the least ad most loaded odes i the system. It is ot clear how to support this operatio efficietly, ad without creatig heavy etwork traffic for these odes with extreme load. 1.4 Notatio I this paper, we will use the followig otatio. is the umber of odes i system N is the umber of items stored i system usually N l i is the umber of items stored at ode i L = N/ is the average desired load i the system Wheever we talk about the address space of a PP routig protocol such as Chord, we assume that this space is ormalized to the iterval [0,1]. We further assume that the addresses 0 ad 1 are idetified, i.e. that the address space forms a rig.. ADDRESS-SPACE BALANCING I this sectio we give a protocol that improves cosistet hashig i that every ode is resposible for a O1/ fractio of the address space with high probability whp, without use of virtual odes. This improves space ad badwidth usage i Chord by a logarithmic factor over traditioal cosistet hashig. The protocol is dyamic, with a isertio or deletio causig Ologlog other odes to chage their positios. Each ode has a fixed set of Olog possible positios called potetial odes ; it chooses exactly oe of those potetial odes to become active at ay time this is the oly ode that it actually operates. A ode s set of potetial odes depeds oly o the ode itself their addresses computed e.g. as hashes hi,1,hi,,...,hi,clog of the ode idetifier i, makig attacks o the etwork more difficult. We deote the address b + 1 a by a,b, where a ad b are itegers satisfyig 0 a ad 0 b < a 1. This yields a uambiguous otatio for all addresses with fiite biary represetatio. We impose a orderig o these addresses accordig to the legth of their biary represetatio breakig ties by magitude of the address. More formally, we set a,b a,b iff a < a or a = a ad b < b. This yields the followig orderig: 0 = We are ow goig to describe our protocol i terms of the ideal state it wats to achieve. Ideal state: Give ay set of active odes, each possibly iactive potetial ode spas a certai rage of addresses betwee itself ad the succeedig active ode o the address rig. Each real ode has activated the potetial ode that spas the miimal uder the orderig address.

4 Our protocol cosists of the simple update rule that ay ode for which the ideal state coditio is ot satisfied, istead activates the potetial ode for which the coditio is locally satisfied. I other words, each ode occasioally determies which of its Olog potetial odes spas the smallest address accordig to. The ode the activates that particular potetial ode. We will ow prove that such a system has a uique ideal state. This shows that the protocol is Markovia, i.e. that the resultig state does ot deped o the costructio history, ad there is o bias itroduced by a sequece of isertios ad deletios. This is similar to treaps [0] where items are iserted with radom priorities, yet the result does ot deped o the order of the isertios. Theorem 1 The followig statemets are true for the above protocol, if every ode has clog potetial addresses that are chose Ωlog-idepedetly at radom. i For ay set of odes there is a uique ideal state. ii Give ay startig state, the local improvemets will evetually lead to this ideal state. iii I the ideal state of a etwork of odes, whp all eighborig pairs of active odes will be at most 4 + ε/ apart, for ay ε 1/ with c 1/ε. This boud improves to + ε/ for very small ε. iv Upo isertig or deletig a ode ito a ideal state, i expectatio at most Ologlog odes have to chage their addresses for the system to agai reach the ideal state..1 Proof of Theorem 1 The uique ideal state ca be costructed as follows. The potetial ode immediately precedig address 1 will be active, sice its real-ode ower has o better choice ad caot be blocked by ay other ode from spaig address 1. That real ode s other potetial odes will the be out of the ruig for activatio. Of the remaiig potetial odes, the oe closest to address 1/ will become active for the same reaso, ad so o. We cotiue i this way dow the -ordered list of addresses. This greedy process clearly defies the uique ideal state, showig claim i. We will show claim ii by arguig that every local improvemet reduces the distace from the curret state to the ideal state with distace appropriately defied. For this, let A be the set of addresses with fiite biary expasio, i.e. the addresses i the orderig. Fix the set X of active odes i the system. We defie a fuctio f X : A R as follows. For each address a A, cosider the active ode x that spas the address iterval cotaiig a. If that iterval does ot cotai ay smaller i the sese of address tha a, the let f X a be the address distace betwee x ad a, otherwise let f X a :=. Two differet sets X ad Y of active odes will lead to differet fuctios f X ad f Y. Cosider the lexicographic orderig o all of these fuctios, i.e. fuctios are compared by first cosiderig their relative value at 0, the at 1, 4 1, 3 4, ad so o i the order of, util the first uequal value determies the relative orderig of the fuctios. It is the straightforward to show cf. [18, Lemma 4.5] that a amog all choices of active odes, the ideal state leads to the smallest possible fuctio f X uder this lexicographic orderig, ad b every local improvemet makes the fuctio f X become smaller i.e. reduces the distace to the ideal state. Combied with the facts that there is oly oe ideal state, i.e. oly oe state i which o local improvemet is possible claim i, ad that there is oly a fiite umber of potetial-ode choices for active odes, this shows claim ii. To prove the remaiig two claims, we will assume for simplicity that the potetial odes addresses are chose idepedetly at radom. But just as with the origial cosistet hashig scheme [10], our results cotiue to hold for Ωlog-wise idepedet choices of potetial addresses. This follows by a stadard applicatio of results of Cheroff [19]. I our proof, we will use a Cheroff boud to show that each address iterval of them form [a ε/,a] will whp cotai at least oe potetial ode. This ca be rephrased as a stadard balls-ad-bis experimet, to which the results of Cheroff [19] directly apply. To prove claim iii, recall how we costructed the ideal state for claim i above by successively assigig odes to icreasig addresses i the order. I this process, suppose we are cosiderig oe of the first 1 ε addresses i. Cosider the iterval I of legth ε/ precedig this address i the address space. Sice this is oe of the first 1 ε first addresses, at least ε of the real odes have ot yet bee give a place o the rig. Amog the cεlog potetial positios of these odes, with high probability oe will lad i the legth-ε/ iterval I uder cosideratio. So whp, for each of the first 1 ε addresses i the order, the potetial ode spaig that address will lad withi distace ε/ precedig the address. Sice these first 1 ε addresses break up the uit circle ito itervals of size at most 4/, claim iii follows. Note that for very small ε, the first 1 ε addresses actually break up the uit circle i itervals of size /, which shows the additioal claim. For claim iv, it suffices to cosider a deletio sice the system is Markovia, i.e. the deletio ad additio of a give ode are symmetric ad cause the same umber of chages. Cosider what happes whe a ode assiged to some address a 0,b 0 gets deleted from the etwork. The some ode previously assiged to a address a 1,b 1 a 0,b 0 may get reassiged to a 0,b 0, causig some ode previously assiged to a,b a 1,b 1 to move to a 1,b 1, ad so o. This results i a liear sequece of chages that, whe complete, have produced the ideal state. Sice odes oly move to smaller i terms of addresses, the umber of movemets is clearly fiite. We will ow show that this umber is Ologlog i expectatio. Let i be the umber of odes assiged to active addresses a,b with a,b a i,b i. Note that the ode movig to address a i,b i is uiformly radom amog these i odes, thus i+1 i / with probability at least 1/. Sice this is true for all i, whp Olog movemets suffice to achieve log halvigs, which will reduce i to zero. This shows that whp Olog odes have to chage their addresses upo the deletio or isertio of a ode. However, the exchage sequece ca also stop before i is reduced to zero. This happes if oe of the i odes has a potetial positio betwee address a i,b i ad the active ode preceedig it. Sice we assumed the system to be load-balaced, the distace betwee a i,b i ad the preceedig active ode is O1/. The gap therefore cotais l = Olog potetial odes whp. The probability that oe of the l potetial odes i the gap is owed by ay of the i active odes that would wat to move to them is 1 i l exp l i /. This probability becomes a costat whe / i = Ωl, i.e. i = O/log, at which poit we perform i expectatio oly a costat umber of additioal moves. By the above discussio, i halves i every step i expectatio, so it takes oly log O/log =

5 Ologlog movemets i expectatio to get i = O/log. Thus the expected umber of moves i total is Ologlog + O1 = Ologlog.. Discussio Ituitively, the protocol achieves load-balace for two reasos. First, sice the odes prefer to be ear a address that is small i the orderig, there will be active odes close to almost all small accordig to addresses. Secod, the addresses i ay begiig segmet of the orderig almost uiformly subdivide the address rage [0,1]. Combiig these two facts implies the load-balace. We ote that the above scheme is highly efficiet to implemet i the Chord PP protocol, sice oe has direct access to the addresses of successors i the address rig. Moreover, the protocol ca also fuctio whe odes disappear without ivokig a proper deletio protocol. By havig every ode occasioally check whether they should move, the system will evetually coverge towards the ideal state. This ca be doe with isigificat overhead as part of the geeral maiteace protocols that have to ru ayway to update the routig iformatio of the Chord protocol. Oe possibly udesirable aspect of the above scheme is that Ologlog odes chage their address upo the isertio or deletio of a ode, because this will cause a Ologlog/ fractio of all items to be moved. However, sice every ode has oly Olog possible positios, it ca cache the items stored at previous active positios, ad will evetually icur little data migratio cost: whe returig to a previous locatio, it already kows about the items stored there. Alteratively, if every real ode activates Ologlog potetial odes istead of just 1, we ca reduce the fractio of items moved to O1/, which is optimal withi a costat factor. This is show by a straightforward variatio o our aalysis, usig the fact that with loglog activated odes per real ode, each move oly ivolves o average a 1/loglog fractio of the data. All other performace characteristics are carried over from the origial scheme. It remais ope to achieve O1/ data migratio ad O1 virtual odes while attaiig all the other metrics we have achieved here. 3. ITEM BALANCING We have show how to balace the address space, but sometimes this is ot eough. Some applicatios, such as those aimig to support rage-searchig operatios, eed to specify a particular, oradom mappig of items ito the address space. I this sectio, we cosider a dyamic protocol that aims to balace load for arbitrary item distributios. To do so, we must sacrifice the previous protocol s restrictio of each ode to a small umber of potetial ode locatios istead, each ode is free to migrate aywhere. This is uavoidable: if each ode is limited to a bouded umber of possible locatios, the for ay odes we ca eumerate all the places they might possibly lad, take two adjacet oes, ad address all the items i betwee them: this assigs all the items to oe ufortuate ode. Our protocol is radomized, ad relies o the uderlyig PP routig framework. If the ode distributio is very skewed, it might be ecessary to augmet the routig ifrastructure, see Sectio 3..3 below. The protocol is the followig where ε is ay costat with 0 < ε < 1/4. Recall that each ode stores the items whose addresses fall betwee the ode s address ad its predecessor s address, ad that l j deotes the load o ode j. Here, the idex j rus from 1,,..., i the order of the odes i the address space. Item balacig: Each ode i occasioally cotacts aother ode j at radom. If l i εl j or l j εl i the the odes perform a load balacig operatio assume wlog that l i > l j, distiguishig two cases: Case 1: i = j + 1: I this case, i is the successor of j ad the two odes hadle adjacet address itervals. Node j icreases its address so that the l i l j / items with lowest addresses i i s iterval get reassiged from ode i to ode j. Both odes ed up with load l i + l j /. Case : i j + 1: If l j+1 > l i, the we set i := j + 1 ad go to case 1. Otherwise, ode j moves betwee odes i 1 ad i to capture half of ode i s items. This meas that ode j s items are ow hadled by its former successor, ode j + 1. This protocol also quickly balaces the load, startig with a arbitrary load distributio. Lemma Startig with a arbitrary load distributio, if every ode cotacts Olog radom odes for the above protocol, the whp all odes will ed up with a load of at most 16 ε L. Aother roud of everyoe cotactig Olog other odes will also brig all loads ε to at least 16 L. Proof Sketch: Sice the proof is similar to the oe give for Theorem 3 below, we oly sketch the geeral outlie. Cosider oe particular ode with load at least 16 ε L. If this ode cotacts a radom ode, the with probability at least 1/ it will be able to eter i a load exchage. Cotactig Θlog other odes will therefore lead to Θlog load exchages whp, each reducig the ode s load by a costat factor. Thus, idepedet of the startig load, the fial load will be at most 16 ε L. For the lower boud of the load, a similar argumet applies. To state the performace of the protocol oce a load balaced state has bee reached, we eed the cocept of a half-life [11], which is the time it takes for half the odes or half the items i the system to arrive or depart. Theorem 3 If each ode cotacts Ωlog other radom odes per half-life as well as wheever its ow load doubles or halves, the the above protocol has the followig properties. i With high probability, the load of all odes remais betwee ε L ad ε L. ii The amortized umber of items moved due to load balacig is O1 per item isertio or deletio, ad OL per ode isertio or deletio. 3.1 Proof of Theorem 3 For part i, cosider a ode with load l. Suppose it eters a load exchage with a ode of load l εl. The the ode s load reduces by a factor of at least l 1 l + l l = =: β. 1 l + εl 1 + ε Cosider oe particular ode, whose load icreases beyod ε L. We will show that its load will drop below ε L before ever risig as high as 16 ε L. For this it suffices if this ode has log β 16/ε /ε = log β 8 = Θ1 successful load exchages before its load icreases to 16 ε L. By Markov s iequality, half the odes have load at most L. So if a ode with load exceedig ε L cotacts a radom ode, the with probability 1/ it will eter a load exchage, ad Θlog ivocatios of the protocol will lead to a load exchage whp. It

6 therefore suffices to ivoke the protocol Olog times before the load icreases from 16 ε L to ε L. What ca cause the load of a ode to icrease? First, the umber of items stored at the ode ca icrease. Secod, the value of L = N/ ca drop globally, either by a decrease of the umber of items i the system, or a icrease of the umber of odes i the system. The effective chage i relative load of a ode is the product of these three effects. Thus, for the relative ode to chage by a factor of 8 from 16 ε L to ε L, at least oe of the values has to chage by a factor of. So the update rate stated i the Theorem is sufficiet to maitai the claimed load-balace. A similar argumet yields a lower boud o the load of all odes, otig that the upper boud of 16 ε L o the load implies that a costat fractio of all odes have a load of at least L/. This meas that a ode with little load i.e. less tha ε L load is likely to cotact oe of these odes at radom. For part ii, we use a potetial fuctio argumet. We show that item ad ode isertios ad departures cause oly limited icreases i the potetial, while our balacig operatio causes a sigificat decrease i the potetial if it is large. This potetial fuctio is Φl := δ l i logl i N logl, i=1 where δ is a sufficietly large costat, e.g. δ = 8. Recall that l i is the load of ode i, i.e. the umber of items stored at the ode sice we are cosiderig the uweighted case. Our potetial fuctio is related to the etropy of the item distributio. More precisely, up to a additive term idepedet of the item distributio, the potetial fuctio is exactly the egative of the etropy. Thus, our fuctio gets miimized whe all odes have the same load. The amortized cost of a operatio isertio, deletio, or load exchage will be its actual cost plus the resultig chage i potetial fuctio, i.e. amortized cost = actual cost + Φ after Φ before. Item isertio The actual cost of isertig a item is 1, sice the affected item has to be hadled. So the amortized cost of isertig a item at a ode j is 1 + δ l i logl i + l j + 1logl j + 1 N + 1log N + 1 i j l i logl i + N log N i = 1 + δ l j + 1logl j + 1 l j logl j + N log N N + 1log N + 1 l j + 1 l j = 1 + δ logl j log log N l j N/ N+1 +log = 1 + δ N + 1/ logl j log l j log N l j +log 1 1 N+1 N + 1 = 1 + δ logl j + 1 logl + Θ1 = Θ 1 + logl j + 1 logl. If l j = OL, the cost reduces to O1. Item deletio The actual cost of a deletio is 0, sice o item has to be moved. The chage i potetial is the egative of a item isertio s, ad thus Θ1 logl j + logl. This cost is O1 sice l j = ΩL. Node isertio The actual cost of addig a ew ode is zero, as o items are moved. The chage i potetial fuctio ad therefore the amortized cost is δn log N δn log N + 1 N = δlog = δlog L Node deletio = Θloge L = ΘL. Whe deletig ode j ad movig its items to ode k, we icur a actual cost of l j items. The amortized cost of deletig a ode is therefore: l j + δ = l j + δ log = l j + Θ log = Θl j + l k L. l j + l k logl j + l k l j logl j l k logl k log L 1 + l k l j e l k e l j e L If l j,l k = OL, the amortized cost is OL. Load balacig operatio l j 1 + l lk j L l k It remais to show that the expected amortized cost of a load exchage is egative. For this, we will eed the assumptio that ε < 1/4. Let us first cosider the case i = j + 1. Whe movig items from ode i to ode j, the iitial loads o those odes are l i ad l j, while both odes will ed up with a load of l i + l j /. Thus, l i l j / items have to be moved, which is the actual cost of the load exchage. The amortized cost therefore comes out to l i l j + δ l i + l j log l i + l j l i logl i l j logl j. We have to show that this quatity is at most 0. For otatioal simplicity, let η := l j l i ε. The we have the cost 1 η l i + δ l i 1 + ηlogl i + log1 + η 1 l i logl i ηl i logηl i

7 1 η = l i + δ 1 + ηlogl i ηlog1 + η 1 + η logl i ηlogη ηlogl i 1 η = l i + δ 1 + ηlog1 + η ηlogη 1 + η 1 η = l i + δ l i 1 + δ l i δ, log1 + η + log η 1 + η η η log / 1 + η 1/ usig η ε < 1/. Thus, for δ > 4.17, we obtai that load exchages are paid for by the drop i potetial fuctio. The case i j + 1 ivolves the three odes i, j ad j + 1. To simplify otatio, we set x := l i, y := l j ad z := l j+1. Recall that we have y εx ad z x. The actual umber of items moved is x/ + y. The three odes cotributio to the potetial fuctio is δx logx + ylogy + zlogz before the update, ad x δ log x + x log x + y + zlogy + z after the update. So the chage i potetial fuctio is Φ = δ x log x + y + zlogy + z xlogx + ylogy + zlogz = δ x + y + zlogy + z ylogy zlogz. Note that the fuctio f y,z := y + zlogy + z ylogy zlogz is icreasig i y ad z for y,z 0, sice 1 f = logy + z logy 0 y ad likewise 1 f 0 by symmetry. z Thus, the cost of the load balacig operatio gets maximized for y = εx, ad z = x. The maximal amortized cost therefore is x + y + δ x + f εx,x = x 1 + y + xδ εlog1 + εx εlogεx logx = x 1 + y + xδ εlog1 + ε εlogε = x 1 + y + xδ + log1 + ε + log ε ε 3 x 4 + δ 1 + log log 4 5 x δ. This is less tha 0 if δ > Discussio The traffic caused by the update queries ecessary for the protocol is sufficietly small that it ca be buried withi the maiteace traffic ecessary to keep the PP etwork alive. Cotactig a radom ode for load iformatio oly uses a tiy message, ad does ot result i ay data trasfers per se. Of greater importace for practical use is the umber of items trasferred, which is optimal to withi costats i a amortized sese. The protocol ca also be used if items are replicated to improve fault-tolerace, e.g. whe a item is stored ot oly o the ode primarily resposible for it, but also o the Olog followig odes. I that settig, the load l j refers oly to the umber of items for which a ode j is primarily resposible. Sice the item movemet cost of our protocol as well as the optimum icrease by a factor of Olog, our scheme remais optimal withi a costat factor Selectig Radom Nodes A crucial step i our protocol is the ability to be able to cotact a radom ode i the PP etwork. This is easy to achieve if the odes are almost uiformly distributed i the address space: we pick a address uiformly at radom, ad the cotact the ode succeedig that address. The probability of selectig a ode is equal to the fractio of address space spaed by it. It ca be show that our aalysis still holds if all these probabilities are withi a costat factor of 1. If the ode distributio is skewed, the we have to employ a differet scheme. Oe solutio is that every ode maitais a additioal presece i.e. virtual ode o the address rig. This virtual ode is ot used for storig items i.e. does ot take part i the item balacig protocol, its sole purpose is to be cotacted for the radom ode selectio. Usig a balacig protocol such the oe i Sectio to distribute these virtual odes evely i the address space, we ca agai select a radom ode by choosig a radom address ad returig the virtual ode followig it. Aother solutio for skewed ode distributios is the use of the radom skip list which we might also use for routig see Sectio 3..3 below. Selectig a radom elemet i such a data structure is a relatively straightforward task. 3.. Weighted Items I may circumstaces, items stored i a PP etworks are ot all equal i terms of the load they put o the hostig ode. For example, items might have differet sizes, so storig a larger items requires more disk space, or the popularity of items ca differ, with some items beig requested more ofte, leadig to a higher I/O load o the hostig ode. We ca model this by assigig a weight wx to every item x. It turs out that our load balacig algorithm also works for the case where the load is defied as the sum of item weights, as opposed to the umber of items stored at a ode. There is oe obvious restrictio: load will be balaced oly up to what the items themselves allow locally. As a example cosider two odes, oe ode storig a sigle item with weight 1, the other ode a sigle item with weight 100. If these two odes eter i a load exchage, the there is o exchage of items what will equalize the two loads. Apart from this restrictio, all the above aalysis carries over to the weighted case, by simply treatig a item with weight w as w items of weight 1. Corollary 4 Theorem 3 cotiues to hold for weighted items, with the followig chages: i Load ca be balaced oly up to what the items weights allow locally see previous discussio. ii The amortized total weight moved upo the isertio or deletio of a item with weight w is Ow Routig i Skewed Distributios If the ode distributio i the address space is very skewed, e.g. with very dese or adversarial clusterigs of odes, the Chord s routig is ot guarateed to take oly Olog hops. Sice for

8 our protocol the ode distributio mirrors the item distributio, this possibility caot be excluded a priori. I such a case, a alterate routig ifrastructure has to be created to retai the Olog hop routig guaratee. Oe possibility is to use a radom skip list: the skip list is directed aroud the address rig, ad odes isert themselves at their active address with radomly chose levels, where the probability of choosig higher levels decreases geometrically. For idepedet choices of levels, this retais the Olog whp routig guaratee of basic Chord, but works for arbitrary distributios of odes i the address space Rage Searches Our protocol ca provide load balace eve for data that caot be hashed. I particular, give a ordered data set, we may wish to map it to the [0,1 iterval i a order-preservig fashio. Our protocol the supports the implemetatio of a rage search data structure. Give a query key, we ca use Chord s stadard lookup fuctio or the routig structure described i Sectio 3..3 to fid the first item followig that key i the keys defied order. Furthermore, give items a ad b, the data structure ca follow ode successor poiters to retur all items x stored i the system that satisfy a x b. Sice all odes have a load of ΘL, we obtai the first such protocol that achieves a Olog + K/L query time where K is the size of the output. 4. CONCLUSION We have give several provably efficiet load balacig protocols for distributed data storage i PP systems. Our algorithms are simple ad easy to implemet, so a obvious ext research step should be a practical evaluatio of these schemes. I additio, three cocrete ope problems follow from our work. First, it might be possible to further improve the cosistet hashig scheme as discussed i Sectio.. Secod, it would be iterestig to determie whether our item balacig protocol also works for the case where the cost of storig a item is ode-depedet, e.g. because some odes have greater storage capacity or badwidth tha others. Ad fially, our rage search data structure does ot easily geeralize to more tha oe order. For example whe storig music files, oe might wat to idex them by both artist ad year, allowig rage queries accordig to both orderigs. Sice our protocol rearrages the items accordig to the orderig, doig this for two orderigs at the same time seems difficult. A simple solutio is to rearrage ot the items themselves, but just store poiters to them o the odes. This requires far less storage, ad makes it possible to maitai two or more orderigs at oce. It is ope how to solve this problem without such a added level of idirectio. Ackowledgmets We would like to thak the aoymous reviewers for their helpful commets. 5. REFERENCES [1] M. Adler, E. Halperi, R. M. Karp, ad V. V. Vazirai. A Stochastic Process o the Hypercube with Applicatios to Peer-to-Peer Networks. I Proceedigs STOC, pages , Jue 003. [] A. Aagostopoulos, A. Kirsch, ad E. Upfal. Stability ad Efficiecy of a Radom Local Load Balacig Protocol. I Proceedigs FOCS, pages , Oct [3] J. Aspes ad G. Shah. Skip Graphs. I Proceedigs SODA, pages , Ja [4] P. Gaesa ad M. Bawa. Distributed Balaced Tables: Not Makig a Hash of it all. Techical Report , Staford Uiversity, Database Group, Nov [5] M. Harre, J. M. Hellerstei, R. Huebsch, B. T. Loo, S. Sheker, ad I. Stoica. Complex Queries i DHT-based Peer-to-Peer Networks. I Proceedigs IPTPS, pages 4 50, Mar. 00. [6] R. Huebsch, J. M. Hellerstei, N. Laham, B. T. Loo, S. Sheker, ad I. Stoica. Queryig the Iteret with PIER. I Proceedigs VLDB, pages 31 33, Sept [7] F. Kaashoek ad D. R. Karger. Koorde: A Simple Degree-optimal Hash Table. I Proceedigs IPTPS, Feb [8] D. Karger, E. Lehma, T. Leighto, M. Levie, D. Lewi, ad R. Paigrahy. Cosistet Hashig ad Radom Trees: Tools for Relievig Hot Spots o the World Wide Web. I Proceedigs STOC, pages , May [9] J. Kubiatowicz, D. Bidel, Y. Che, S. Czerwiski, P. Eato, D. Geels, R. Gummadi, S. Rhea, H. Weatherspoo, W. Weimer, C. Wells, ad B. Zhao. OceaStore: A Architecture for Global-Scale Persistet Storage. I Proceedigs ASPLOS, pages , Nov [10] D. M. Lewi. Cosistet Hashig ad Radom Trees: Algorithms for Cachig i Distributed Networks. Master s thesis, Massachusetts Istitute of Techology, May [11] D. Libe-Nowell, H. Balakrisha, ad D. Karger. Aalysis of the Evolutio of Peer-to-Peer Systems. I Proceedigs PODC, pages 33 4, July 00. [1] D. Malkhi, M. Naor, ad D. Ratajczak. Viceroy: A Scalable ad Dyamic Emulatio of the Butterfly. I Proceedigs PODC, pages , July 00. [13] P. Maymoukov ad D. Mazières. Kademlia: A Peer-to-peer Iformatio System Based o the XOR Metric. I Proceedigs IPTPS, pages 53 65, Mar. 00. [14] M. Naor ad U. Wieder. Novel Architectures for PP Applicatios: the Cotiuous-Discrete Approach. I Proceedigs SPAA, pages 50 59, Jue 003. [15] A. Rao, K. Lakshmiarayaa, S. Suraa, R. Karp, ad I. Stoica. Load Balacig i Structured PP Systems. I Proceedigs IPTPS, Feb [16] S. Ratasamy, P. Fracis, M. Hadley, R. Karp, ad S. Sheker. A Scalable Cotet-Addressable Network. I Proceedigs ACM SIGCOMM, pages , Aug [17] A. Rowstro ad P. Druschel. Pastry: Scalable, distributed object locatio ad routig for large-scale peer-to-peer systems. I Proceedigs IFIP/ACM Iteratioal Coferece o Distributed Systems Platforms Middleware, pages , Nov [18] M. Ruhl. Efficiet Algorithms for New Computatioal Models. PhD thesis, Massachusetts Istitute of Techology, Sept [19] J. P. Schmidt, A. Siegel, ad A. Sriivasa. Cheroff-Hoeffdig bouds for applicatios with limited idepedece. I Proceedigs SODA, pages , Ja [0] R. G. Seidel ad C. R. Arago. Radomized Search Trees. Algorithmica, 164/5: , Oct./Nov [1] I. Stoica, R. Morris, D. Karger, F. Kaashoek, ad H. Balakrisha. Chord: A Scalable Peer-to-peer Lookup Service for Iteret Applicatios. I Proceedigs ACM SIGCOMM, pages , Aug. 001.