Distributed, Secure Load Balancing with Skew, Heterogeneity, and Churn

Transcription

1 Ditributed, Secure Load Balancing with Skew, Heterogeneity, and Churn Jonathan Ledlie and Margo Seltzer Diviion of Engineering and Applied Science Harvard Univerity Abtract Numerou propoal exit for load balancing in peer-to-peer (pp) network. Some focu on namepace balancing, making the ditance between node a uniform a poible. Thi technique work well under ideal condition, but not under thoe found empirically. Intead, reearcher have found heavytailed query ditribution (kew), high rate of node join and leave (churn), and wide variation in node network and torage capacity (heterogeneity). Other approache tackle thee le-thanideal condition, but give up on important ecurity propertie. We propoe an algorithm that both facilitate good performance and doe not dilute ecurity. Our algorithm, k-choice, achieve load balance by greedily matching node target workload with actual applied workload through limited ampling, and limit any fundamental decreae in ecurity by baing each node et of potential identifier on a ingle certificate. Our algorithm compare favorably to four other in trace-driven imulation. We have implemented our algorithm and found that it improved aggregate throughput by 0% in a widely heterogeneou ytem in our experiment. I. INTRODUCTION Decentralized tructured overlay and ditributed hah table proffer a unique viion of computing: each machine eamlely contribute to and benefit from a large ervice-oriented network. Thi viion ha yet to be realized, in part, becaue machine are not identical, becaue the workload applied to the ytem may be heavy-tailed, and becaue node availability and churn rate may change over time. Learning to adapt to thee characteritic through load balancing in a decentralized, calable, and ecure manner i a tep toward realizing thi ideal of computing. Several exiting propoal for load balancing algorithm in thi context have focued on ideal condition [], [6], [30], [3]. They have made unrealitic aumption about node heterogeneity, workload kew, and node churn. In general, they have aumed that node are uniform, that there i no kew in the workload, and that node are neither arriving nor departing. Deployed ytem do not adhere to thee idealitic condition [39], [45]. Other propoal have attempted to handle kew, churn, and heterogeneity [], [0], [35]. Thoe that achieve good performance let node join a normal and then reactively poition node to arbitrary location in the namepace. Arbitrarily chooing identifier (ID) forfeit an important ecurity goal for pp ytem: a verifiable identifier. Without verifiable ID tying virtual overlay addree to pecific agent, application building block uch a reputation [3], micropayment [46], and auction [3] are not poible outide of a truted network. In thi paper, we propoe k-choice, a load balancing algorithm for tructured overlay that upport wide variation in kew, heterogeneity, and churn while retaining the ecurity and application advantage afforded by verifiable ID. At a high level, the algorithm work a follow: (a) each node generate a et of verifiable ID baed on a ingle unit of certified information; (b) at join time, a node greedily reduce dicrepancie between capacity and load both for itelf and for node that will be affected by it join; and (c) optionally, each node experiencing overload or underload may periodically probe the network and repoition itelf to another element from it et of verifiable ID. Minimizing dicrepancie between load and capacity achieve load balance, and limiting ID to a well-defined et keep the algorithm ecure. Thi paper proceed a follow. In Section II, we introduce our model and aumption. In Section III, we preent the k- Choice algorithm in detail. In Section IV, we review four tate-of-the-art algorithm for load balancing in pp ytem. In Section V and VI, we preent reult from trace-driven imulation where we vary ytem characteritic, including node heterogeneity, kew, and churn. We alo preent reult from an implementation of k-choice. Section VII and VIII preent related work and concluion, repectively. II. MODEL In thi ection, we introduce our model and aumption for load balancing in pp ytem. Overload. Phyical node, i.e., computer, participate in pp ytem. Each node n i ha a capacity C i, which correpond to the maximum amount of load that node can proce per unit time. Node create virtual erver (VS), which join the pp network. A node n might have j VS v, v,..., v j, each with load w, w,..., w j, repectively. Load i applied to node via their virtual erver. In a unit of time, node n i might have load (work) W i = w + w w j. Overload occur when, for a node n i, W i > C i. An overloaded node i not able to tore object given to it, route packet, or perform computation, depending on the application. A node fail to proce requet that impoe work beyond it capacity. Per unit time, the ucceful work per node i: { Wi, if (W S i = i C i ) C i, otherwie The utilization of a node n i i W i /C i. Node may want to operate below their capacity C to prevent fluctuation in work-

2 virtual erver Fig.. workload w 4 w 3 w w Uδ node target workload L δ Target and Capacity Workload. node capacity load from temporarily overloading them. Uing terminology from Rao et al. [35], we ay a node n i ha an upper target U i and lack U δ uch that U i = C i U δ. If a node find itelf receiving more work than U i, it conider itelf overloaded. Node alo have a lower target L i below which they conider themelve underloaded. How a node repond to either of thee condition depend on the algorithm. An illutration of how we repreent node i hown in Figure. We aume each node know it capacity C and it upper and lower target. Each node tore it virtual erver in a et, called VSet of ize VSet.ize. Depending on the algorithm, thi ize may have an upper bound of VSet.maxize. Routing. Structured overlay allow routing of meage to detination on top of an underlying network contantly undergoing topology change [36], [38], [43], [47]. Each meage detination ID i a number on the overlay namepace D, e.g., D = 60. Meage travere overlay hop from a ource VS to a detination VS. The number of hop i typically O(log(N)), where N i the current number of VS. Each VS ha a unique ID choen from the namepace D. In our model, the detination of a meage i the VS with the next larget logical identifier on the namepace mod D. The VS with the next larget (mallet) ID i called the ucceor (predeceor). We denote the ditance in the namepace between two virtual erver i and j with dit(i, j). Each tructured overlay allow new VS to join the ytem. In general, each VS join and departure require O(log(N)) maintenance meage. Reactive load balancing algorithm ue artificial join and departure to change ID. Network a Bottleneck. We focu on how load balancing algorithm function at the routing level. Blake and Rodrigue provide evidence that even in remote torage application, network bandwidth i likely to be the primary bottleneck []. A torage become cheaper and cheaper relative to bandwidth, particularly lat-mile bandwidth, thi cae will likely become more common. In compute-dominated cenario, whether the proceing or the network will be the bottleneck depend on the application. We let a node n i capacity C i be the number of routing hop it can provide per unit time. We compare algorithm on the percentage of meage that uccefully reach their detination. Security. A key iue in the operation of a pp network i whether or not one aume it may contain maliciou node. A maliciou node can ubvert content or attempt to control particular portion of the identifier pace. Attack that center around the falification of a node identifier are called Sybil attack [4]. Douceur outline the main difficultie in allowing node to chooe their own ID. He how that validating node mut verify all other node credential imultaneouly, an act that may exceed the verifier reource. A ytem may acquire a low level of ecurity by requiring that ID be baed on the hah of the node IP addre []. However, falifying IP addree i traightforward; baing any level of authentication on IP addree would not repel a determined attacker. For thi reaon, Catro et al. propoe that each ID k i certified by a central authority, which generate k cert [9]. Thi option i calable becaue each node contact thi authority once, the firt time it join the ytem. Intead of having thi authority certify ID, we propoe that it certify a unique number x for each node, creating x cert. Each node can then ue thi number to generate it own ID uing an ID-generating hah function h. For a node with ID k, a verifier verifie that k = h(x cert ) intead of k = k cert. k-choice create a et of verifiable ID by generating each k = h(x cert + c) where c ha a well-known bound. We refer to x cert a x below for purpoe of preentation. The k-choice olution we propoe retain thi Sybil attack reilience. Algorithm that permit a node to relocate it virtual erver to an arbitrary node ID location do not have thi quality. Algorithm that do not allow for certified ID can only be expected to function in a truted environment. Sytem Characteritic. Although tructured overlay are targeted to provide the framework for application uch a application-level multicat [8], ditributed torage [], [6], and publih-ubcribe content ditribution [34], [4], there are no benchmark workload. Gummadi et al. and other have found Zipf query ditribution in their trace analyi of Kazaa [3], [4], [39] and thi ditribution i common to many other uage (e.g., web page file acce [8], file popularity [7]). We examine load balancing under uniformly random and Zipf querie. A Zipf workload with parameter α mean that detination are ranked by popularity. Detination with rank i i α time more likely to be acceed than that with rank i+. A characteritic related to kew i workload hift. Shift refer to a change in workload kew. For example, on one day, one tored object might be the mot popular, on the next, a different one might be, but the general ditribution would be the ame. Studie of object popularitie in deployed pp ytem have found the exitence of hifting Zipf kewed workload [4]. A third characteritic i the ditribution of node capacitie. A i generally the cae in pp cenario, bandwidth i the main capacity limiter []. In the trace which we draw from, node capacitie vary by ix order-of-magnitude [39] and a imple function doe not capture the trace bandwidth ditribution well. A final characteritic i the ditribution of node join and departure (churn). A we dicu in Section V, thi cannot be captured with a imple rate λ. Intead, churn tend to be Pareto: heavy-tailed and memory-full. Node that have been in the ytem for a long time tend to remain longer than average [3]. Pareto ditribution have two parameter, hape

3 { ID Choice # would hift w work from b to a w (now) ID # w b (f.) w b (b,exiting) (a, joining) {, { ID Choice # would hift w work from c to a Deired Workload w (now) w ID # c (f.) w c (c,exiting) (ame a, joining) { Deired Workload, Ue imple cot function ID Choice k to find bet workload would hift... hift match and join there., ; Fig.. A part of the Join proce, k-choice hift workload for each of the VS that are created. K-CHOICES VS JOIN(t a ) K {k 0 h(x + 0),..., k κ h(x + κ )} Remove in-ue ID from K 3 for each k in K 4 do Query ucc(k) for w (n) 5 r dit(pred(k),k) dit(pred(k),ucc(k)) 6 w a r w (n) 7 w (f) ( r) w (n) 8 c t w (f) 9 Join at k with minimum c return w a and t + t a w a t w (n) K-CHOICES NODE JOIN(T ) T (U i + L i )/ i κ/ 3 while T > 0 and i > 0 4 do T T K-CHOICES VS JOIN(T ) 5 i i Fig. 3. k-choice join algorithm. w (n) work now and in the future, repectively. and w (f) α and cale β, and have a mean of α β α. III. k-choice ALGORITHM denote the ucceor k-choice i a greedy, cot-baed load balancing algorithm for tructured overlay. It matche node workload goal with guee about how their choice of identifier will affect both their own workload and thoe of their neighbor. At each VS inertion, k-choice minimize the dicrepancy between work and capacity by ampling from a mall et of potential ID. By limiting the number of potential ID, k-choice i practical for network containing maliciou participant. k-choice function primarily at node join time a hown in k-choice Node Join in Figure 3. When a node join, it chooe a total target workload and an upper bound on the number of VS to create (line -). Then, it invoke k-choice VS Join and reduce it remaining capacity by the anticipated work of that VS (line 3). Thi continue until it ha created κ/ VS or reached it target workload. Making everal VS together at join time amortize the cot of ampling. The join for each VS i compoed of four tep, a hown in Figure and in k-choice VS Join in Figure 3. A mall menu of potential ID i choen, limited by a well-known contant κ (line -). Thee ID are verifiable becaue they are all baed on the certified x and becaue κ i bounded. To verify that a node i uing a valid ID, k, another node imply ha to check that there exit ome i < κ uch that k = h(x + i). Next, each potential ID ucceor i probed to dicover what i likely to happen were thi VS to be placed at thi location (line 4). It guee that the current work for thi location will be plit baed on the percentage of the addre pace the joining VS will take on (line 6-7). The node ue thi to compute the change from the current ituation (line 8). Each term in the cot function i the difference between target work and real work. The firt two term are the um of the difference if thi VS i created and the lat i the current ituation. We normalize each term baed on the node capacity. Thu, the lower the cot, the maller the difference between target and actual work. The lat tep of the join proce i to join at the ID with lowet cot. Becaue node et their target lower than their capacitie, if all node minimized the mimatch m = t w = 0, then lo would be zero. If node do not attempt to perform any additional load balancing after joining, we ay they are paive. Being paive ha it advantage: no additional churn i induced through VS relocation. However, over time one of the other potential ID for thi VS can become ignificantly better in term of improving target/workload mimatch. If we permit reelection of ID, we ay that k-choice i active. To minimize network probing, node reelect only a ingle VS ID at a time. They pick the v VSet with the maximum mimatch. They check if any new ID for v improve the aggregate mimatche of themelve and their neighbor by ɛ, a parameter that dampen improvement of minimal benefit. If it doe, the movement i performed. ɛ i applicationdependent: when a ytem i ued for routing, moving will be relatively painle, a VS can gracefully notify incoming pointer of their departure; if object are tored and need to be ent over the network, the cot might be ignificantly greater. Node only examine the poibility of relocating if they are overloaded or underloaded. If node have relocated more than VSet.ize time and are till overloaded or underloaded, they create or detroy VS within the range (, κ). In practice we found that node did not create more than a handful of additional VS. k-choice poee everal attractive feature and make certain aumption. When run in paive mode, it add no

4 reactive churn. In fact, without an active component, it require natural churn. By making a good choice before routing i et up, object are tored, or computation are tarted, k-choice leen or eliminate thi reactive load balancing penalty. We aume that node do not lie or that Ditributed Algorithmic Mechanim Deign technique could be ued to encourage the truthfulne of the information they provide about load [9], [4], another reaon why verifiable ID are important. We alo aume that the ytem i not primarily being ued for range querie. Limited ID aignment provably cannot balance load in thi cae [6]. Note that VS could keep more accurate track of where work i landing in their namepace to make w a and w (f) more accurate. Intead, we decided to ue a imple exponentiallyweighted moving average to reduce the amount of tate ent during probing. Optimal ID Choice. k-choice exhibit diminihing return a κ approache the ize of the namepace D. When κ = D, each joining VS would ample every poible ID (auming a perfect hah function). In fact, it i feaible to find the ID (or ID) with the lowet cot by examining only a few variable for each exiting VS. While even thi ampling would be prohibitively expenive in an implementation, performing it offline within a imulator i not. For each potential ucceor, we know it target t and it. The goal i to find the percent of the addre pace r between pred() and that give the minimum cot c and to find what the cot i for thi. The optimal ID choice will be the pred() + r dit(pred(), ) with the globally actual work w (n) lowet cot. We know that w (n) 0 and that m = t w (n) i fixed regardle of the r choen. If we do not normalize for each node capacity, there are four mutually excluive cae for r and c: cae cae cae cae t 0 and t a w (n) : r = ; c = t a t + w (n) rw (n) t a 0 and t w (n) : r = 0; c = t t a w (n) + rw (n) t a + t < w (n) r ( ta c = c m w (n) :, t w (n) : t t a w (n) w (n) t a + t w (n) r (, ); c = w (n) t a t ); c = t a + t w (n) In cae and, we do not eliminate r becaue ID cannot be identical. The actual choice will need to be a mall ditance away. IV. PRIOR LOAD BALANCING TECHNIQUES In thi ection, we dicu the four exiting load balancing algorithm againt which we will compare k-choice: log(n) VS, Proportion, Tranfer, and Threhold. The firt, log(n) VS, olely attempt to evenly partition the namepace between node, ignoring heterogeneity and kew. Proportion create { Capacity node: (a) Slack VS (b) VS per node (a) Slack VS (b) 3 VS per node Fig. 4. log(n) VS: During join, a node can divide itelf into everal virtual erver, which then join independently. When all node do thi, dicrepancie in the average total namepace per node diminih. VS in proportion to capacity at join time and make adjutment baed on workload. Tranfer and Threhold ue arbitrary VS relocation to adjut to kew and heterogeneity. Tranfer and Threhold, in particular, are repreentative of the current tate-of-the-art in load balancing algorithm for tructured overlay. Proportional i particularly intereting becaue of it complete decentralization. log(n) VS allow u to how the pro and con of pure namepace balancing. Becaue Proportional limit VSet.ize to ome well-known maximum, it alo doe not fundamentally change the ecurity characteritic of the ytem. However, it performance i ignificantly inferior to Tranfer, Threhold, and k-choice. A. log(n) Virtual Server The implet load balancing technique we dicu i log(n) VS. It balance node namepace and doe not permit arbitrary ID. It wa firt introduced by Karger et al. [5]. The log(n) VS load balancing algorithm follow from the obervation that randomly choen node ID do not uniformly cover the identifier pace. In fact, the ditribution of namepace i roughly Poion, with the larget being O(log(N)) time the average. log(n) VS i predicated on the aumption that workload and capacity are uniform. When thee aumption hold, if each node ha a ingle VS, thoe few node at the tail become bottleneck. By the Central Limit Theorem, the more VS each node make, the more normal (and balanced) the average (total) namepace of each node become. Becaue there are drawback to having too many VS, thi algorithm ugget that each node having log(n) VS reache a good compromie. All node then have average load within a contant factor. An illutration of thi i hown in Figure 4. Thi technique i nonreactive: it make no attempt to rebalance load after a node join. Thi algorithm work well for the cae when it aumption on capacitie and workload ditribution hold. However, increaing the number of VS caue a few problem. Firt, it increae churn becaue when one node depart, it mut take it log(n) VS with it, cauing log(n) time more adjutment to be made. Second, each node mut hold log(n) time a much routing tate. Finally, becaue there are more VS in the ytem, the number of hop per lookup (and latencie) increae. Propoal have been made to mitigate the lat two problem, but they have not been evaluated [].

5 Overload New VS (a) (a) load hedding (b) (b) load acquiition (a) (b) tranfer (a) (a) (b) plit and tranfer Fig. 5. Proportion: Underloaded node create new virtual erver (up to ome maximum). Overloaded node detroy their own virtual erver (keeping one). PROPORTION-ADJUST() ( Initially create VS in proportion to capacity ) if overloaded and VSet.ize > 3 then Delete VS that will bet unload u W 4 if underloaded and W + V Set.ize < U 5 and VSet.ize < VSet.maxize 6 then Create VS.id h(x + VSet.ize ) Fig. 6. Proportion Adjut algorithm. Becaue everal improvement to thi baic namepace balancing concept have been propoed (ee Section VII), the log(n) VS algorithm provide a baeline to ugget how thi type of algorithm can be expected to perform under condition of heterogeneity and kew in particular. B. Proportion Proportion target heterogeneity primarily, not workload kew. An adminitrator initially configure a node with a number of VS in proportion to it capacity. In addition, previouly oberved workload may be taken into account. After thi initial tep, each node add or hed load without any communication with other node. It wa firt propoed by Dabek et al. []. After etup, each node periodically follow Proportion- Adjut, hown in Figure 5 and 6. A node running Proportion independently create and detroy virtual erver. If a node i overloaded and i running more than one virtual erver, it elect the leat loaded VS that will make it underloaded and delete it (line -3). If a node i underloaded and believe that adding a VS will not put it over it target load, it create a virtual erver (line 4-6). Without any extra communication, underloaded node actively take on more work. The goal of the algorithm i that, over time, thi will eae the burden on overloaded node becaue they will aume a maller percentage of the workload a the number of VS increae. Load balancing in complete iolation ha it drawback. Firt, a node with only a few VS may not be able to form a good etimate of what the cot of creating a new one will be. Second, a meager machine till might be overloaded even if it i only running one VS. If a new phyical erver enter and ha ignificantly le capacity than the current low-end erver, the ytem may take a long time to adjut to thi new lowet common denominator. Third, if an overloaded node delete one of it VS, thi may overload it neighbor, reulting in cacade Fig. 7. Tranfer: Overloaded node attempt to tranfer virtual erver to underloaded node. If they only have one VS and are till overloaded, they plit the VS in two equal halve (and tranfer one). TRANSFER() if!overloaded then return 3 if VSet.ize > 4 then Contact node n at random 5 Chooe v VSet uch that: 6 (a) Tranferring v to n will not overload n 7 (b) v i the leat loaded virtual erver 8 that will halt overload; 9 Failing that, let v be mot loaded VS ele v VSet [0] Create VS.id v.id + dit(pred(v),v) mod D TRANSFER Fig. 8. Tranfer Split and Tranfer algorithm. of delete. Finally, when the ytem i underloaded, Proportion can caue all node to create their maximum number of VS, greatly increaing tate, routing hop, and churn. C. Tranfer Tranfer focue on actively unloading overloaded node. Intead of having underloaded node take on more work in iolation like Proportion, overloaded node following the Tranfer algorithm actively eek out underloaded node to inquire about load tranfer. Thu, node elect arbitrary ID at two point: when they plit and when they receive tranfer. Thi idea wa firt propoed by Rao et al. [35]. The algorithm work a hown in Figure 7 and 8. If a node i overloaded and it ha only one VS, then it plit the VS into two equal part (line ). If a node i overloaded and if it ha more than one VS (one of which may have jut been created via a plit), it attempt to contact an underloaded node and tranfer an appropriate VS (line 3-9). The tranfer fail if all VS would overload the potential receiver. Tranfer move work around effectively. Node are never tranferred work they cannot handle. However, when the ytem i near capacity, overloaded node may need to contact many other to perform a ucceful tranfer. Tranfer ha a few permutation. The cheme preented here and ued in the experiment i known a one-to-one becaue one node contact a ingle other node per unit time. The ame work alo propoed one-to-many and many-to-many variation and found they utilized node imilarly. Godfrey et al. propoe a more complex variation where node randomly

6 THRESHOLD(v, t) v.level t log ρ ( v.util c ) if v.level t v.level t 3 then return 4 v adjacent neighbor with lowet level 5 if v.level t < v.level t 6 then ( ρ ) dit(pred(v), v) 7 if v = pred(v) 8 then pred(v).id pred(v).id + 9 ele v.id v.id ele /* find new predeceor */ S et of log(n) random VS Chooe S uch that: 3 (a) i the leat utilized 4 (b) w + w ucc() < U ucc() 5.id pred(v).id + dit(pred(v),v) mod D Fig. 9. Threhold load balancing algorithm. chooe one of a handful of well-known exchange point that periodically reallocate work []. D. Threhold Threhold focue on keeping all node utilization within a ratio ρ, a oppoed to between target overload and underload value like the other algorithm. It alo keep the number of VS to a minimum (one per node). Threhold allow the election of arbitrary ID in both it neighbor adjutment and VS relocation phae. We preent a modified verion of Ganean et al. algorithm [0]. We made two modification: (a) we ue utilization intead of workload becaue the original algorithm aume homogeneou capacitie and (b) node only initiate rebalancing when they increae in level. Each node ha exactly one VS whoe ID i initially choen at random. The rebalancing algorithm hown in Figure 9 i called by a node with VS v at time t. Node et their current utilization level uch that a level increae by one if work ha increaed by a factor ρ, where c i ome mall contant (line ). If a node level ha increaed, it tart load balancing (line ). It firt attempt to make adjutment with it neighbor (line 4-9). VS v firt ee if local adjutment in the ID of it ucceor or predeceor are feaible, potentially hifting ome work to them. If the predeceor i lightly loaded compared to v, it ID i hifted toward v (line 8). Thi action hould reult in it taking ome of v load. v can alo move it own ID cloer to it predeceor, which potentially hift work from v to it ucceor (line 9). If making neighbor adjutment fail, it relocate a lightly loaded node to be it new predeceor (line -5). Tie between ucceor and predeceor are broken arbitrarily. If neither of thee option i available, v attempt to find a new predeceor to take (ideally) half of v load. v pick a et of VS at random and relocate the mot underutilized whoe departure will not overload it ucceor (line -5). Threhold diminihe the importance of the tuning param- Percentage of Node per Lifetime Quartile Fig Firt Quartile Second Quartile Third Quartile Fourth Quartile 0,000,000 Bandwidth (Kbp, log cale) 0,000,000,000 CDF of Downtream Bandwidth per Average Lifetime Quartile. eter δ, but introduce a ignificant parameter in ρ. If ρ i too large, load balancing will occur too lowly. If it i too mall, node will make many unneceary adjutment. A compromie i to et ρ for low adjutment but to induce load balancing if the node become overloaded even if level have not changed. We included thi compromie in our implementation. Becaue Threhold alway chooe the leat utilized VS to relocate, VS with very high capacity (and therefore low utilization) may tend to be relocated frequently. V. SIMULATOR We built a imulator to compare the load balancing algorithm dicued in Section III and IV. While imulator exit for everal pp algorithm, none upport virtual erver or drop packet under overload [6], []. Thi ection decribe the imulator and how querie ucceed and fail. The imulator operate in dicrete time tep. Each time tep conit of the following phae: node arrival and departure, routing table update, querie, and load balancing. Node arrival and departure. At each tep, node arrive and depart. A typical method for generating birth/death procee i to aume Poion ditributed lifetime (and deathtime) with ome mean λ [8], [9], [33]. However, Butamente et al. have found, through trace analyi of Gnutella, that pp ytem do not follow thi memory-le ditribution and, in fact, approximate longer-tailed Pareto ditribution more cloely [3]. For our trace-baed experiment, we ue a Gnutella trace directly [40]. Becaue we wanted to include the correlation between node lifetime and their capacitie, we extracted from the trace the node for which uptream or downtream bandwidth were available. The extracted trace conit of 5508 node joining and leaving the Gnutella network for 60 hour. We baed churn on the time when the IP addree of the node could be reached in the trace. The median lifetime of a node wa about one hour. We converted from the trace bandwidth information to meage per econd by auming an average meage ize of KB. The median node could forward 9 meage per econd. We how the bandwidth ditribution and modet correlation between bandwidth and lifetime in Figure. The trace doe not include any topology information, and we do not include any in our imulation. For the experiment where we vary node lifetime, we intead generated everal Pareto birth/death ditribution with varying

7 mean. Becaue Pareto ditribution can take a long time to tabilize, we only took a naphot of the ditribution after thi tabilization had occurred. We ued α = and varied β, avoiding intabilitie with maller value of α []. One unnatural apect of both the ynthetic and trace-driven churn i a large number of birth at the beginning of each experiment. Becaue each algorithm need ome workload information to operate, they did not activate until a hort period into each experiment. We chooe an activation time of 400 econd, a thi wa when all of the node in the Gnutella trace had firt joined. In addition, we recorded tatitic only for the econd half of each experiment. Routing Table Update. New VS tart off with an empty routing table. They follow the Chord mechanim to find a node to fill each of their log(n) lot [43]. Each node with ID a fill it i th entry with the node whoe ID i the ucceor to a + i mod D. Each routing table entry, or finger, ha a timeout et to 30 econd on average. Each time thi finger i ued uccefully, the timeout i reet. Thi imple technique typically ha been found to be effective in upreing maintenance meage [7]. Node do not invalidate their finger on a failed attempt at forwarding becaue they do not know if the receiver i dead or overloaded. When node gracefully change their VS identifier, other virtual erver pointing to them are notified. When node die, VS pointing to them are not notified (i.e., death i ungraceful), a would be the cae were a uer to witch off hi or her machine. Node make certain their ucceor finger are alway valid. Querie. Querie initiate from node uniformly at random with detination choen from either uniform or Zipf ditribution, depending on the experiment. Each hop in the query ue the appropriate finger to route toward the detination. Each ue of a VS for routing or maintenance add one unit of load per that VS node. If a hop i to a node whoe load for that unit of time matche or exceed it capacity, the query fail. Querie ucceed when they reach their detination. Load Balancing. Node check on their load balance once every 30 econd on average. They determine their utilization by examining an exponentially-weighted moving average of the work their VS perform. They check if they are above or below their target, which were et to.95 and.05 capacity, repectively. If they are out-of-balance, they perform whichever reactive algorithm i currently under tet. VSet.maxize wa et to 8 a uggeted by the Chord reearch group. Each node running Tranfer began with five VS a uggeted by Rao et al. [35]. VI. RESULTS The following ummarize our experimental reult: In Section VI-A, we how that imple namepace balancing i effective when workload are uniform and node capacitie are a contant (it aumed condition). We portray the diminihed value in thi form of balancing a workload become kewed. Utilization Leat-quare fit line Namepace per node Fig.. There exit a trong correlation between a node namepace and it utilization when the workload i uniform and capacitie are contant. TABLE I WORKLOAD SKEW UNDER NAMESPACE BALANCING Skew Random Balanced r Succ. % r Succ. % Uniform > Zipf (α = ) Zipf (α =.) Zipf (α =.4) In Section VI-B, we explore parameter choice for κ for k-choice and find that κ = 8 perform well for the workload we examine. In Section VI-C, we compare how the algorithm repond to varying applied workload when node follow tracebaed churn and capacity. We find that only k-choice and Tranfer can upport large amount of kewed load. We how that k-choice can upport high churn rate in Section VI-D. Section VI-E portray that k-choice utain high ucce rate throughout hifting workload. We alo find that Tranfer, Threhold, Proportion exhibit inconitent reult over time. In Section VI-F, we how that none of the algorithm can upport very kewed workload (e.g., α = 4.8) and that they increae in variance a kew increae. In Section VI-G, we find our implementation of k- Choice within Patry [38] improve throughput by 0% on an implementation in a heterogeneou-bandwidth networked environment. More information on the experiment, imulator, tuning, and validation i available in the accompanying technical report [7]. A. Namepace Balancing Thee firt experiment confirm that, under condition of contant or near-contant capacity and uniform query ditribution, imple namepace balancing i highly effective. However, when either of thee condition fail to hold, it i not. In order to ee the correlation between a node namepace and it utilization, we ran a imple et of experiment in which we varied workload kew in a ytem that wa performing no load balancing. We monitored the incoming routing and maintenance meage for each node and compared thi to

8 95th Percentile Utilization 95th Percentile Utilization No LB No LB Uniform: Number of ID Sampled (κ) Zipf: Number of ID Sampled (κ) Paive Active 8 Paive Active Pct. Succefully Routed Meage Pct. Succefully Routed Meage.0.0 k-choice Tranfer Proportional Threhold No LB Uniform: Querie per node Zipf: Querie per node 0 0 Fig.. k-choice 95 th percentile utilization decreae a κ increae. Fig. 3. Percent of uccefully routed querie for trace-driven imulation with varying load. the fraction of the ID pace for which that node wa ued a a hop or detination. We ran two et of experiment: one where VS identifier were choen at random and a econd where they were et offline to be exactly equal. Thi econd cae how the bet that namepace balancing could achieve. We ued 4096 node and et all node capacitie o that they could route 0 meage per econd. No churn wa ued becaue the exactly equal ID computation i only performed offline. We varied workload kew from uniform to Zipf with α =.4. Becaue no active algorithm wa ued and there wa no churn, each experiment tabilized immediately. Every node had one virtual erver and there were querie per econd ( querie per alive node). We plot the correlation between namepace and node utilization for a uniform workload in Figure. A i expected, the average namepace per node i Becaue no load balancing i ued, the ditribution of namepace i long-tailed. Analytically, the larget ditance between two VS hould be 4096 log(4096).009, cloe to the meaured value of.005. Utilization with random ID ranged from almot 0 to about 4. In contrat, the cae where the namepace were completely balanced yielded an extremely mall range of utilization from 0.55 to A we relax the aumption that workload are uniform, the benefit in perfectly uniform addre pace decline. Table I how how the correlation and ucce rate for querie decline a workload kew increae. Separate experiment confirm a imilar decline a heterogeneity in node capacitie change from a contant. We can conclude from thi that, in order to achieve reaonable performance, a load balancing algorithm mut include ome workload parameter and cannot aim for addre pace balancing alone. B. Varying κ The econd et of experiment explore k-choice parameter for Gnutella-like ytem. Our goal wa to find a reaonable et of parameter for the ubequent experiment. We generated a ynthetic churn trace of 4k node with Pareto ditributed average lifetime of 60 minute and a Gnutella-like capacity ditribution with average capacity of 0 meage/econd. Each node initiated querie/econd. We ran each experiment for three hour and monitored node utilization. We varied κ and ran k-choice in active and paive mode. The 95 th percentile utilization are plotted in Figure. When κ =, k-choice i not in ue, howing the ituation without any load balancing. The reult how that active k- Choice lower utilization at a ignificantly fater rate than paive doe a κ increae. In both lookup cenario, the 95 th percentile utilization do not decreae much beyond when κ = 8 in active mode. The reult alo how that a kewed query ditribution (α =.) ha minimal impact on utilization for k-choice. In fact, it even lower peak utilization a node with more bandwidth are able to poition their VS where the workload i concentrated. A noted above, there are ubtantial drawback to large number of VS per node and to etting κ to a large value (e.g., large number of probe). Therefore, we ued κ = 8 in ubequent experiment, unle otherwie noted. A thee reult portend, preliminary experiment with Optimal ID choice ugget that k-choice work well without a huge ampling of ID. We alo experimented with value for ɛ, which we et to 5 in our experiment. Thee reult how that k-choice need only a mall number of choice to produce a ubtantial decreae in node utilization. We ran imilar experiment to find good parameter for Threhold. It two parameter τ and c were et to 8 and repectively. C. Trace Reult Our third experiment examine how the load balancing algorithm reponded to different degree of applied workload

9 Pct. Succefully Routed Meage Pct. Succefully Routed Meage Fig min 5 min 30 min hr k-choice Tranfer Proportional Threhold No LB hr Uniform: Avg. Node Lifetime (log cale) 30 min hr hr Zipf: Avg. Node Lifetime (log cale) Percentage of uccefully routed querie for varying rate of churn. uing trace-driven churn and capacity. In almot all cae, we found k-choice performed the ame a or better than the other algorithm. Each experiment ued the Gnutella trace a decribed in Section V. Each ran for twelve hour with tatitic recorded for the econd half of the experiment. We varied the applied query load by order-of-magnitude and recorded the percentage of querie that reached their detination. Thi experiment capture factor uch a artificial churn and large number of VS per node that ome of the algorithm induce. We plot the reult in Figure 3. They how that all of the algorithm, except for Threhold, can utain high ucce rate when querie are uniform, although k-choice and Tranfer do lightly better Proportion. At 0 querie/node, the 95 th percentile of the number of VS/node wa 8 for Proportion (the maximum), compared to.9 for k-choice and 6. for Tranfer. Performance for k-choice in paive mode decline after querie/node. We plot κ = 6; κ = 8 performed about % wore and κ = 64 performed about % better at thi workload level. When querie are kewed (α =.), only k-choice and Tranfer can utain high query rate. At thi level, the other algorithm are unable to maintain low utilization of low capacity node. Log(N) VS performed wore than No Load Balancing in thee experiment and i not hown in the figure. D. Varying Churn Becaue k-choice help node make good load balancing choice proactively, we hypotheized that at high churn rate, it would offer better performance than the other algorithm. To tet thi, we created a et of ynthetic churn trace with varying average lifetime and ued the ame capacity ditribution from the trace. We ran each algorithm with each node initiating querie per econd on average. 4hr 4hr We plot the reult from uniform and kewed (α =.) query ditribution in Figure 4. The data confirm our hypothei that k-choice adapt well to rate of high churn. We found that both Tranfer and Proportion were able to utain high natural churn rate for uniform querie, but that they induced..5 and 5 more artificial churn, repectively, than k-choice. Again, the variation in ucce rate i more prominent with kewed querie. Thi i becaue k-choice monitor workload before inertion. No Load Balancing improve lightly a lifetime increae becaue finger remain valid for longer. Again log(n) VS had wore performance than No Load Balancing. In both uniform and Zipf, Threhold ucce rate decline a node lifetime increae. Thi occur becaue Threhold make the gap between VS o non-uniform that it ignificantly increae the average number of hop, e.g., from 5.6 for 5 minute lifetime to 7.3 for 4 hour lifetime for uniform querie. Becaue querie are taking more hop and node are imilarly load balanced, each query i le likely to ucceed. E. Workload Shift For the fifth imulation experiment, we wanted to ee how the algorithm reponded to workload hift. We ran each algorithm uing trace-driven churn and capacity for ten hour. Halfway through each run, we changed the query detination from one moderately kewed et to another (both with α =.). We recorded tatitic throughout the trace. A noted, each algorithm activate after 400 econd. Each node initiated querie per econd on average. We monitored ucce rate and VS activity. VS activity capture the amount of tate tranfer that occur due to natural and artificial churn. When a node enter or leave the ytem, the number of VS action equal the number of VS in ue. Creating or detroying a VS i alo a VS action. Each k- Choice and Threhold relocate i two VS action; each tranfer i one. Conervatively, we did not include Threhold neighbor-adjutment or Tranfer plit a VS action. The reult are plotted in Figure 5. We how the ucce rate on the left y-axi and VS activity on the right y-axi. The reult how that active k-choice utain > 75% ucce rate, recovering immediately after the workload hift. Paive k-choice (not hown) gradually plateau at about 40%. We found that in ytem with higher rate of churn, paive reached equilibrium more quickly. A oon a active k-choice i activated, the ucce rate dramatically increae. With current tuning, however, active produce an order-ofmagnitude more VS activity than paive. After the hift, k- Choice active ettle to a lightly lower ucce rate becaue querie heading to the new highet ranked pot take lightly more hop per average query: a change from 6.8 to 7.3. Proportion, Tranfer, and Threhold all portray greater variance in ucce rate than k-choice. Proportion exhibit the greatet average VS activity and ha the larget average hop count at.5 hop per ucceful query. The performance of Threhold teadily decline a it gap become tightly

10 Pct. Succefully Routed Meage Pct. Succefully Routed Meage Pct. Succefully Routed Meage Pct. Succefully Routed Meage Fig k-choice (active): Time (hour) Tranfer: Time (hour) 4 5 Proportional: Time (hour) 4 5 Threhold: Time (hour) Plot of ucce rate and VS activity during a workload hift. clutered. That Tranfer tabilize at different level had two caue. Firt, a burt of birth oon after the hift caued more accurate finger than average and a burt of death at 6 hour caued the decline becaue many finger became invalid. Second, after the hift, the average path to highet ranked detination wa fewer hop than before. Although to a leer extent than Threhold, Tranfer hop count teadily rie a node move to arbitrarily compreed location. F. Varying Skew Some workload are heavily kewed and everal of the algorithm were able to upport up to α =.. We wanted to examine how much kew they might upport. To tet thi, we ued the ynthetic 60 minute average lifetime trace and the capacity ditribution from the trace a we varied α. A before, we ran each algorithm with each node initiating querie per econd on average. We plot the reult in Figure 6. They how that none of the algorithm can upport an extremely kewed workload, e.g., one where the top detination i almot 5 that of the next rank. Not only do the algorithm decline in their average ucce rate, but they alo all become le table. For example, the tandard deviation of ucce rate ampled over time for k-choice at uniform i 0% and at α = 4.8 it i 8%. To ee if increaing κ had an impact at high kew, we ran k-choice with κ = 6. We found that it performed better (at 4%) than κ = 8, but alo exhibited high variance. VS Activity (log) VS Activity (log) VS Activity (log) VS Activity (log) Pct. Succefully Routed Meage Fig Unif.. Skew: Zipf alpha k-choice Tranfer Proportional Threhold No LB Percentage of uccefully routed querie for varying rate of kew. TABLE II SUCCESSFUL QUERIES FOR EMULAB EXPERIMENT G. Emulab Experiment.4 Completed querie BW (MB/) No LB k-choice (+35%) (+%) (+3%) (+%) All (+0%) To examine k-choice effect on a working ytem, we implemented it within Patry and ran a query-and-download cenario. Our primary goal wa to meaure change in throughput with k-choice uing a fairly large real topology. Our ue of nearet-neighbor-baed Patry demontrate that k- Choice generalize beyond Chord emantic. We baed our k- Choice implementation on FreePatry [5]. We ran k-choice in paive mode with κ = 6. We ued VS per node becaue FreePatry doe not currently upport multiple VS. We were required to anticipate load baed on namepace ditance becaue low bandwidth node were unable to uccefully join the network when querie were already taking place. For thi ame reaon, querie were only for uniformly ditributed detination. If the detination reponded, each node attempted to download an 8KB block. A query completed if both the query and download were ucceful. We ran our experiment on Emulab, a tetbed for networking reearch that upport precie bandwidth tuning [44]. The topology conited of 56 node. There were 64 node of each bandwidth level; the level were 40Mb/, 4Mb/, Mb/, and Mb/. Although Emulab ha been working on making their ytem more calable to upport larger experiment, at the time, thi wa the larget topology we could run. Table II how the total number of querie completed by bandwidth type. Each value i averaged over two trial that conited of one hour of querie. All node ued one of the 40MB/ node a their boottrap. A a reult, they were frequently in other node routing table and had a higher meage routing workload. Thi i why their completed querie are fewer than the 4MB/ node. A expected, the average number of hop wa a jut le than, with minimal variance. The main experimental reult, however, i that a 0% improvement in throughput confirm that k-choice can have a ubtantial 4.8

11 poitive impact on performance in a heterogeneou topology while retaining the important ecurity propertie of verifiable ID. VII. RELATED WORK Object load balancing. We have oriented our examination of load balancing around routing, where a node requet mut reach the detination ID for it to be ucceful. If the network i intead being ued for torage, other load balancing technique can be applied. Byer et al. decribe a technique that hahe data to be tored uing two ditinct hah function, providing two potential location [4]. The le loaded of the two poibilitie i choen. During data lookup, the query mut contact both poible torage location, or appropriate forwarding pointer mut be ued. Under uniform workload and capacity aumption and with no churn, they have recently generalized thi reult to how that the maximum load at any erver i loglog(n)/log(d) + O() where d i the number of choice [5]. Their method i an example of the power of two choice [3]. Our ID election proce i imilar in pirit, in that we alo ue multiple hah function, although here we do o to provide VS with a menu of identifier. Object may be cached in the network to reduce hot pot or overload. Rouopoulo and Baker develop a cooperative requet cheme where node direct requet toward the highet capacity replica [37]. They aume that the ource of each lookup i aware of the capacity of each poible replica holder. Source of requet learn the replica by firt contacting the root of the query, a key primary torage node, o it mut till perform ome work for their method to function. Thee torage-oriented load balancing technique are orthogonal and complementary to the method examined in thi paper, including k-choice. For example, k-choice reduce an overburdened node namepace, preventing it from being contacted in the firt place, and Rouopoulo technique prevent it from being contacted frequently after the replica et i known. Namepace balancing. While the imple log(n) VS per node achieve O(/N) namepace balance per node, more recent algorithm have achieved tighter bound with fewer virtual erver. Thee algorithm are baed on the aumption that the capacity of node and workload are uniform; they do not include any workload caling parameter. Becaue of thee factor, they would approximate the behavior and reult of the log(n) VS algorithm. If they did achieve perfect namepace balancing at zero cot, they could be expected to perform a Balanced doe in Table I. Four algorithm fall into thi category. Firt, Karger and Ruhl propoe that each node ha log(n) potential ID, only one of which i activated at once [6]. Node activate and deactivate their VS to balance the ditance between themelve and their ucceor. Becaue thi algorithm allocate node a limited number of ID, it ha tronger ecurity propertie than the remainder of thi group. Second, Manku algorithm reduce the ratio of the larget to the mallet partition to at mot 4 w.h.p. and ha low arrival and departure cot [30]. Third and fourth, Adler [] and Naor [3] alo have low cot algorithm to achieve namepace balancing baed on unlimited virtual erver movement. Both algorithm depend on the hitory of node ID that each node ha ued and their analye are given only for the inertion, not deletion, cae. Range querie. While we have examined uniform and Zipf query ditribution in our imulation, we have not examined load balancing algorithm targeted at pp ytem when performing range querie are common. However, if one conider uing a pp ytem more like a typical databae where each node i analagou to a dik, it i clear that ordering data by key might be warranted. We are aware of two load balancing algorithm that are targeted for thi new domain [0], [6]. We evaluated Ganean Threhold in thi paper. Both require unlimited ID election and, therefore, uffer from Sybil attack liabilitie, making them unuitable for non-cooperative environment. However, it i unlikely that a load balancing technique for range querie exit that upport calable ecure ID. VIII. CONCLUSIONS We introduced a novel anticipatory load matching algorithm for balancing load in peer-to-peer network. Thi algorithm make explicit the workload aignment problem that previou work attempted to olve implicitly. The algorithm work preemptively a the node i joining to hift the right amount of work to the joining node. Optionally, it can continue to readjut workload mimatch over time. After examining the k-choice algorithm independently, we benchmarked it performance and that of other load balancing algorithm for tructured overlay under condition of node heterogeneity, kew, churn, and workload hift uing tracebaed imulation. Prior work on load balancing for pp ytem ha either focued on namepace balancing or on ytem with more heterogeneou characteritic. We howed that even perfect namepace balancing reult in poor performance under realitic condition. Prior algorithm that do work well under thee condition, Tranfer and Threhold, both allow the election of arbitrary ID, everely circumcribing their utility on inecure network. We have hown that k-choice can provide good load balancing under realitic condition while retaining trong ecurity propertie neceary for wide-area application. ACKNOWLEDGMENT The author would like to thank Miguel Catro, Antony Rowtron, and Michael Mitzenmacher for helpful dicuion. REFERENCES [] M. Adler, E. Halperin, R. Karp, and V. Vazirani. A Stochatic Proce on the Hypercube with Application to Peer-to-Peer Network. In STOC 003, San Diego, CA, June 003. [] C. Blake and R. Rodrigue. High Availability, Scalable Storage, Dynamic Peer Network: Pick Two. In Proceeding of HotOS IX, Lihue, HI, May 003. [3] F. Butamante and Y. Qiao. Friendhip that lat: Peer lifepan and it role in PP protocol. In Eighth International Workhop on Web Content Caching and Ditribution, Hawthorne, NY, October 003.

12 [4] J. Byer, J. Conidine, and M. Mitzenmacher. Simple Load Balancing for Ditributed Hah Table. In Proceeding of the Second International Workhop on Peer-to-Peer Sytem (IPTPS 03), Berkeley, CA, February 003. [5] J. Byer, J. Conidine, and M. Mitzenmacher. Geometric Generalization of the Power of Two Choice. In SPAA 004, Barcelona, Spain, June 004. [6] M. Catro, M. Cota, A. Kermarrec, and A. Rowtron. SimPatry. antr. [7] M. Catro, M. Cota, and A. Rowtron. Performance and Dependability of Structured Peer-to-Peer Overlay. In Dependable Sytem and Network, Florence, Italy, June 004. [8] M. Catro, P. Druchel, A. Kermarrec, A. Nandi, A. Rowtron, and A. Singh. SplitStream: High-Bandwidth Multicat in Cooperative Environment. In Proceeding of the 9th ACM SOSP, Bolton Landing, NY, October 003. [9] M. Catro, P. Druhel, A. Ganeh, A. Rowtron, and D. Wallach. Secure Routing for Structured Peer-to-Peer Overlay Network. In OSDI 0, Boton, MA, 00. [] L. Cox and B. Noble. Samara: Honor Among Thieve in Peer-to-Peer Storage. In Proceeding of the 9th ACM SOSP, Bolton Landing, NY, October 003. [] M. E. Crovella and L. Lipky. Long-lating tranient condition in imulation with heavy-tailed workload. In Proceeding of the 997 Winter Simulation Conference, December 997. [] F. Dabek, M. F. Kaahoek, D. Karger, R. Morri, and I. Stoica. Widearea cooperative torage with CFS. In Proceeding of the 8th ACM Sympoium on Operating Sytem Principle (SOSP 0), October 00. [3] R. Dingledine, N. Mathewon, and P. Syveron. Reputation in PP Anonymity Sytem. In Workhop on Economic of Peer-to-Peer Sytem, Berkeley, CA, June 003. [4] J. Douceur. The Sybil Attack. In Proceeding of the Firt International Workhop on Peer-to-Peer Sytem (IPTPS 0), Cambridge, MA, March 00. [5] P. Druchel, R. Gil, Y. Hu, S. Iyer, A. Ladd, A. Milove, A. Nandi, A. Pot, C. Rei, A. Singh, and R. Zhang. Rice FreePatry implementation. FreePatry. [6] P. Druchel and A. Rowtron. PAST: A large-cale, peritent peer-topeer torage utility. In Proceeding of the 8th ACM Sympoium on Operating Sytem Principle (SOSP 0), October 00. [7] D. Ellard, J. Ledlie, P. Malkani, and M. Seltzer. Paive NFS tracing of Workload. In Proceeding of the 003 USENIX Conference on File and Storage Technology, San Francico, CA, March 003. [8] L. Fan, P. Cao, J. Almeida, and A. Z. Broder. Summary cache: a calable wide-area web cache haring protocol. IEEE/ACM Tranaction on Networking, 000. [9] J. Feigenbaum and S. Shenker. Ditributed Algorithmic Mechanim Deign: Recent Reult and Future Direction. In Sixth International Workhop on Dicrete Algorithm and Method for Mobile Computing and Communication, September 00. [0] P. Ganean, M. Bawa, and H. Garcia-Molina. Online Balancing of Range-Partitioned Data with Application to Peer-to-Peer Sytem. In VLDB 004, Toronto, September 004. [] T. Gil, F. Kaahoek, J. Li, R. Morri, and J. Stribling. ppim. http: // [] B. Godfrey, K. Lakhminarayanan, S. Surana, R. Karp, and I. Stoica. Load Balancing in Dynamic Structured PP Sytem. In INFOCOM 004, Hong Kong, March 004. [3] B. Gro and A. Acquiti. Balance of Power on ebay: Peer or Unequal? In Workhop on Economic of Peer-to-Peer Sytem, Berkeley, CA, June 003. [4] K. Gummadi, R. Dunn, S. Saroiu, S. Gribble, H. Levy, and J. Zahorjan. Meaurement, Modeling, and Analyi of a Peer-to-Peer File-Sharing Workload. In Proceeding of the 9th ACM SOSP, Bolton Landing, NY, October 003. [5] D. Karger, E. Lehman, T. Leighton, M. Levine, D. Lewin, and R. Panigrahy. Conitent Hahing and Random Tree: Ditributed Caching Protocol for Relieving Hot Spot on the World Wide Web. In ACM Sympoium on Theory of Computing, El Pao, TX, May 997. [6] D. Karger and M. Ruhl. Simple Efficient Load Balancing Algorithm for Peer-to-Peer Sytem. In SPAA 004, Barcelona, Spain, June 004. [7] J. Ledlie and M. Seltzer. Ditributed, ecure load balancing with kew, heterogeneity, and churn. Technical Report TR-3-04, Harvard Univerity, December 004. [8] J. Ledlie, J. Taylor, L. Serban, and M. Seltzer. Self-organization in peer-to-peer ytem. In Tenth SIGOPS European Workhop, September 00. [9] D. Liben-Nowell, H. Balakrihnan, and D. Karger. Analyi of the Evolution of Peer-to-Peer Network. In PODC 00, July 00. [30] G. S. Manku. Balanced Binary Tree for ID Management and Load Balance in Ditributed Hah Table. In PODC 004, St. John, Newfoundland, Canada, July 004. [3] M. Mitzenmacher, A. Richa, and R. Sitaraman. The Power of Two Choice: A Survey of Technique and Reult. Kluwer Academic Publiher, Norwell, MA, 00. [3] M. Naor and U. Wieder. Novel Architecture for PP Application: the Continuou-Dicrete Approach. In Fifteenth ACM Sympoium on Parallelim in Algorithm and Architecture, San Diego, CA, June 003. [33] G. Pandurangan, P. Raghavan, and E. Upfal. Building low-diameter pp network. In IEEE Sympoium on Foundation of Computer Science, page , 00. [34] P. Pietzuch and S. Bhola. Congetion Control in a Reliable Scalable Meage-Oriented Middleware. In Middleware 003, Rio de Janeiro, Brazil, June 003. [35] A. Rao, K. Lakhminarayanan, S. Surana, R. Karp, and I. Stoica. Load Balancing in Structured PP Sytem. In Proceeding of the Second International Workhop on Peer-to-Peer Sytem (IPTPS 03), Berkeley, CA, February 003. [36] S. Ratnaamy, P. Franci, M. Handley, R. Karp, and S. Shenker. A Scalable Content-Addreable Network. In Proceeding of the ACM SIGCOMM 0 Conference, Augut 00. [37] M. Rouopoulo and M. Baker. Practical Load Balancing for Content Requet in Peer-to-Peer Network. Reearch Report c.ni/00903, Stanford Univerity, January 003. [38] A. Rowtron and P. Druchel. Patry: Scalable, decentralized object location, and routing for large-cale peer-to-peer ytem. In Middleware, November 00. [39] S. Saroiu, K. P. Gummadi, R. J. Dunn, S. D. Gribble,, and H. M. Levy. An analyi of internet content delivery ytem. In Proceeding of 5th Sympoium on Operating Sytem Deign and Implementation (OSDI 0), Boton, MA, December 00. [40] S. Saroiu, P. K. Gummadi, and S. D. Gribble. A meaurement tudy of peer-to-peer file haring ytem. In Proceeding of the Multimedia Computing and Networking (MMCN), San Joe, CA, January 00. [4] J. Shneidman and D. Parke. Specification Faithfulne in Network with Rational Node. In PODC 004, July 004. [4] I. Stoica, D. Adkin, S. Zhuang, S. Shenker, and S. Surana. Internet Indirection Infratructure. In SIGCOMM 0, Pittburg, PA, Augut 00. [43] I. Stoica, R. Morri, D. Karger, M. F. Kaahoek, and H. Balakrihnan. Chord: A calable peer-to-peer lookup ervice for internet application. In Proceeding of the ACM SIGCOMM 0 Conference, Augut 00. [44] B. White, J. Lepreau, L. Stoller, R. Ricci, S. Gurupraad, M. Newbold, M. Hibler, C. Barb,, and A. Joglekar. An integrated experimental environment for ditributed ytem and network. In Proceeding of 5th Sympoium on Operating Sytem Deign and Implementation (OSDI 0), Boton, MA, December 00. [45] B. Wilcox-O Hearn. Experience deploying a large-cale emergent network. In Proceeding of the Firt International Workhop on Peerto-Peer Sytem (IPTPS 0), Cambridge, MA, March 00. [46] B. Yan and H. Garcia-Molina. PPay: Micropayment for Peer to Peer Sytem. In ACM Conference on Computer and Communication Security, Wahington, D.C., October 003. [47] B. Zhao, J. Kubiatowicz, and A. Joeph. Tapetry: An infratructure for fault-tolerant wide-area location and routing. UCB/CSD-0-4, U.C. Berkeley, April 00. Reearch Report