Network delay-aware load balancing in selfish and cooperative distributed systems

Transcription

1 Network delay-aware load balancing in selfish and cooperative distributed systes Piotr Skowron Faculty of Matheatics, Inforatics and Mechanics University of Warsaw Eail: Krzysztof Rzadca Faculty of Matheatics, Inforatics and Mechanics University of Warsaw Eail: Abstract We consider a geographically distributed request processing syste coposed of various organizations and their servers connected by the Internet. The latency a user observes is a su of counication delays and the tie needed to handle the request on a server. The handling tie depends on the server congestion, i.e. the total nuber of requests a server ust handle. We analyze the proble of balancing the load in a network of servers in order to iniize the total observed latency. We consider both cooperative and selfish organizations (each organization aiing to iniize the latency of the locally-produced requests). The proble can be generalized to the task scheduling in a distributed cloud; or to content delivery in an organizationally-distributed CDNs. In a cooperative network, we show that the proble is polynoially solvable. We also present a distributed algorith iteratively balancing the load. We show how to estiate the distance between the current solution and the optiu based on the aount of load exchanged by the algorith. During the experiental evaluation, we show that the distributed algorith is efficient, therefore it can be used in networks with dynaically changing loads. In a network of selfish organizations, we prove that the price of anarchy (the worst-case loss of perforance due to selfishness) is low when the network is hoogeneous and the servers are loaded (the request handling tie is high copared to the counication delay). After relaxing these assuptions, we assess the loss of perforance caused by the selfishness experientally, showing that it reains low. Our results indicate that a set of servers handling requests, connected in a heterogeneous network, can be can be efficiently anaged by a distributed algorith. Additionally, even if the network is organizationally distributed, with individual organizations optiizing perforance of their requests, the network reains efficient. I. INTRODUCTION One of the ost iportant aspects affecting the perceived quality of a web service is the delay in accessing the content. To avoid servers congestion, the content of the web pages is coonly replicated in ultiple locations. Additionally, in order to iniize the network latency, the replicas are placed close to users. Because the intensity of the web traffic changes dynaically, efficient irroring requires both expensive infrastructure and effective load balancing algoriths. As the result any organizations decide to handle the task of irroring their data to dedicated platfors content delivery networks (CDNs) [18], [29]. The CDNs have been very successful in the recent years: Akaai [25], [27], [34], the largest CDN, handles around 15-20% of the Internet traffic. Consider an apparently different distributed syste: a cloud of datacenters perforing coputationally-intensive parallel calculations. Each datacenter attepts to accelerate its calculations by distributing soe of its load to less loaded and faster datacenters. However, the datacenters in reote locations ust be avoided as the tie needed to transfer the input and the result ay doinate the processing tie. Routing the requests in a CDN and distributing the load in a cloud are strongly related probles. In both there are systes of servers connected by the network (for siplicity, in cloud, we refer to a single datacenter as a server, as we will not explore the parallelis inside a datacenter). The handling tie on a server depends both on its perforance etrics and its load. The final perceived latency coes fro the network delays (required for transitting the input data and the result) and fro the handling tie on the servers. Finally, in both cases every server has its initial load: in a CDN, the load is the current nuber of the data access requests to the server; in a cloud, the load is the nuber of initial tasks. We generalize these two probles to a load balancing of reote services. We assue that in the balanced syste, the handling tie of a single request on a server linearly depends on the total nuber of requests to be processed by the server. A linear dependency reflects a constant throughput of a server. In real systes, increasing the level of concurrency too uch ay overload the server decreasing its throughput (trashing). However, assuing that the aount of work in the syste as a whole is reasonable, there should be no overloaded servers in the balanced state. Siilar assuptions are usually taken in congestion gaes [14], [31] and in the queuing theory, where a linear dependency is expressed by Little s law. We assue that the transission duration of a single request does not depend on the nuber of sent requests. Although soe odels (e.g., routing gaes [26]) consider the cases when the bandwidth of a link ay becoe a bottleneck, we focus on a widespread network, in which there are ultiple cost-coparable routing paths between the servers. Thus, sending any data fro one server to another should not significantly increase the network delay between the. These assuptions are also justified by our experients in

2 Appendix we discuss how the intensity of the network load, generated between the servers, influences the RTT between the servers in PlanetLab environent. In other words, we consider that the load our syste iposes on the network is negligible: thus, the network delay is caused only by the latency (resulting fro e.g., geographical distribution). Our proble forulation assues the knowledge of such latencies; this is not a liitation because onitoring the pairwise latencies, which can change in tie, is a well studied proble with known solutions (e.g. see [11], [35]). Optiizing latency is iportant for instance when streaing video files: a large latency delays the start, and, in case of counication probles, can be perceived as breaks in transission. Balancing of the servers loads and finding the irroring iniizing network delays are analyzed in the literature, but usually separately (see Section VIII). Distributed systes should consider both counication and coputation. On one hand, clouds get geographically distributed, thus cannot ignore network latency. On the other, a CDN handling coplex, dynaically-created content of the odern web, can no longer ignore the load iposed by the requests. For delivering large static content, like ultiedia, soe currently used techniques cache the content at specially designated front-end servers. A particular server is chosen by the round-robin algorith. This approach which is inefficient as, for instance, unpopular files are cached in ultiple places. Benefits fro optiizing requests redirections can be significant. For instance, [12] proposes a caching schee that is both consistent (requests for the sae content are redirected to the sae front-end servers) and proportional (each server handles a desired proportion of requests). In this case, our algoriths can be viewed as a copleentary optiization technique to caching once the content ust be downloaded fro the back-end servers, we show how to efficiently distribute the download requests. In cloud coputing, our odel fits for instance processing streas of data in the real tie or when the data strea is continuously produced and too large to be processed offline. Consider a user interacting with a siulated virtual environent (e.g. [4]): user s actions are captured by caeras; their iage streas are analyzed in the real tie to build a 3D odel; then this odel interacts with the virtual world odel. Other applications include extracting statistics on users actions in the Internet; or iage analysis. In addition to a classic syste with a central anageent, we analyze an organizationally-distributed syste. Instead of a single, syste-wide goal (iniize the overall request handling tie), the organizations are selfishly interested only in optiizing the handling tie of their local requests. This odel reflects a CDN created as an agreeent between e.g., any ISPs; or a federation of clouds, each having a different owner. Because typically the load changes dynaically, with peaks of deand followed by long periods of low activity, individual organizations are otivated to enter such a syste: a peak can be offloaded, whereas handling foreign requests in the period of low activity is relatively inexpensive. The lack of central coordination in the organizationallydistributed syste increases the average processing tie. The price of anarchy [23] expresses the worst-case relative increase in the latency in coparison with relinquishing the control to a centrally-anaged organization (like Akaai s CDN). As the price of anarchy varies considerably between systes (fro relatively sall in congestion gaes to unbounded in selfish replication [32]), we were curious to check it in our syste. Our contribution is the following: (i) We show that the proble of network delay-aware load balancing can be stated as an optiization proble in the continuous doain; we proove that the optiization proble is convex and, in particular, polynoially solvable. This result also indicates that the local optiization techniques can be applied to the proble. (ii) We propose a distributed algorith that iteratively balances the servers load towards the optiu. We confir the algorith s efficiency through theoretical analysis and in siulations. (iii) We show how the proble can be extended and, in particlaur, how it can be applied to find the optial distribution of the data (even with their replicas) when the access pattern of the users is known (Section VII). (iv) In a network of selfish servers, after soe siplifying assuptions, we prove that the price of anarchy is low (1 + O(2cs/ )). The experients show that the loss of perforance caused by the selfishness reains low (below 1.15) also without these assuptions. II. MATHEMATICAL MODEL Organizations, servers, tasks The syste consists of a set of organizations, each owning a server (or a cluster) connected to the Internet. The servers are unifor; each server i has a constant processing speed s i. The i-th organization has its own load consisting of a large nuber n i of sall, individual tasks (or requests). The aount of load n i can be considered as a nuber of tasks at a particular tie oent (snapshot); or, alternatively, as a steady state rate of incoing requests in a syste continuously processing requests. A task corresponds to, e.g., in a coputational cloud, a unit-size coputation (e.g.: a single work unit in a BOINC-type application; or a single invocation of a apreduce function); or, in a CDN, a request for reote data coing fro a user assigned to server i (typically, a user would be assigned to the closest server). In the basic odel we assue that the sall tasks have the sae sizes (e.g. this corresponds to the divisible coputation load; or in a CDN to the case where the stored data chunks have constant sizes); thus the execution of the single request on the i-th server takes 1/s i tie units. In Section VII we show how to easily extend our results to the tasks of different sizes. 2

3 Relaying tasks, counication delays Each organization can relay soe of its own requests to other servers. If the request is relayed, the observed handling tie is increased by the counication latency on the link. We denote the counication latency between i-th and j-th server as c i j (with c ii = 0). Since counication delay of a single request does not depend on the aount of exchanged load (which is explained in Section I and which is confired by our experients on PlanetLab see Appendix) c i j is just a constant instead of a function of the network load. We assue that the routing in the syste is correct (optiized by the network layer). Thus, we will not consider optiizing counication tie by relaying requests fro i to j through a third server k (if c ik + c k j < c i j, the network layer would also discover the route i k j and update the routing table accordingly, so that c i j := c ik + c k j ). We assue that each request can be sent to and executed on any server. However, if we set soe of the counication delays to infinity, we restrict the basic odel to the case when each organization is allowed to relay its requests only to the given subset of the servers (its neighbors) which odels e.g. the trust relationship. Relay fractions, current loads We use a fractional odel in which a relay fraction ρ i j denotes the fraction of the i- th organization s own requests that are sent (relayed) to be executed at j-th server ( i, j ρ i j 0 and i j= j=1 ρ i j = 1). The load balancing proble is to find the appropriate values of the relay fractions (foralized in the further part of this Section). Once the fractions are known, each organization knows how any of its own requests it should send to each server; the tasks are sent and executed at the appropriate servers. The fractional odel ight be considered as a relaxation of a proble of handling non-divisible requests; in Section VII we show how to round the solution of a fractional odel to a discrete odel. Moreover, the fractional odel itself fits the divisible load odel used in the scheduling theory. For the sake of the clarity of the presentation we use the additional notation for the nuber of requests redirected fro server i to j r i j (thus r i j = n i ρ i j ), and for the current load of the server i, i.e. the nuber of requests relayed to i by all other organizations, including the organization owning the server itself l i (thus, l i = j=1 r ji). Copletion ties We don t assue any particular order of requests executed on a server. First, since the nuber of requests is large, considering any particular order on the servers would increase the coputational coplexity. Second, in a continuously running systes, we have no control over the order in which requests are produced (especially as they can be also delayed by the network); the usual FIFO policy results in an arbitrary order. Thus, for each of the l j request that are actually processed on j- th server, the expected processing tie of each request is equal to 1/l j l j 1 i/s i = l j /2s j (constant oitted for clarity). Since i-th organization relayed r i j requests to j, the expected total copletion tie of requests relayed by i to j is equal to r i j (l j /2s j +c i j ). The expected total processing tie C i of the i-th organization s own requests is a su over all servers j of the expected total copletion ties of the requests owned by i and relayed to j. C i = j=1 ( ) l j + c i j r i j = 2s j j=1 (( c i j + k=1 ρ k j n k 2s i )ρ i j n i ) (1) We consider the expected (or the average) processing tie, rather than the akespan of an organization for several reasons. The average processing tie is siilar to the widely-used su of processing ties criterion (ΣC i ). We assue that the workload of each organization is created by any users. ΣC i odels users perforance better than the akespan [17]. In all the contexts otivating our work fro Section I (e.g., processing streas of data in the real tie, delivering content to the users) we are focused on the average user perforance. Also, while C i depends on the vector ρ = [ρ kl ] quadratically, the relation between the akespan and ρ is just linear, which akes the proble considerably easier. Thus, we believe that soe of our results could be adapted for the cases when soe different fro the pointed applications of our odel would require optiizing the akespan. The total processing tie of all the requests in the syste is denoted as C i = i= i=1 C i. Proble forulation We consider two related probles. First, the goal is to find such a vector of the fractions, ρ, that the total processing tie of the requests, C i, is iniized. This goal corresponds to a centrally-anaged syste having a unique owner and a single goal. Second, we analyze the case when servers are a coon good, but each organization is selfishly iniizing the processing tie of its own requests. The i-th organization is responsible for sending its own requests to appropriate servers. In other words, the i-th organization adjusts the values of ρ i j in order to iniize C i. This approach is siilar to the selfish job odel [37], in which jobs selfishly choose processors to iniize their execution tie. Siilar agreeents exist in real-life systes: e.g., PlanetLab servers are treated as a coon good anaged by a central entity; PlanetLab users choose the servers they want to use for their experients. Also, in acadeic grids (e.g. Grid5000 in France), participating organizations grant control over their resources to a central entity; in return, users can subit their jobs to any resource. In this case, we look for such a vector of the fractions ρ for which the syste reaches the Nash equilibriu. By coparing the resulting C i with the result for the centrally-anaged syste, we will find the price of anarchy, quantifying the effect of selfishness on the total processing tie.. 3

4 III. OPTIMAL SOLUTION In this section we assue that there is a central processing unit that has the coplete knowledge about the whole syste. Given the counication latencies c i j and the organizations own loads n i, our goal is to find an algorith setting relay fractions ρ i j so that the total processing tie of all the requests C i is iniized. Below we show that the proble can be stated as a quadratic prograing proble with the positive-definite Q atrix. This eans that the optiization proble is convex and, in particular, it is polynoially solvable. We consider this result powerful, as it indicates that any local optiization techniques can be applied to the proble. In particular, in the next section we present a distributed algorith that, in practice, is very efficient. Theore 1. The proble of iniizing C i can be expressed as the quadratic prograing proble: C i = ρ T Qρ + b T ρ, with the positive-definite Q atrix. In particular, this eans that the iniization proble is convex. Proof: We express the total processing tie C i in a atrix for as C i = ρ T Qρ + b T ρ, where: ρ is a vector of relay fractions with eleents. ρ (i, j), the eleent at (i + j)-th position, denotes the fraction of local requests of i-th server that are relayed to j-th server ρ i j, thus: ρ = [ρ (1,1),ρ (1,2),...,ρ (1,),ρ (2,1),...,ρ (,) ] T ; Q is 2 -by- 2 atrix in which q (i, j),(k,l) denotes the eleent in (i + j)-th row and in (k + l)-th colun: n i n k /s j if j = l and i < k; q (i, j),(k,l) = n i n k /2s j if j = l and i = k; (2) 0 otherwise; Figure 1 presents the structure of atrix Q. b is a vector with 2 eleents with b i j denoting an eleent at (i + j)-th position: b (i, j) = c i j n i. The following derivation shows how the atrix Q is constructed: ρ T Qρ = i, j ρ (i, j) q (i, j),(k, j) ρ (k, j) (3) k i n i n k ρ (k, j) = ρ (i, j) ( + n2 i ρ (i, j)) (4) i, j k>i s j 2s j = i j k n i n k ρ (i, j) ρ (k, j) 2s j = i r i j l j. (5) j 2s j (3) follows fro the construction of the atrix Q (only eleents k i are non-zero). (4) substitutes q (i, j),(k,l) with the values defined in (2). (5) uses coutativity of ultiplication and substitutes l j = k n k ρ (k, j) and r i j = n i ρ (i, j). The constraints that ρ i j are the fractions ( i, j ρ i j 0 and i j= j=1 ρ i j = 1) can also be expressed in the atrix for. First, ρ 0 2, where 0 2 is a vector of length 2 consisting n i n k /s j 0. n i n k /2s j Figure 1: Matrix Q: denotes non-zero values Algorith 1: CALCBESTTRANSFER(i, j) input: (i, j) the identifiers of the two servers Data: k r ki initialized to the nuber of requests owned by k and relayed to i ( k r k j is defined analogously) Result: The new values of r ki and r k j foreach k do r ki r ki + r k j ; r k j 0; end l i k r ki ; l j 0 ; servers sort [k] so that c k j c ki < c k j c k i = k is before k ; foreach k servers ( do (s ) r ik j in j l i s i l j ) s i s j (c k j c ki ) (s i +s j ),r ki ; if r ik j > 0 then r ki r ki r ik j ; r k j r k j + r ik j ; l i l i r ik j ; l j l j + r ik j ; end end return for each k: r ki and r k j of zeros. Second, Aρ = 1, where 1 is a vector of length and consisting of ones, and A is a -by- 2 atrix defined by the following equation: { 1 if i j < (i + 1) a i j = (6) 0 otherwise. Miniization of C i (ρ) = ρ T Qρ + b T ρ with constraints ρ 0 2 and Aρ = 1 is an instance of quadratic prograing proble. As an upper triangular atrix, atrix Q has 2 eigenvalues equal to the values at the diagonal: n 2 i /2s j (1 i, j ). All eigenvalues are positive so Q is positivedefinite. Corollary 1. The proble of iniizing C i is polynoially solvable. According to [22], the best running tie reported for solving quadratic prograing proble with linear constraints is O(n 3 L) [21], where L represents the total length of the input coefficients and n the nuber of variables (here n = 2 ), so the coplexity of the best solution is O(L 6 ). However, Theore 1 encourages to apply the local optiization techniques, like the algorith presented in the next section. IV. DISTRIBUTED ALGORITHM The centralized algorith requires the inforation about the whole network the size of the input data is O( 2 ) 4

5 Algorith 2: Min-Error (MinE) algorith perfored by server id. Notation: ipr (i, j) this function calculates the iproveent of C i when transferring requests between i and j. The nuber of requests that should be transferred can be coputed by calcbesttransfer(i, j). partner argax j (ipr(id, j)); rely (id, partner, calcbesttransfer(id, partner)); and the Q atrix has O( 3 ) non-zero entries. A centralized algorith has thus the following drawbacks: (i) collecting inforation about the whole network is tie-consuing; oreover, loads and latencies ay frequently change; (ii) a standard solver takes significant tie (recall O(L 6 ) in Section III); (iii) the central algorith is ore vulnerable to failures. Motivated by these liitations we introduce a distributed algorith for finding the optial solution. The distributed algorith requires that each server has up-to-date inforation about the loads on the other servers and about the counication delays fro itself to the other servers (and not for all pairs of servers). Thus, for each server, the size of the input data is O(). As indicated in Section I, the proble of onitoring the latencies is well-studied. The loads can be disseinated by a gossiping algorith. As gossiping algoriths have logarithic convergence tie, if the gossiping is executed about O(log()) ties ore frequently than our algorith, each server has accurate inforation about the loads. Each organization, i, keeps for each server, k, the inforation about the nuber of requests that were relied to i by k. The algorith iteratively iproves the solution the i-th server in each step counicates with the locally optial partner server j (Algorith 2). The pair (i, j) locally optiizes the current solution by adjusting, for each k, r ki and r k j (Algorith 1). In the first loop of the Algorith 1, i, one of the servers, takes all the requests that were previously assigned to i and to j. Next, all the organizations [k] are sorted according to the ascending order of (c k j c ki ). The lower the value of (c k j c ki ), the ore profitable it is to run requests of k on j rather than on i in ters of the network topology. Then, for each k, the loads are balanced between servers i and j. In Section III we have shown that the optiization proble is convex. Thus, it is natural to try local optiization techniques. The presented echanis requires only two servers involved in each optiization step, thus it is very robust to failures. This echanis is siilar in spirit to the diffusive load balancing [1], [2], [7]; however there are substantial differences related to the fact that the achines are geographically distributed: (i) In each step no real requests are transferred between the servers; this process can be viewed as a siulation run to calculate the relay fractions ρ i j. Once the fractions are calculated the requests are transfered and executed at the appropriate server. (ii) Each pair (i, j) of servers exchanges not only its own requests but the requests of all servers that relayed their requests either to i or to j. Since different servers ay have different counication delays to i and j the local balancing requires ore care (Algoriths 1 and 2). Algorith 2 has the following interesting properties: (i) The single optiization step requires only two servers to be available (thus, it is very robust to failures). (ii) Any algorith that in a single step involves only two servers cannot perfor better (Theore 2). (iii) It does not require any requests to be unnecessarly delegated once the rely fractions are calculated the requests are sent over the network; also, each request is transferred over the network at ost once. (iv) The coplexity of the algorith does not depend on the nuber of requests (only on the nuber of servers), thus it is suitable for the systes with a large nuber of sall requests. (v) In each step of the algorith we are able to estiate the distance between the current solution and the optial one (Proposition 1). A. Correctness The following Lea shows how to optially exchange the requests owned by organization k between a pair of servers i and j. Lea 1. Consider two servers i and j that execute r ki and r k j requests of the k-th organization. The total processing tie, C i, is iniized when the k-th server relies r ik j fro r ki requests to be additionally executed on j-th server: r ik j = (s jl i s i l j ) s i s j (c k j c ki ) (s i + s j ) r ik j = ax(0,in(r ki, r ik j )) Proof: If the k-th server oves soe of its requests fro i to j, then it affects the copletion tie of all requests that were relayed either to i or to j (initial requests of all servers). Recall that l i and l j are the loads of the servers, respectively, i and j, that is they include all tasks relayed to, respectively, i and j. Thus, if k reoves r of its requests fro i, then the new processing tie of all tasks on the server i will be (l i r) 2 /2s i. Thus, we want to find r ik j that iniizes the function f : f ( r) = (l i r) 2 + (l j + r) 2 rc ki + rc k j 2s i 2s j We can find iniu by calculating derivative: d f d r = r l i + r + l j c ki + c k j = 0 s i s j r ik j = (s jl i s i l j ) s i s j (c k j c ki ) (s i + s j ) Also r 0,r ki, which proves the thesis. 5

6 The following lea proves the correctness of Algorith 1. Theore 2. After execution of Algorith 1 for the pair of servers i and j, it is not possible to iprove C i only by exchanging any requests between i and j. Sketch of Proof: First we show that after the second loop no requests should be transferred fro i to j. For each organization k the requests owned by k were transferred fro i to j in soe iteration of the second loop; also, each of the next iterations of the second loop could only cause the increase of the load of j (and decrease of i); thus transferring ore requests of k fro i to j would be inefficient. Second, we will show that after the second loop no requests should be transferred back fro j to i either. Let us take the last iteration of the second loop in which the requests of soe organization k were transferred fro i to j. After this transfer we know that r ik j = (s jl i s i l j ) s i s j (c k j c ki ) (s i +s j ) 0 (otherwise the transfer would not be optial). However, this iplies that r ik j = (s jl i s i l j ) s i s j (c k j c k i ) (s i +s j ) 0 for each server k considered before k. As r ik j 0 we get r jk i 0. B. Error estiation The following analysis bounds the distance of the current solution of the distributed algorith to the optiu as a function of the disparity of servers load. When running the algorith, this result can be used to assess whether it is still profitable to continue running the algorith: if the load disparity is low, the current solution is close to the optiu. We introduce the following notation for the analysis. ρ is the snapshot (the current solution) derived by distributed algorith. ρ is the optial solution that iniizes C i (if there are ultiple optial solutions with the sae C i, ρ is the closest solution to ρ in the Manhattan etric). (P, ρ) is a weighted, directed error graph: ρ[i][ j] indicates the nuber of requests that should be transferred fro server i to j in order to reach ρ fro ρ ( ρ[i][ j] requests either belong to i, or to j, and not to another server k). We define dir as the direction of transport: dir(i, j) = 1 if i transfers to j its own requests; dir(i, j) = 1 if i returns to j the requests that initially belonged to j. Let succ(i) denotes the set of successors in the error graph: succ(i) = { j : ρ[i][ j] > 0}; prec(i) denotes the set of predecessors: prec(i) = { j : ρ[ j][i] > 0}. In the error graph, a negative cycle is a sequence of servers i 1,i 2,...,i n such that (i) i 1 = i n ; (ii) j {1,...n 1} ρ[i j ][i j+1 ] > 0; and (iii) n 1 j=1 dir(i j,i j+1 )c i j i j+1 < 0. A negative cycle is sequence of servers that essentially redirect their requests to one another. A solution without negative cycles has a saller processing tie: after disantling a negative cycle, loads on servers reain the sae, but the counication tie is reduced. In Appendix, we show how to detect and reove negative cycles; in order to siplify the presentation of the subsequent analysis, we consider that there are no negative cycles. Proposition 1. If (i) the error graph ρ has no negative cycle; and (ii) j ax k (( s 1 j + s 1 ) r k jk ) = R ( r i j is the nuber of requests which in the current state ρ would be relied to j-th server by the i-th server (as the result of Algorith 1), then ρ ρ 1 (4+1) R i s i, where 1 denotes the Manhattan etric. Proof: The proof is presented in the full version of the article [33]. Proposition 1 gives the estiation of the error for such partial solutions that do not have a negative cycles. Therefore the algorith that cancels negative cycles (see Appendix) should be run whenever the estiation for distance to the optial solution is needed. Our experients show, however, that the negative cycles are rare in practice and that pure Algorith 2 can reove the efficiently (Section VI). V. SELFISH ORGANIZATIONS In this section we consider the case when the organizations are acting selfishly the i-th of the tries to iniize the total processing tie of its own requests C i. We are interested in a steady state in which all the peers have no interest in redirecting any of its requests to different servers the Nash equilibriu. A. Hoogeneous network In this section we present the characteristic of the Nash equilibriu in case when all the servers have equal processing power ( i s i = s), and when all the connections between servers have the sae counication delay ( i j c i j = c). We consider hoogeneous odel, as the odeling of a heterogeneous interconnection graph is coplex. The siulation experients (Section VI-C) show that in the case of selfish servers the average relative degradation of the syste goal on heterogeneous networks is siilar to, or lower than on the hoogeneous networks. Lea 2. For every two servers i and j the difference between their average loads is bounded: l i l j c s Proof: (by contradiction) Assue l i l j > c s. Without loosing the generality, l i > l j. Recall that r i j is the nuber of redirected requests r i j = n i ρ i j. For each sever k (k i), it is not profitable to put ore of its requests to the ore loaded server, so r k j r ki. Now we want to find the relation between l i,l j,r i j and r ii. In a Nash equilibriu, it is not profitable for i to redirect any additional x of its own requests fro itself to j, which can be forally expressed by the equation: 0 (l i x)(r ii x) 2s l ir ii 2s l jr i j 2s cr i j, + (l j + x)(r i j + x) 2s + c(r i j + x) 6

7 equivalent to: r i j r ii + 2x l i l j 2c s. Because the inequality ust hold for every positive x, and because l i l j > c s r i j r ii > c s 2c s = c s Now we can show the contradiction, because k= l j = k=1 k= r k j > fro which it follows that k=1 l i l j < c s. r ki c s = l i c s Let us denote the average load on the server as, thus = 1 i= i=1 l i. The following theore gives the tight estiation of the price of anarchy when the servers are loaded copared to the delay ( 2cs). If the servers are not loaded then the thesis of Theore 3 is still correct, but our estiation of the price of anarchy is doinated by O(( l c av ) 2 ) eleent, thus it is not tight). Theore 3. The price of anarchy in the hoogeneous network is: PoA = 1 + 2cs + O(( l cs av ) 2 ). Proof: (upper bound) We denote the load ibalance on the i-th server as i = l i. It follows that i= i=1 i = 0. Also, fro Lea 2 we have i c s. Additionally, each request can be relied at ost once, thus the total tie used for counication is bounded by c. Therefore, the total processing tie in case of selfish peers, C i (self) is bounded: (l i ) C i (self) c + 2 i i 2s = c + i ( + i ) 2 = l2 av 2s + 2 i 2s + c l2 av 2s + c2 s 2 + c The total processing tie is the sallest when the servers have equal load (each server processes exactly requests) and do not counicate, thus the optiu is bounded by (C i ) l2 av 2s. Thus, the price of anarchy is bounded by: PoA l2 av + 2 cs + c 2 s 2 l 2 av 2s = 1 + 2cs + ( cs ) 2 (tightness) Consider an instance with servers having equal initial load: i n i =. In the optial solution no requests will be redirected. When servers are selfish, the i-th server will redirect to j-th server (i j) 2c s requests and will execute (2c s+ 2c s of its own requests on itself. As a result: l i =. This is a Nash Equilibriu state, because it is not profitable for any server to redirect any x ore of its own requests to the other server, nor to execute any x ore requests on itself instead of soe other server, as the two following inequalities hold for every positive x: 0 < x (2c s + 2c s x) + + x ( 2c s + x) 2s 2s + cx 2s (2c s + 2c s ) 2s ( 2c s ) 0 < + x (2c s + 2c s 2s cx 2s (2c s + 2c s + x) + x ( 2c s x) 2s ) 2s ( 2c s ) Thus, we get the lower bound on the price of anarchy: PoA l2 av + ( 2c s 2c s )c 2s lav 2 = 1 + 2cs 4( sc ) 2 2( 2c 2 s 2 ) lav cs 4( cs ) 2. Suarizing: 1 + 2cs 4( cs ) 2 PoA 1 + 2cs + ( cs ) 2 The price of anarchy depends on the average load on the server and on the network delay. For the ore general case, in Section VI-C we present the estiations derived fro siulations. VI. SIMULATION EPERIMENTS In this section we show the results fro the two groups of experients. First, we investigate convergence tie of the distributed algorith. Second, we assess the loss of perforance in an organizationally-distributed syste copared to the optial, central solution. The loss is coputed as a ratio of the total processing ties. A. Settings We experiented on two kinds of networks: hoogeneous, with equal counication latencies (c i j = 20); and heterogeneous, where latencies were based on easureents between PlanetLab nodes 1 expressed in illiseconds 2. In the initial experients, we analyzed networks coposed of 20, 30, 50, 100, 200 and 300 serves. We also perfored soe experients on larger networks (500, 1000, 2000, 3000 servers). The processing speeds of the servers s i were uniforly distributed on the interval 1, 5. We conducted the experients for exponential and unifor distribution of the initial load over the servers. For each distribution we analyzed five cases with the average load equal to 10, 20, 50, 200 and 1000 requests (assuing that The dataset does not contain latencies for all pairs of nodes, so we had to copleent the data by calculating inial distances. 7

8 total processing tie ( Ci) iteration #servers = 500 #servers = 1000 #servers = 2000 #servers = 3000 #servers = 5000 Figure 2: The convergence of the distributed algorith for peak distribution of initial loads. processing a single request on a single server takes 1s). We also analyzed the case of peak distribution with requests owned by a single server. We evaluated the result based on the distance to the optial solution, which because of the O( 6 ) coplexity of standard solvers (see Section III) was approxiated by our distributed algorith. B. Convergence tie of the distributed algorith In the first series of experients, we evaluated the efficiency of the distributed algorith easured as the nuber of iterations the algorith ust perfor in order to decrease the difference between the total processing ties in the current and the optial requests distributions to less than 2% of the average load. In a single iteration of the distributed algorith, each server executes Algorith 2; if there were any pairs of the servers to be optiized we run optiization in the rando order. Table I suarizes the results. The results indicate that the nuber of iterations ostly depends on the size of the network and on the distribution of the initial load. The type of the network (planet-lab vs. hoogeneous) does not influence the convergence tie. Larger networks and peak distribution result in higher convergence ties. In all considered networks, the algorith converged in at ost 9 iterations. Next, we decreased the required precision error fro 2% to 0.1%, and ran the sae experients. The results are given in Table II. In this case, siilarly, the required nuber of iterations was the highest for peak distribution of the initial load. In each case the algorith converged in at ost 11 iterations. Even for 300 servers the average nuber of iterations is below 8. Also, the standard deviations are low, which indicates that the algorith is stable with respect to its fast convergence. Also, we assessed whether a variation of the distributed algorith that does not eliinate negative cycles (Appendix A) has a slower convergence tie. Although required to prove the convergence (Section IV-B), eliinating the 50 = 100 = 200 = 300 # iterations average ax st. dev. unifor exp peak unifor exp peak unifor exp peak unifor exp peak Table I: The nuber of iterations of the distributed algorith required to obtain at ost 2% relative error in the total processing tie ΣC i. 50 = 100 = 200 = 300 # iterations average ax st. dev. unifor exp peak unifor exp peak unifor exp peak unifor exp peak Table II: The nuber of iterations of the distributed algorith required to obtain at ost 0.1% relative error in the total processing tie ΣC i. negative cycles is coplex in ipleentation and doinates the execution tie. We copared two versions of the distributed algorith: without negative cycle reoval; and with the reoval every two iterations of the algorith. The nuber of iterations for two versions of the algorith were exactly the sae in all 6000 experients. These result show that the cycles which happen in practice can be efficiently reoved by pure Algorith 1. Also, the negative cycles are rare in practice. Finally, we analyzed the convergence of the distributed algorith without negative cycles eliination on larger networks (Figure 2). The previous experients shown that the algorith convergence is the slowest for peak distribution of the initial load, therefore we chose this case for the analysis. The experients used heterogeneous network. The results indicate that even for larger networks the total processing tie decreases exponentially. C. Cost of selfishness In the second series of experients we experientally easured the cost of selfishness as the ratio between total processing ties in cases of selfish and cooperative servers (Table III). In each experient, the Nash equilibriu was approxiated by the following heuristics. Each server was playing its best response to the current distribution of 8

9 const si unifor si 30 = = Ratio avg. ax st. dev. c i j = PL c i j = PL c i j = PL c i j = PL c i j = PL c i j = PL Table III: Experiental assessent of the cost of selfishness: ratios between total processing ties in cases of selfish and cooperative servers. requests. We terinated when all servers in two consecutive steps changed the distribution of their requests by less than 1%. We coputed the ratio of the total processing ties: the (approxiated) Nash equilibriu to the optial value. The cost of selfishness is low. The average is below 1.06; and the axial value is below The estiation of the cost of selfishness is higher in case of constant processing rates s i. It additionally depends on the ratio between the average initial load and the network latency and on the structure of the network. The highest cost is for hoogeneous networks with constant processing rates and having ediu initial load about 2 ties longer than the ean counication delay. The experients show that the cost of selfishness is independent of the size of the network and the type of distribution of initial loads. VII. ETENSION: REQUESTS OF DIFFERENT PROCESSING TIMES; REPLICATION Up to this point, we odeled a distributed request processing syste, in which requests have the sae size. In this section we show how our results extend to the odel where the individual requests (constituting the load) have different durations and where the requests additionally have redundancy requireents (These extensions are particularly relevant for the proble of finding the replica placeent in CDNs here different data pieces have different popularities and data redundancy is a coon requireent for increasing the availability). We introduce the following additional notation. A task is an individual request. J i = {J i (k)} denotes the set of tasks of organization i; p i (k) is the size (processing tie) of the task J i (k). First, let us analyze a proble in which the tasks have no redundancy requireents, i.e. each task has to be processed on exactly one server. In order to find the optial solution in this extended odel, we start with solving the original proble (as defined in Section II) with n i = k p i (k). In order to derive the actual distribution of the tasks, we discretize the fractions ρ i j as follows. i should relay to j such subset S i ( j) J i of its own tasks, so that the total error Σerr(S i ( j)) is iniized: err(s i ( j)) = k:j i (k) S i ( j) p i (k) ρ i j n i. The rounding proble is the ultiple subset proble with different knapsack capacities [10]. The proble is NPcoplete but has a polynoial approxiation algorith. Now consider a proble in which each organization ust execute at least R copies of each task; each copy of the task should be executed at a different location (the execution of the tasks is replicated). This setting odels a CDN, but also job processing, where to increase survivability iportant jobs are replicated on different partions of a datacenter or on different datacenters. In this extended proble we have to introduce additional constraint on the fractions ρ i j for the original proble (Section II): i, j ρ i j R 1, which guarantees that Rρ i j 1. With this constraint we can interpret Rρ i j as the probability of placing a copy of J i (k) at j; here the expected nuber of copies of J i (k) is j Rρi j = R. VIII. RELATED WORK The congestion gaes [14], [23], [26], [31] define the odel for analyzing the selfish behavior of users copeting for coonly available resources. Siilarly to our odel, the cost of a particular resource is linearly proportional to the nuber of copetitors using the resource. In contrast, our odel ore closely describes the cost of using a resource which depends also on the counication delay. The assuptions in our odel are siilar as in the literature on network virualization [5]. However in network virualization the probles regard locating services which is different fro optiizing the quality of serving the coon user requests. The coplexity of the solutions depend on the nuber of configurations (which here is unbounded) thus the solutions cannot be applied to our odel. The continuos allocation of requests to servers in our odel is analogous to the divisible load odel [19] with constant-cost counication (a special case of the affine cost odel [6]) and ultiple sources (ultiple loads to be handled, [16], [36]). The ain difference is the optiization goal: akespan is usually optiized in the divisible load odel; in contrast, we optiize the average processing tie, which, we believe, better odels situations in which the load is coposed of ultiple, sall requests issued by various users (the difference is analogous to C ax versus ΣC i debate in the classic ultiprocessor job scheduling). The other difference is how the network topology is odelled. The divisible load theory typically studies datacenter-type systes, in which the network topology is known and is a liiting factor, thus the transissions ust be scheduled in a siilar way to the coputations. Distributed algoriths for load balancing ostly relay on local optiization techniques (see [1], [7], [38]). One 9

10 of the ost popular techniques is diffusive load balancing, siilar in spirit to our distributed algorith (see [2] and the references inside for the current state of the art and [38] for the basic echanis description). These solutions, however, disregard the geographic distribution of the servers. Our algorith uses different idea the diffusive process is used for calculating the relay fractions instead of for balancing the load. As the result, our local balancing ust take into account different latencies between the servers which requires ore subtle exchange echaniss (Algoriths 1 and 2). Our gae-theoretic approach is coparable to the selfish job odel [37]: the jobs independently chose the processor on which to execute. While soe studies consider ixed case equilibria (aking the odel continuous siilarly to ours), our odel considers also counication latency. The coon infrastructure odels tend to have a low price of anarchy (of order log/loglog [37]) the low price of anarchy in our odel extends these results. Content delivery networks are one of the otivations for our odel. Large copanies, like Akaai, specialize in delivering the content of their custoers so that the end users experience the best quality of service. Akaai s architecture is based on DNS redirections [24], [25], [34]. However, the description of the algoriths optiizing replica placeent and request handling are not disclosed. Still, Akaai s infrastructure is owned and controlled by a single entity (Akaai), thus they do not need to solve the gae-theoretic equivalent of our odel. CoralCDN [18] is a p2p CDN consisting of users voluntarily devoting their bandwidth and storage to redistribute the content. In CoralCDN the popular content is replicated aong ultiple servers (which can be viewed as relaying the requests); the requests for content are relayed only between the servers with constrained pairwise RTTs (which ensures the proxiity of delivering server). Our atheatical odel foralizes the intuitions behind heuristics in CoralCDN. [13] shows a CDN based on a DHT and heuristic algoriths to iniize the total processing tie. Although each server has a fixed constrains on its load/bandwidth/storage capacity, the paper does not consider the relation between server load and its perforance degradation. The evaluation is based on siulation; no theoretical results are included. The proble of irroring in the Internet is analyzed in [15], [30]. Both papers show different approaches to choosing locations for replicas so that the average network delay between data locations and end-users is iniized. The ipact of servers congestion is not taken into consideration. I. CONCLUSIONS In this paper we present and analyze a odel of a distributed syste that iniizes the request processing tie by distributing the requests aong servers. Existing odels assue that the processing tie is doinated either by the network counication delay or by congestion of servers. In contrast, in our odel, the observed latency is the su of the two delays: network delay and congestion. Our odel can be used in different kinds of probles in distributed systes, ranging fro routing in content delivery networks to load balancing in a cloud of servers. We show that the proble of iniizing the total processing tie can be stated as an optiization proble in the continuous doain. We proove that the optiization proble is convex and, in particular, polynoially solvable. We propose a distributed algorith that, according to our experiental evaluation, even in a network consisting of thousands of servers requires only a dozen of essages sent by each server to converge to a solution worse than at ost 0.1% of the optiu (not counting the gossiping to exchange the inforation). We present the properties of the distributed algorith. We show how to estiate the distance between the current solution found by the algorith and the optial solution. The estiation requires solving the subproble of finding the axial flow of the inial cost in a graph. However, the distributed algorith still outperfors standard optiization techniques. Based on the experients, we argue that in practice this part of the algorith can be oitted, as it does not influence the algorith efficiency. We also analyze how the lack of coordination influences the total processing tie. We give theoretical bounds for the price of anarchy for hoogeneous networks and high average loads. Additionally, we assess the price of anarchy experientally on heterogenous networks. In both cases the price of anarchy is low (1 + 2cs in the theoretical analysis, and below 1.15 in the experients). We show how the proble can be extended and, in particlaur, how it can be applied to find the optial distribution of the data (even with their replicas) when the access pattern of the users is known, and how it can be applied to the case when the jobs have with different processing ties. Our results the low price of anarchy and an efficient distributed optiization algorith indicate that a fully distributed query processing syste can be efficient. Thus, instead of buying services fro dedicated cloud providers or CDN operators, saller organizations, such as ISPs or universities, can gather in consortia effectively serving the participators needs. REFERENCES [1] H. Ackerann, S. Fischer, M. Hoefer, and M. Schöngens. Distributed algoriths for qos load balancing. In Proceedings of the twenty-first annual syposiu on Parallelis in algoriths and architectures, SPAA 09, pages , [2] C. P. J. Adolphs and P. Berenbrink. Iproved bounds for discrete diffusive load balancing. In IPDPS, pages IEEE Coputer Society, [3] R. K. Ahuja, T. L. Magnanti, and J. B. Orlin. Network Flows: Theory, Algoriths, and Applications. Prentice Hall,

11 [4] J. Allard, S. Cotin, F. Faure, P. Bensoussan, F. Poyer, C. Duriez, H. Delingette, L. Grisoni, et al. Sofa-an open source fraework for edical siulation. In Medicine Meets Virtual Reality, MMVR 15, [5] D. Arora, A. Feldann, G. Schaffrath, and S. Schid. On the benefit of virtualization: Strategies for flexible server allocation. In Proceedings of USENI Workshop on Hot Topics in Manageent of Internet, Cloud, and Enterprise Networks and Services (Hot-ICE 11), [6] O. Beauont, H. Casanova, A. Legrand, Y. Robert, and Y. Yang. Scheduling divisible loads on star and tree networks: results and open probles. Parallel and Distributed Systes, IEEE Transactions on, 16(3): , [7] P. Berenbrink, M. Hoefer, and T. Sauerwald. Distributed selfish load balancing on networks. In Proceedings of the Twenty-Second Annual ACM-SIAM Syposiu on Discrete Algoriths, SODA 11, pages , [8] D. P. Bertsekas. Auction algoriths for network flow probles: A tutorial introduction. Coputational Optiization and Applications, 1:7 66, [9] D. P. Bertsekas and J. N. Tsitsiklis. Parallel and distributed coputation: nuerical ethods. Prentice-Hall, [10] A. Caprara, H. Kellerer, and U. Pferschy. A PTAS for the ultiple subset su proble with different knapsack capacities. Inforation Processing Letters, 73(3-4), [11] E. Chan-Tin and N. Hopper. Accurate and provably secure latency estiation with treeple. In NDSS. The Internet Society, [12] A. Chawla, B. Reed, K. Juhnke, and G. Syed. Seantics of caching with spoca: a stateless, proportional, optiallyconsistent addressing algorith. In USENIATC, [13] Y. Chen, R. H. Katz, and J. Kubiatowicz. Dynaic replica placeent for scalable content delivery. In IPTPS, Proceedigs, pages , London, UK, Springer-Verlag. [14] G. Christodoulou and E. Koutsoupias. The price of anarchy of finite congestion gaes. In STOC, Proceedings, pages 67 73, [15] E. Cronin, S. Jain, C. Jin, A. R. Kurc, D. Raz, Y. Shavitt, and S. Meber. Constrained irror placeent on the internet. In JSAC, pages 31 40, [16] M. Drozdowski and M. Lawenda. Scheduling ultiple divisible loads in hoogeneous star systes. Journal of Scheduling, 11(5): , [17] P. Dutot, L. Eyraud, G. Mounié, and D. Trystra. Bi-criteria algorith for scheduling jobs on cluster platfors. In IPDPS, Proc., pages ACM, [18] M. Freedan. Experiences with coralcdn: A five-year operational view. In NSDI USENI, Proceedings, [19] M. Gallet, Y. Robert, and F. Vivien. Divisible load scheduling. In Y. Robert and F. Vivien, editors, Introduction to Scheduling. CRC Press, Inc., [20] A. V. Goldberg and R. E. Tarjan. Finding iniu-cost circulations by successive approxiation. Math. Oper. Res., 15: , July [21] D. Goldfarb and S. Liu. An o(n3) prial interior point algorith for convex quadratic prograing. Math. Progra., 49: , January [22] D. S. Hochbau. Coplexity and algoriths for nonlinear optiization probles. Annals OR, 153(1): , [23] E. Koutsoupias and C. H. Papadiitriou. Worst-case equilibria. Coputer Science Review, 3(2):65 69, [24] F. Leighton and D. Lewin. Global hosting syste. US Patent No. 6,108,703. [25] R. Mahajan. How akaai works? co/en-us/u/people/ratul/akaai.htl. [26] N. Nisan, T. Roughgarden, E. Tardos, and V. V. Vazirani. Algorithic Gae Theory, chapter Routing Gaes. Cabridge University Press, [27] E. Nygren, R. K. Sitaraan, and J. Sun. The Akaai network: a platfor for high-perforance internet applications. SIGOPS Oper. Syst. Rev., 44:2 19, August [28] B. D. P. Network Optiization: Continuous and Discrete Models. Athena Scientific, [29] G. Pallis and A. Vakali. Content delivery networks. Counications of the ACM, 49(1):101, [30] L. Qiu, V. N. Padanabhan, and G. M. Voelker. On the placeent of web server replicas. In IEEE INFOCOM, Proceedings, pages , [31] A. Roth. The price of alice in linear congestion gaes. In WINE, Proceedings, pages , [32] K. Rzadca, A. Datta, and S. Buchegger. Replica placeent in p2p storage: Coplexity and gae theoretic analyses. In ICDCS, Proceedings, pages , [33] P. Skowron and K. Rzadca. Network delay-aware load balancing in selfish and cooperative distributed systes. CoRR, abs/ , [34] A.-J. Su, D. R. Choffnes, A. Kuzanovic, and F. E. Bustaante. Drafting behind Akaai. SIGCOMM Coput. Coun. Rev., 36: , August [35] M. Szyaniak, G. Pierre, and M. Steen. Scalable cooperative latency estiation. In ICPADS, [36] B. Veeravalli and G. Barlas. Efficient scheduling strategies for processing ultiple divisible loads on bus networks. Journal of Parallel and Distributed Coputing, 62(1): , [37] B. Vocking. Selfish load balancing. In N. Nisan, T. Roughgarden, E. Tardos, and V. V. Vazirani, editors, Algorithic Gae Theory. Cabridge University Press, [38] C. u and F. C. Lau. Load Balancing in Parallel Coputers: Theory and Practice. Kluw. Acad. Pub.,

12 APPENDI REMOVING NEGATIVE CYCLES The proble of negative cycles reoval can be reduced to finding the axial flow of the inial cost in a graph. The proble of finding the axial flow of the iniu cost is well studied in the literature; there are any algoriths [3], [28]. In particular the auction algoriths [8], ε- relaxation ethod [9], and the approxiation ethod for finding iniu circulation [20] are the exaples of the distributed algoriths solving the proble. For the purpose of proving the reduction we introduce the following notation. out(ρ,i) denotes the total aount of requests that in a partial solution ρ are relied by a server i to all other servers: out(ρ,i) = j i r i j. in(ρ,i) denotes the total aount of requests that in ρ are relied by all other servers to i, in(ρ,i) = j i r ji. We construct the ax-flow in-cost graph as follows. For each server i we introduce two graph vertices: the front i f and the back i b. There are two additional vertices: s (source) and t (target). The source s is linked with each front node, i f with an edge (s,i f ) with zero cost and capacity equal to out(ρ,i). Each back node, i b is linked with the target t with an edge (i b,t) with zero cost and capacity equal to in(ρ,i b ). Intuitively, capacity of (s,i f ) equal to out(ρ,i) represents a server i sending out the sae load out(ρ,i) as in ρ ; conversely, capacity of (i b,t) equal to in(ρ,i) represents server i accepting the sae aount of load as in ρ. There are also edges between front and back nodes: for each pair (i f, j b ), i j there is an edge with cost equal to c i j and infinite capacity. The axial flow of the inial cost f between s and t can be apped to a new partial solution ρ : a flow on an edge (i f, j b ) f i j corresponds to server i relying f i j of its own requests to server j. Observe that, as capacity (s,i f ) = out(ρ,i) and capacity (i b,t) = in(ρ,i), the load of i-th server in ρ is equal to its load in ρ. Additionally, there are no negative cycles in ρ: a negative cycle would result in a non-inial cost of the flow f. APPENDI VALIDATION OF THE CONSTANT LATENCY We experientally verified how the aount of the load sent over the network influences the counication delay between the servers. We randoly selected 60 PlanetLab servers, scattered around Europe, and siulated different intensity of the background load in the following way. Each server choses its 5 neighbors randoly, but in a way that each server has exactly 5 neighbors. Then the servers start sending data with constant throughput to its 5 neighbors. In different experients, we used 8 values of the throughputs: 10KB/s, 20KB/s, 50KB/s, 100KB/s, 200KB/s, 500KB/s, 1MB/s, 2MB/s. If a particular throughput was not achievable, the server was just sending data with the axial achievable throughput. By the nature of t b e(,,t b ) µ σ 10 KB/s KB/s KB/s MB/s t b e(,,t b ) µ σ 0.2 MB/s MB/s MB/s MB/s Table IV: The relative deviation of the average throughput caused by the increase of the background load (after reoval of 5% largest deviations). experients on PlanetLab, we were not granted a dedicated access to the achines; thus other experients running on the sae servers added further, unknown network transfers. Additionally, alost all PlanetLab servers do not specify the bandwidth of their Internet connection, nor the historical bandwidth usage. For each value of the background load we calculated the average round trip tie (RTT) between the server and each of its 5 neighbors (we used the average fro 300 RTT saples). Let rtt(s i,s j,t b ) denote the average rtt between servers s i and s j with the background load generated with throughput t b. For each pair of the servers s i and s j for which we easured the RTT, and for each value of the background throughput t b we calculated the relative deviation of the average throughput caused by the increase of the background load copared to the inial throughput 10KB/s: e(s i,s j,b t ) = rtt(s i,s j,t b ) rtt(s i,s j,10kb/s) rtt(s i,s j,10kb/s). For each value of the background throughput, we reoved 5% of the largest deviations and then calculated the ean fro deviations e(s i,s j,b t ), averaged over all pairs of servers (µ). For each value of the background throughput we additionally calculated the standard deviations (σ). These results are presented in Table IV. Fro the data we see that up to b t = 0.2MB/s, which corresponds to the case where each server accepts = 8Mb/s of incoing data, the average RTT was not influenced by the background throughput. This is also confired by the statistical analysis of the data run for the RTTs (instead of for deviations). For b t 0.2MB/s the ANOVA test (which we run for the whole population without reoving 5% of the highest RTTs) confired the lack of dependency null hypothesis (that the background throughput does not influence the RTTs) for over 56% of the pairs of servers. For b t 0.1MB/s (corresponding to 4Mb/s of incoing throughput) the ANOVA test confired null hypothesis for over 70% of the pairs of servers and for b t 50KB/s (corresponding to 2Mb/s of incoing throughput) for over 90% of the pairs. We consider that these results strongly justify the assuption of a constant latency in our odel. 12