Load Balancing in Dynamic Structured P2P System

Transcription

1 Load Balancing in Dynamic Structured P2P System B. Godfrey, K. Lakshminarayanan, S. Surana, R. Karp, I. Stoica Ankit Pat November 19, Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

2 Outline Outline Introduction Preliminaries Background Load Balancing Algorithm Empirical Evaluation Conclusion Discussion. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

3 Introduction Introduction Most DHT supporting P2P systems distribute objects (data) randomly among nodes Some nodes have Θ(log N) imbalance Other factors resulting in imbalance non-uniform distribution of objects in ID space heterogeneity in object loads node capacities variability of a node s load with time. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

4 Introduction Introduction This paper proposes the first algorithm for dynamic load balancing in heterogenous, structured P2P systems data items inserted/deleted continuously nodes join/depart continuously Conducts extensive simulations to show its validity B. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

5 Preliminaries Definitions Load: Represents the needed storage space, popularity, needed processor time etc. of the object Movement Cost: between nodes Cost associated with moving an object Capacity: Each node has a fixed capacity for e.g. disk space, processor speed, bandwidth etc. Node Utilization: Total load divided by capacity for a node System Utilization: capacity of all nodes Total load across nodes divided by total. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

6 Preliminaries Goals Minimizing the load imbalance across nodes Minimizing the amount of load moved between nodes. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

7 Background Virtual Servers Most DHTs map a region of the ID space to a node Unique IDs are attached to the object and the responsible node in the same ID space With virtual servers, this mapping is done on virtual servers instead of node A node now has multiple virtual servers and hence IDs No need to change underlying DHT with joining/departing of nodes (advantage) B. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

8 Background Static Load Balancing Techniques One-to-one scheme: node at random lightly loaded node periodically contacts a One-to-many scheme: a heavy node contacts a directory node which is contacted by random light nodes Many-to-many scheme: each directory maintains load information of a set of heavy & light nodes. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

9 Node next periodic balance. Load BalancingThe Algorithm directory then schedules immediate transfers from n to more lightly loaded nodes. More precisely, each node n runs the following algorithm. Node(time period T,thresholdk e ) Initialization: Send (c n, {l v1,...,l vm }) to RandomDirectory() Emergency action: When u n jumps above k e : 1) Repeat up to twice while u n >k e : 2) d RandomDirectory() 3) Send (c n, {l v1,..., l vm }) to d 4) PerformTransfer(v, n ) for each transfer v n scheduled by d Periodic action: Upon receipt of list of transfers from adirectory: 1) PerformTransfer(v, n ) for each transfer v n 2) Report (c n, {l v1,..., l vm }) to RandomDirectory() Reass 1) p 2) F le 3) F li (l 4) R We br Period transfers signvs However is (abou to emerg discussio Choic threshold load inc threshold. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

10 Directory n if it wouldload not Balancing overload Algorithm n, i.e. if l n + l v c n. Thus a transfer may be aborted if the directory scheduled a transfer based on outdated information (see below). Each directory runs the following algorithm. Directory(time period T,thresholdsk e,k p ) Initialization: I {} Information receipt and emergency balancing: Upon receipt of J =(c n, {l v1,...,l vm }) from node n: 1) I I J 2) If u n >k e : 3) reassignment ReassignVS(I,k e ) 4) Schedule transfers according to reassignment Periodic balancing: Every T seconds: 1) reassignment ReassignVS(I,k p ) 2) Schedule transfers according to reassignment 3) I {} The subroutine ReassignVS, given a threshold k and the load information I reported to a directory, computes a load mo values o of great Stale times a at whic our sim directori and the reassign We us algorithm the em the bal (Se the util. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

11 Load Balancing Algorithm Reassignment of Virtual Servers bove a reports for d s ediate rithm. algorithm runs in O(m log m) time, where m is the number of virtual servers that have reported to the directory. ReassignVS(Load & capacity information I, threshold k) 1) pool {} 2) For each node n I, whilel n /c n >k,removethe least loaded virtual server on n and move it to pool. 3) For each virtual server v pool, from heaviest to lightest, assign v to the node n which minimizes (l n + l v )/c n. 4) Return the virtual server reassignment. Ran- We briefly discuss several important design issues. Periodic vs. emergency balancing. We prefer to schedule ansfer transfers in large periodic batches since this gives ReassignVS more flexibility, thus producing a better balance.. Godfrey, K. Lakshminarayanan, However, we S. Surana, do Load R. not Balancing Karp, have I. Stoica in Dynamic the Structured luxury P2P tosystem wait when November a node 19, / 30

12 Load Balancing Algorithm Some Design Issues Periodic vs. emergency balancing: large T is preferred but emergency situations are taken care of Choice of parameters: (1 + ˆµ)/2 threshold k e is set to 1 and k p is set to Stale information: not synchronized node reporting times across directories is. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

13 Empirical Evaluation Metrics Load Movement Factor: total movement cost due to load balancing divided by the total most of moving all objects in the system once 99.9th percentile node utilization: maximum over all simulated times t of the 99.9th percentile of node utilizations at time t B. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

14 Empirical Evaluation Basic Effect of Load Balancing oncluding that 60% as much o node arrivals g unrelated to riation has little two primary 99.9th Percentile Node Utilization Periodic Periodic + Emergency movementcost y the total cost Note that since Load Movement Factor itially insert it, at the balancer Fig th percentile node utilization vs. load moved, for our quired to insert periodic+emergency algorithm and for only periodic action. Tradeoff between load movement and 99.9th percentile node d as the utilization maxiemergency load balancing. In the simulations of the rest of this 9.9th percentile paper, emergency balancing is enabled as in the description of t. Recallfrom Rest ofour the algorithm simulations in Sectionhave IV. emergency balancing is enabled its load divided. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

15 Number of virtual servers per node 12 Number of directories 16 Empirical Evaluation Load Movement vs. 99.9th Percentile Node Utilization a Rescaled to obtain the specified system utilization. b We discard samples outside the range [10, 5000]. c ˆµ is the average utilization of the nodes reporting to a particular directory th Percentile Node Utilization Sys Util = = 0.8 = 0.7 = 0.6 = Load Movement Factor 99.9th Percentile Node Utilization Fig. 2. Tradeoff between 99.9th percentile node utilization and load movement factor as controlled by load balance period T, for various system utilizations. Fig. 3. Tradeoff betw movement factor as co numbers of objects in t 99.9% of nodes are underloaded for load movement factor <0.08 all nodes) number of virtual servers per node. We make two points. First, our algorithm achieves a good IDs are uniformly di choose m = Θ(log N. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

16 16 Empirical Evaluation Load Movement vs. 99.9th Percentile Node Utilization n.. g to a particular directory th Percentile Node Utilization Num Objs = 1 million = 750, = 500,000 = 250,000 = 100, Load Movement Factor ilization and load od T, for various chieves a good particular, the Fig. 3. Tradeoff between 99.9th percentile node utilization and load movement factor as controlled by load balance period T, for various numbers of objects in the system. For at least 250,000 objects, good load balance and load movement factor of 0.11 is achieved IDs are uniformly distributed. To avoid this problem, we can choose m = Θ(log N) as suggested in [2], reducing the max B. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

17 Empirical Evaluation Load Movement vs. 99.9th Percentile Node Utilization 99.9th Percentile Node Utilization Num Dir = 1 = 4 = 16 = 64 = 256 Load Movement Factor Num VS = == = Load Movement Factor Fig. 4. Tradeoff between 99.9th percentile node utilization and load movement factor as controlled by load balance period T, for various numbers of directories. Number of directories has a small impact on the metrics 4 Num VS = Num VS = 2 = 1.4 = = = 8 = 1 = 12. Godfrey, K. Lakshminarayanan, S. 1.2Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30 tion Fig. 6. Load movem numbers of virtual serv ilization

18 Number of Virtual Servers Empirical Evaluation Load Movement Factor Fig. 4. Tradeoff between 99.9th percentile node utilization and load movement factor as controlled by load balance period T, for various numbers of directories. 99.9th Percentile Node Utilization Num VS = 2 = 4 = 8 = System Utilization Fig th percentile node utilization vs. system utilization for various numbers of virtual servers per node. systemmarked utilization decrease in 99.9th percentile node utilization, as well as alesspronouncedincreasein99.9thpercentilenodeutilization as N grows Fig. 6. Load movem numbers of virtual serv 99.9th Percentile Node Utilization Num VS = = = = Fig th percen various numbers of vir capacities. 99.9th percentile node utilization increases roughly linearly with Assuming the syst be able to make all node utilization will B. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

19 Empirical Evaluation Number of Virtual Servers = 1 = Num VS = 2 = 4 = 8 = 12 Load Movement Factor System Utilization ilization and load od T, for various Fig. 6. Load movement factor vs. system utilization for various numbers of virtual servers per node. Increase in virtual servers looks good for load movement factor ilization Num VS = 2 = 4 = 8 = 12 B. Godfrey, K. Lakshminarayanan, S. Surana, 3 Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

20 Empirical 0.2 Evaluation System Utilization Heterogenous Fig. 6. Node Capacities lization and load d T, for various Load movement factor vs. system utilization for various numbers of virtual servers per node. 99.9th Percentile Node Utilization Num VS = 2 = 4 = 8 = Number of nodes in the system m utilization for Fig th percentile node utilization vs. number of nodes for various numbers of virtual servers per node, with homogeneous node capacities. Uses homogeneous node capacities and number of virtual servers tion, as well as Assuming the system state is such that our load balancer will nodeutilization Growsbe inable 99.9th to make percentile all nodes underloaded, of nodes the roughly 99.9th percentile linear in log N node utilization will simply be slightly below 1.0, sincethis. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

21 Empirical Evaluation Heterogenous Node Capacities 99.9th Percentile Node Utilization Num VS = 2 = 4 = 8 = 12 Probability Number of nodes in the system Fig th percentile node utilization vs. number of nodes for various numbers of virtual servers per node, with heterogeneous node Uses heterogeneous capacities (default Pareto capacity distribution). distribution 1e-05 0 Fig. 10. PDF of num an impulse of 10% o the ID space, at an i Achieves remarkable decrease in 99.9th percentile node Sys Util = 0.5 = utilization with growth in N = 0.7 = B. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30 ctions tor

22 Empirical Evaluation Node Arrivals and Departures Load Movement Factor Node InterarrivalTime = 10s = 30s = 60s = 90s Load movement factor Iden Uncorrela System Utilization Fig. 12. Load moved by the load balancer as a fraction of the load Fig. 14. Load movem by the DHT vs. system utilization, for various node interarrival choices of distributions o Load moved by the load balancer as a fraction of the load moved times. by DHT vs. system utilization 6 Node InterarrivalTime = 10s balancing algorithm r = 30s = 60s For the default modified to handle thi 5 12 virtual servers per node, = 90s the algorithm never directory-based approa moves more than 60% of the load compared to DHT. 4 Beneficial effect of Section V-D, having n nt Factor. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

23 Empirical Evaluation Node Arrivals and Departures System Utilization Fig. 12. Load moved by the load balancer as a fraction of the load moved by the DHT vs. system utilization, for various node interarrival times Fig. 14. Load movem choices of distributions Load Movement Factor Node InterarrivalTime = 10s = 30s = 60s = 90s balancing algorithm modified to handle thi directory-based appro Beneficial effect of Section V-D, having use fewer virtual serv case. It would be in impact of the degree servers needed to bal Number of Virtual Servers Fig. 13. Load moved by the load balancer as a fraction of the load moved by the DHT vs. number of virtual servers, for various rates of node arrival. by the DHT vs. number of virtual servers this environment by preferring to move virtual servers with high load and low movement cost when possible. V Most structured P2 object IDs are uniform number of objects per where N is the num improves this factor b (i.e., a node along w when deciding what p node. Chord [2] was servers as a means o Load moved by the load balancer as a fraction of the load moved B. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

24 Empirical Evaluation Object Movement Cost Identical Uncorrelated Load movement factor System utilization raction of the load s node interarrival = 10s = 30s = 60s = 90s Fig. 14. Load movement factor vs. system utilization for various choices of distributions of object load and object movement cost. Load movement factor vs. system utilization for two cases of object load balancing andalgorithm objectrun movement at our directories costwould have to be modified to handle this generalization, although our underlying directory-based approach should remain effective. Beneficial effect of heterogeneous capacities. As shown in B. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

25 Number of nodes in the system Empirical Evaluation Fig th percentile node utilization vs. number of nodes for various numbers of virtual servers per node, with heterogeneous node capacities (default Pareto distribution). Non-uniform Object Arrival Patterns Fig. 10. PDF of numb an impulse of 10% of the ID space, at an init Number of Emergency Actions Sys Util = 0.5 = 0.6 = 0.7 = 0.8 Load Movement Factor Impulse Spread Fig. 9. Number of emergency actions taken vs. fraction of ID space Fig. 11. Load moveme Impulse over which refers impulse to occurs, objects for various in a contiguous initial system utilizations. interval in inthe 10% of IDthe ID space space with aggregate load equalling 10% of total system load seconds. Since we fix the steady-state number of nodes in given threshold. Und Objects thearrival system to is 4096, tuneda node so that interarrival periodic timeload of10balancing seconds ment does cost not is propor corresponds to a node lifetime of about 11 hours. also attempts to mini run while emergency load balancing may be invoked To analyze the overhead of our load balancing algorithm assumption holds if s in this sectionload webalancing study the in Dynamic load moved Structured byp2p thesystem algorithm November as not19, be2013 true in 25 / the 30. Godfrey, K. Lakshminarayanan, S. Surana, R. Karp, I. Stoica

26 Empirical Evaluation Non-uniform Object Arrival Patterns System Utilization = 0.8 = 0.7 = 0.6 = 0.5 Probability e Number of emergency actions ber of nodes for erogeneous node Fig. 10. PDF of number of emergency actions that a node takes after an impulse of 10% of the system utilization concentrated in 10% of the ID space, at an initial system utilization of 0.8. PDF of number of emergency actions taken after an impulse of 10% concentrated 0.3 in 10% of the ID space Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30 tor

27 e-05 0 Empirical 5 Evaluation Non-uniform Object Arrival Patterns ber of nodes for erogeneous node Number of emergency actions Fig. 10. PDF of number of emergency actions that a node takes after an impulse of 10% of the system utilization concentrated in 10% of the ID space, at an initial system utilization of Load Movement Factor Num VS = 4 = 8 = System Utilization tion of ID space utilizations. r of nodes in of 10 seconds rs. Fig. 11. Load movement factor vs. system utilization after an impulse in 10% of the ID space. Load movement factor vs. system utilization after an impulse in 10% of the ID space given threshold. Under the assumption that an object s movement cost is proportional to its load, the balancer therefore also attempts to minimize total movement cost. But while that assumption holds if storage is the bottleneck resource, it might. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

28 Conclusion Conclusion Proposed a load balancing algorithm for dynamic, heterogeneous P2P systems Heterogeneity implies varying object loads node capacity continuous insertion and deletion of objects skewed object arrival patterns continuous arrival/departure of nodes Achieves load balancing for system utilizations of 90% while moving only 8% of the arriving load Moves less than 60% of the load the underlying DHT moves for node arrivals and departures Heterogeneity can help improving scalability. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

29 Discussion Discussion 1 Why are the times at which the nodes report to the directories not synchronized? 2 Glich in technical presentation in the Load Balancing Algorithm section! 3 How about reporting Directory utilization in Node(T, k e ) 4 Possible usage of Kalman Filters?. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30

30 THANK YOU!. Godfrey, K. Lakshminarayanan, S. Surana, Load R. Balancing Karp, I. Stoica in Dynamic Structured P2P System November 19, / 30