THE computational grid is a promising platform that

Transcription

1 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 19, NO., FEBRUARY Game-Theoretic Approach for Load Balancing in Computational Grids Riky Subrata, Member, IEEE, Albert Y. Zomaya, Fellow, IEEE, and Bjorn Landfeldt, Senior Member, IEEE Abstract Load balancing is a very important and complex problem in computational grids. A computational grid differs from traditional high-performance computing systems in the heterogeneity of the computing nodes, as well as the communication links that connect the different nodes together. There is a need to develop algorithms that can capture this complexity yet can be easily implemented and used to solve a wide range of load-balancing scenarios. In this paper, we propose a game-theoretic solution to the grid load-balancing problem. The algorithm developed combines the inherent efficiency of the centralized approach and the faulttolerant nature of the distributed, decentralized approach. We model the grid load-balancing problem as a noncooperative game, whereby the objective is to reach the Nash equilibrium. Experiments were conducted to show the applicability of the proposed approaches. One advantage of our scheme is the relatively low overhead and robust performance against inaccuracies in performance prediction information. Index Terms Game theory, grid computing, load balancing, scheduling. Ç 1 INTRODUCTION THE computational grid is a promising platform that provides large resources for distributed algorithmic processing [9]. Such platforms are much more cost-effective than traditional high-performance computing systems. However, computational grids have different constraints and requirements than those of traditional high-performance computing systems, such as heterogeneous computing resources and considerable communication delays. To fully exploit such grid systems, resource management and scheduling are key grid services, where issues of task allocation and load balancing represent a common problem for most grid systems. The load-balancing mechanism aims to equally spread the load on each computing node, maximizing their utilization and minimizing the average task execution time. In general, load-balancing algorithms can be classified as centralized or decentralized and static or dynamic. In the centralized approach (for example, [18]), one node in the system acts as a scheduler and makes all the load-balancing decisions. Information is sent from the other nodes to this node. In the decentralized approach (for example, [6]), all nodes in the system are involved in the load-balancing decisions. It is therefore very costly for each node to obtain and maintain the dynamic state information of the whole system. Most decentralized approaches have each node obtaining and maintaining only partial information locally to make suboptimal decisions.. The authors are with the Advanced Networks Research Group, School of Information Technologies, University of Sydney, Building J1, Sydney, NSW 006 Australia. {efax, zomaya, bjornl}@it.usyd.edu.au. Manuscript received 5 Aug. 006; revised 6 Dec. 006; accepted 14 May 007; published online 5 June 007. Recommended for acceptance by D. Trystram. For information on obtaining reprints of this article, please send to: tpds@computer.org, and reference IEEECS Log Number TPDS Digital Object Identifier no /TPDS Static load-balancing algorithms (for example, [15]) assume that all information governing load-balancing decisions that can include the characteristics of the jobs, the computing nodes, and the communication network are known in advance. Load-balancing decisions are made deterministically or probabilistically at compile time and remain constant during runtime. The static algorithms have one major disadvantage it assumes that the characteristics of the computing resources and communication network are all known in advance and remain constant. Such an assumption may not apply to a grid environment. In contrast, dynamic load-balancing algorithms (for example, [0] and [1]) attempt to use the runtime state information to make more informative load-balancing decisions. Undoubtedly, the static approach is easier to implement and has minimal runtime overhead. However, dynamic approaches may result in better performance. One of the major drawbacks of the dynamic algorithms is their sensitivity to inaccuracies in performance prediction information that the algorithm uses for load-balancing purposes. Some dynamic load-balancing algorithms are more sensitive to the inaccuracies and can generate extremely poor results even when the information accuracy is only slightly less than 100 percent; in real grid environments, however, 100 percent accuracy in information is very hard to achieve and maintain. The so-called hybrid scheduling is another area that has been receiving some attention. In terms of static and dynamic load balancing, a hybrid load balancer attempts to combine the merits of static and dynamic load-balancing algorithms and, by doing so, minimizes their relative inherent disadvantages. Note, however, that the definition between a static and a dynamic load-balancing algorithm in itself is not clear cut, and different authors use slightly different definitions of static and dynamic algorithms. A hybrid job-scheduling and load-balancing algorithm that combines the merits of static and dynamic algorithms was /08/$5.00 ß 008 IEEE Published by the IEEE Computer Society

2 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 19, NO., FEBRUARY 008 discussed in [3]. Also, in [1], a hybrid algorithm for adaptive load sharing in distributed systems was studied. Few works have been done using game-theoretic approaches and models for load balancing in a grid environment. A study on load balancing in distributed systems, formulating them as a noncooperative game with Wardrop equilibrium as the objective, was discussed in [14]. More recently, a noncooperative load-balancing game for distributed systems was presented in [10]; using the assumption of exponential service times and Poisson arrival, an algorithm for computing the Nash equilibrium was derived. However, none of these papers takes into account the communication delays in a grid environment that may affect the completion time of tasks. In this paper, we propose a game-theoretic-based solution to the grid load-balancing problem. The developed algorithm combines the inherent efficiency of the centralized approach and the fault-tolerant nature of the decentralized approach. The scheme can be considered semistatic, as it responds to changes in system states during runtime. However, it does not use as much information as traditional dynamic schemes; as such, it has relatively low overhead, is relatively insensitive to inaccuracies in performance prediction information, and is scalable and stable. The next section presents an overview of the system model including the grid and communication model that we are using. This is followed by the development of a game-theoretic algorithm to solve the grid load-balancing problems. The results section provides a number of detailed experiments that show the applicability of the proposed approaches. SYSTEM MODEL We assume that the computational grid system consists of a set of sites S connected by a communication network. In general, each site may contain multiple computing nodes, where each computing node may have single or multiple processors. The processors in the nodes are heterogeneous, meaning that they may have different processing power. Without loss of generality and to emphasize our main ideas, we assume that each site has one computing node equipped with a single processor; the processors in the different computing nodes have different processing powers. The sites s 1 ;...;s n in S are fully interconnected, meaning that there exists a communication path between any two sites ðs i ;s j Þ in S. Intersite communication is done via message passing, and the underlying network protocol guarantees that messages are received by the intended recipient in the order they are sent. As we are dealing with grid computing, the link is viewed as Internet links and modeled according to [5] and [17]. Our communication model represents network performance between a site s i to a site s j using two parameters a transmission delay e ij representing the start-up cost and contention delays at intermediate links on the path from s i to s j and a data transmission rate c ij representing the bandwidth available on the path from s i to s j. For a message of size m to be transmitted from site s i to s j, the transmission time is then given by L i;j ¼ e i;j þ m c i;j : ð1þ Fig. 1. Relationship of users, schedulers, and processors. e ij and c ij can be calculated from analytical models or historical information or dynamically forecasted by facilities such as the Network Weather Service (NWS) [7]. Each site s i in the grid system can represent one or a combination of the following:. User. This generates tasks to be executed by the processors. Each user sends a task to a scheduler to be scheduled for processing. Note that a user may send tasks to more than one scheduler.. Scheduler. This receives tasks from a set of users and assigns them to the processors in the grid system. Every time a task is received from a user, the scheduler decides which processor will process the task and send the task to that processor. Ideally, there would be many more users than schedulers in the system. As such, the tasks scheduled by the schedulers are an aggregate from many users.. Processor. This executes and processes tasks sent to it. Each processor has a queue that holds tasks to be executed; each task is then processed on a first-come, first-serve (FCFS) basis. Fig. 1 shows the relationship between users, schedulers, and processors. Note that a site in the grid system can be a user, a scheduler, and a processor all at the same time. That is, the site generates tasks that need processing, receives tasks from other users, schedules both its own tasks and others to the processors, and also executes and processes both its own tasks and others. Obviously, the tasks that are executed locally at the site will have minimal communication delay L i;j..1 Application Model The system consists of p users, n schedulers, and m processors. Each user k is assumed to generate tasks with average rate k (tasks per second) according to a Poisson process and independent of the other users. Tasks are then sent by the user to a scheduler that dispatches them to the processors (Fig. 1). Depending on the computational power of the processors, each processor j executes tasks at an average rate j (tasks per second). In our model, we do not assume a specific distribution for task execution time; rather, the task execution time of the applications running on the system can take any distribution (having finite mean

3 SUBRATA ET AL.: GAME-THEORETIC APPROACH FOR LOAD BALANCING IN COMPUTATIONAL GRIDS 3 and variance). Each processor can therefore be modeled as an M/G/1 queuing system. Note that we assume that the task distribution of the applications, once chosen, is consistent throughout the system. For stability, we have the condition/constraint that jobs must not be generated faster than the system can process them (otherwise, the queues will build up to infinity): X n i < Xm where i is the average arrival rate of tasks (in tasks per second) at scheduler i and j is the average processing rate of tasks at processor j. Each scheduler i then sends a fraction r i;j of their jobs to each processor j, whereby X m r i;j 0; r i;j ¼ 1: For stability, the rate of jobs sent to a processor j must not exceed the rate at which jobs can be executed by the processor j (otherwise, the queue at processor j will build up to infinity): X n j ; r i;j i < j :. Objective Function We model the grid load-balancing problem as a noncooperative game, whereby the objective is to reach the Nash equilibrium. In this game, the players are the schedulers and each player tries to minimize its own average task completion time independently. Specifically, each scheduler calculates a strategy r i ¼ r i;1 ;r i; ; ;r i;m such that its average task completion time is minimized. Note that the task completion time includes communication delays, the waiting time at the queue, and the task processing time itself. A strategy r 0 i is always preferred over a strategy r i if it results in a lower average task completion time. For an M/G/1 queuing system [7], the average processing time of a task including the waiting time at the queue at a processor j is given by h j F j ¼ h j þ j ; ð6þ 1 h j j where h j ¼ 1 j is the mean of the job execution distribution, h j is the second moment of the distribution, and j is the average arrival rate of tasks (in tasks per second) at processor j. In our model, a scheduler sends a proportion of its tasks to each processor. That is, each processor j receives tasks from multiple schedulers; therefore, j is a combination of task arrivals from the different schedulers. This leads to the following equation: ðþ ð3þ ð4þ ð5þ h j Pn r i;j i F j ¼ h j þ : ð7þ 1 h j Pn r i;j i Further, scheduler i is connected to a processor j via a link with capacity c i;j in bits/s. Each task is assumed to require an average of b bits of data to be transferred. Using (1), the expected transfer time of a task from player i to processor j is therefore given by L i;j ¼ e i;j þ b : ð8þ c i;j This value represents the average communication delay if a task is to be sent from player i to processor j. The completion of a task involves the execution time of the task, the waiting time at the queue, and the communication latencies and transfer time of the task to the processor. Our objective is to minimize the average completion time of tasks. Using (7) and (8), the average completion time of tasks for player i is given by D i ¼ Xm ¼ Xm r i;j F j þ L i;j " r i;j h j Pn r k;j k # k¼1 r i;j h j þ þ e i;j r i;j þ b r i;j : 1 h j Pn c r k;j i;j k k¼1 ð9þ In our model, each scheduler acts independently of the other schedulers. We introduce a new variable j;i, shown in (10). j;i defines the computational power of processor j that is available to player i and can be estimated for each processor j: Using (10), (9) becomes D i ¼ X m h r i;j h j þ j;i ¼ j Xn k¼1; k6¼i r k;j k : ð10þ r i;j h j r i;j i þ 1 h j;i j þ e i;j r i;j þ b r i i;j : h j j;i r i;j i c i;j ð11þ Equation (11) is the objective function that each scheduler aims to minimize independently and subject to the constraints of (3), (4), and (5). Note that D i is a function of r i;j. The Nash equilibrium for the game is a strategy profile r ¼ fr 1 ; r ; ; r n g in which no player (scheduler) can decrease its average task completion time by unilaterally changing its strategy. There exists a unique (pure) Nash equilibrium for the game because the expected response time functions (see (11)) are continuous, convex, and increasing [10], [3]. As each scheduler acts independently of the other schedulers, to reach the Nash equilibrium, the following process is used: Each scheduler periodically calculates a strategy r i that results in minimum D i (11). This strategy is the best reply given the current state of the system. Each scheduler periodically updates its strategy until an equilibrium is reached (no scheduler wants to change its strategy as it results in an increase in its D i ). The system will then remain in equilibrium until there are changes in the system s states. Periodic scheduling by the

4 4 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 19, NO., FEBRUARY 008 TABLE 1 Relative Processing Power of the Processors TABLE Relative Task Arrival Rate of Each Player schedulers ensures that optimum strategies for each scheduler are maintained. Note that, at each scheduling instant, a best reply strategy is employed by each scheduler. Whether or not such best reply strategy converges to the Nash equilibrium remains an open problem [4]. In the next section, experiments are conducted with different parameters that show convergence for more than two players. The constrained minimization problem has the solution shown below. Equation (1) is the average completion time for an M/G/1 queuing system [7], whereas (13) governs the proportion of tasks each scheduler should send to each processor. Further details are shown in the Appendix. Best reply load balancer. Order/sort the processors according to potential completion time such that t 1;i <t ;i <...<t m;i ; t j;i is given by t j;i ¼ h j h j þ þ L i;j : ð1þ h j h j j;i We then have qffiffiffiffiffiffiffiffiffiffiffi r i;j ¼ h j;i j j;i r ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi i i h j h j þ h ; 1 i d i : ð13þ j h j L i;j is given by the following equation: qffiffiffiffiffiffiffiffiffiffiffi X di j;i i ¼ Xdi h r j j;i ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi h j h j þ h : ð14þ j h j L i;j d i ð1 d i mþ is the maximum positive integer that satisfies the following inequality: X d i Xd i j;i i qffiffiffiffiffiffiffiffiffiffiffi h s j j;i ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi : h j h j þ h j þh j h di h d i L h i;j þ L i;di þ h d i di 3 EXPERIMENTS h j h di di;i ð15þ In this section, we analyze the effects of different strategies to the average completion time of tasks given by (11). To analyze the algorithm presented above, a set of networks and applications were generated. The proposed algorithm is labeled GT. The proportional-scheme algorithm discussed in [6] is also implemented for comparison. The proportional-scheme algorithm allocates tasks to processors in proportion to its computing power (task processing rate); the faster processors are sent more tasks by the players. The proportion of tasks sent to the processors is given by the following: r i;j ¼ j;i P m j;i : ð16þ The proportional-scheme algorithm is labeled PS. The PS algorithm is a distributed, decentralized algorithm; note that the PS algorithm does not take into account the communication delays incurred in transferring tasks from one site to another. The algorithms were then applied to the generated set. In the first set of experiments, we assume that the task service times follow an exponential distribution. The parameters used for the experiments are given below:. The generated network contains m ¼ 15 processors with relative processing power as shown in Table 1. Along with the exponential service times for tasks, the values in Table 1 represent the average processing rate of each processor.. Each task requires, on the average, 1 Mbyte of data to be transferred. The communication links have bandwidth in the range of [15, 00] Kbytes/s. Further, each communication link has a latency in [0.01, 1] seconds. These values are chosen to show the effect of communication delays on the average task completion time. As the bandwidths are increased, the effects of communication delays will diminish.. There are n ¼ 10 schedulers in the system with an arbitrary p number of users. Normally, the number of users would be much greater than the number of schedulers such that p n. The relative job arrival rate for each player is shown in Table. The actual arrival rate i of each player is calculated to give the required overall average system loading and is given by i ¼ i Xm j ; ð17þ where i is the relative task arrival rate of player i. 3.1 Convergence to Equilibrium of the GT Algorithm In the first set of experiments, the average system load is set to ¼ 0:6. The algorithm is then run where the initial strategy r i of each player i is the zero vector. Each player then refines and updates its strategy at each scheduling run. For simplicity, in the experiments, the schedulers update in a sequential manner. An interesting case occurs when the parameters of the system do not change (that is, they are static). In this case, we expect the system to reach a (Nash) equilibrium, whereby no player has a tendency to unilaterally change its strategy (as noted earlier, a unique Nash

5 SUBRATA ET AL.: GAME-THEORETIC APPROACH FOR LOAD BALANCING IN COMPUTATIONAL GRIDS 5 Fig.. Convergence of the GT algorithm. equilibrium does exist in the game). Such a result is shown in Fig.. As shown in the graph, the algorithm converges to a (Nash) equilibrium in 1 iterations (note the log scale of the y-axis); in this paper, we assume that convergence has occurred when the overall percentage change is ). In terms of the periodic scheduling done by each scheduler, an equilibrium is reached when the calculated strategy does not change from one period to another. However, changes to the strategy may be needed in the next period due to changes in the system s states. In each scheduling period, the GT algorithm is executed once by the players, which takes a relatively small computation time. As such, the algorithm requires relatively small computation time by the players. The average task completion time for each player, when the system is at equilibrium, is shown in Fig. 3. The completion time includes the communication delay (transfer time), the waiting time at the queue, and the execution time of the task itself. In the graph, the average job/task time for each player, for both the GT and PS schemes, is normalized by dividing each player s value by the average job time of all the players for the GT scheme. That is, for the GT scheme, each player s value is divided by the average of all the players values of the GT scheme. Similarly, for the PS scheme, each player s value is also divided by the average of all the players values of the GT scheme. As can Fig. 3. Average task completion time for each player. Fig. 4. Number of iterations versus system utilization. be seen from the graph, each player may have different average task completion times (or different payoffs), although the system is in equilibrium; of particular importance is the effect of the communication delay on the overall task completion time. For the PS scheme, we expect the same execution time and waiting time for all players; however, differences in the communication delay between each player and each processor results in differences in the average task completion time between players. Fig. 3 also shows that GT yields better performance than PS for every player (players 1 to 10). Also, note that a player with a lower (or higher) value in the GT scheme does not necessarily have a lower (or higher) value in the PS scheme. 3. Effect of System Loads In this set of experiments, we vary the average load of the system from 0.1 to 0.9. The same set of processors and players are used as in the previous set of experiments. The arrival rate of tasks for each player is then adjusted to give the required average system load. Fig. 4 shows the number of iterations required to reach equilibrium. As can be seen from the graph, as the load on the system increases, the system takes longer to reach equilibrium. At 10 percent load, the system takes six iterations to reach equilibrium, whereas, at 90 percent load, 47 iterations are needed to reach a state of equilibrium. Also, note that, although there is a general trend of increasing iterations (to reach equilibrium) as the system load increases, the increasing trend does not represent a smooth curve over the entire region, and in general, the number of iterations required to reach equilibrium is not predictable. Fig. 5 shows the normalized average job completion times as the system load is varied from 0.1 to 0.9. Similarly, Fig. 6 shows the normalized average job completion times versus system load, but this time with no communication delay, for comparison purposes. As before, the job completion times are normalized by dividing each completion time by the overall average completion time of the GT scheme. In the two graphs, we see a rapid increase in the systemwide average job completion time as the system nears full capacity. This trend is explained by and is a consequence of (11). As the system nears full capacity, the average queue length at the processor gets longer and longer, and as a

6 6 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 19, NO., FEBRUARY 008 Fig. 5. Average task s completion time versus system load. Fig. 7. Average task s completion time for each player. Fig. 6. Average task s completion time versus system load with no communication delay. result, the average completion time will be governed by the average waiting time at the queue; that is, the effect of communication delays and service times becomes less pronounced at high system load. Both the GT and PS schemes show the same trend, although the GT scheme gives lower expected completion times. 3.3 Effect of Communication Delay In this set of experiments, we investigate the effect of link bandwidth and latencies on task completion times. For the first part of this experiment, we show the differences between the GT and PS scheme when there is no communication delay. The result, as shown in Fig. 7, clearly shows that, even when communication delays are not a factor, the GT scheme still performs better for each player than the PS scheme. To this end, we vary the bandwidth of the communication links between the players and the processors. Specifically, the communication bandwidth to each processor from players 3 and 7 is varied. We use the same set of players, processors, and parameters as in the previous experiment and set the average system load to 60 percent. The results are shown in Fig. 8. Fig. 8a shows the normal graph showing the average job time for each player. In Fig. 8b, the communication bandwidth to each processor from players 3 and 7 is reduced on the average by 100 percent to [7.5, 100] Kbytes/sec. In Fig. 8c, the communication bandwidth is further reduced by an average of 100 percent to [3.75, 50] Kbytes/s. As expected, the graph shows the average job time for players 3 and player 7 increases as the average bandwidth to the processors for players 3 and 7 are decreased; this can be clearly seen in Fig. 8c. Although both the GT and PS schemes show an increase in the average job time for players 3 and 7, the effect on the other players is not the same. For the PS scheme, the other players are not affected and have the same average job completion times from Figs. 8a to 8c, as the strategies for players 3 and 7 remain unchanged as their communication bandwidths are decreased. For the GT scheme, however, the strategies for players 3 and 7 may change slightly as the communication bandwidths are changed. As a result, the other players may need to change their strategies to best respond to the changes in strategies of players 3 and 7. The end effect is that a change in communication bandwidth of a player may have an indirect effect on the average job completion time of the other players. This can be seen in Fig. 8, as the average job completion times of the other players also change from Figs. 8a to 8c. 3.4 Effect of System Size In this part of the experiment, we vary the number of processors available in the system and investigate its effect on the convergence of the GT scheme and the average task completion time of each of the players for both the GT and PS schemes. To this end, we vary the number of processors in the system from 10 to 0. The processing rate of each of the processors is shown in Table 3. As can be seen, the latter processors get increasingly more powerful, with the last processor (processor 0) being the most powerful and 0 times more powerful than the slowest processor in the system. As such, we are investigating the effects on convergence and average task completion times of adding more powerful processors into the system. The same set of players and parameters are used as in the previous set of experiments. The average system load is then kept at a constant 60 percent. This is achieved by increasing the total number of tasks being generated in the system as the number of processors is increased, as per (17).

7 SUBRATA ET AL.: GAME-THEORETIC APPROACH FOR LOAD BALANCING IN COMPUTATIONAL GRIDS 7 Fig. 8. Average task s completion time for each player. TABLE 3 Relative Processing Power of the Processors We first investigate the effect of system size on the convergence rate of the GT algorithm. Fig. 9 shows the number of iterations required to reach equilibrium as the number of processors in the system is increased from 10 to 0. As can be seen from the figure, the number of iterations required remains relatively stable as the number of processors are varied between 10 and 0. This is in contrast to Fig. 4, which shows an increase in the required number of iterations as the average system load increases from 0.1 to 0.9. In this experiment, where the average system load is kept at 0.6 but the number of processors is varied between 10 and 0, the required number of iterations remains relatively stable. Next, we investigate the effect of system size on the average task completion time of the players. As can be seen in Fig. 10, for both the GT and PS schemes, the expected task completion time decreases as the number of processors in the system increases. This is especially true as each processor being added is either as powerful as or more powerful than the existing processors in the system. Fig. 10 also shows that the GT schemes results in a lower overall average task completion time than the PS scheme over system sizes ranging from 10 to 0 processors. This shows that an efficient allocation of tasks to the processors is important in grid systems having multiple processors; what is remarkable is that this can be achieved in a distributed manner, with each scheduler (10 schedulers in this case see Table ) minimizing its own task completion time. 3.5 Effect of Service Time In the previous experiments, we assume that the service time of tasks in the processors follows an exponential distribution. It has been suggested in the literature [], [11], [1], [4], [5], however, that the service time of tasks for certain applications follows a heavy-tailed distribution, instead of an exponential distribution. One of the most common distributions used to model such a heavy-tailed distribution is the Bounded Pareto distribution, which is a bounded version of the Pareto distribution [8], [19]; having a bounded range of possible values, the Bounded Pareto has finite first and second moments. As such, in this part of the experiments, we investigate the effects that Bounded-Pareto-distributed service times of tasks may have on the convergence of the GT algorithm and the average task completion time for the players. Fig. 9. Number of iterations versus system size. Fig. 10. Average task s completion time versus system size.

8 8 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 19, NO., FEBRUARY 008 TABLE 4 Relative Processing Power of the Processors The Bounded Pareto distribution is characterized by the following probability density function (pdf): fðxþ ¼ k x 1. k x p; ð18þ 1 k p where k is the minimum job execution time and p is the maximum job execution time; the pdf is zero beyond these bounds. The parameter defines the shape of the hyperbolic curve of the distribution. The mean (first moment) of the distribution is given by h ¼ k k p k 1 1 p 1 ð19þ and the second moment is given by h ¼ k 1. 1 k p k 1 p : ð0þ As before, we use 15 processors in this set of experiments. The parameters used for each of the processors are shown in Table 4. The second row in Table 4 shows the k value for each processor, and the last row in Table 4 shows the p value for each processor. As can be seen in the table, the biggest tasks can take a longer time by several orders of magnitude than the smallest tasks. The hyperbolic curve parameter of the Bounded Pareto distribution is then set to ¼ 1:1. As gets closer to 1.0, the variability of the distribution increases and leads to interesting cases. Table 5 summarizes the above values in terms of the expected task execution time at each of the processors; Table 5 also shows the variance of the task execution time at each of the processors. The same set of players and parameters are then used as in the previous set of experiments. We then vary the system loads from 0.1 to 0.9. We then investigate the effect of the Bounded Pareto service time of tasks on the number of iterations required to reach equilibrium for the GT scheme. The results of the experiment, shown in Fig. 11, show that the GT scheme does converge to an equilibrium when the service time of tasks follow a Bounded Pareto distribution. As can be seen from the figure, the required number of iterations increases as the system load increases from 0.1 to 0.9 as expected; the results are similar to earlier results shown in Fig. 4. The effect of the Bounded Pareto service times on the overall average task completion times are shown in Fig. 1. As can be seen in the figure, there is a rapid increase in the systemwide average job completion time as the system nears full capacity. More importantly, this increase is more rapid than the previous result shown in Fig. 5, where the service times follow an exponential distribution. Note that, as before, this behavior in the trend is as a result of (11); this means that there will always be an increase in the rate of change of the average system time as the system nears full capacity. The GT scheme gives a lower overall task completion time than the PS scheme. 3.6 Fairness of Schemes In this part of the experiment, we investigate the fairness of each of the different schemes. Fairness is achieved when the average task completion time for each of the players is the Fig. 11. Number of iterations versus system utilization. TABLE 5 Relative Processing Power of the Processors Fig. 1. Average task s completion time versus system load.

9 SUBRATA ET AL.: GAME-THEORETIC APPROACH FOR LOAD BALANCING IN COMPUTATIONAL GRIDS 9 Fig. 13. Fairness versus system load. same. If one player has a lower average task completion time and another has a higher average task completion time, then the scheduling scheme can be considered unfair, as it gives some players an advantage and other players a disadvantage. Fairness is another important performance measure for load-balancing schemes besides the average task completion time. A load-balancing scheme that results in a few schedulers having extremely long delays may not be preferable, as these few schedulers would become unwanted schedulers for the users in the system though they may be arbitrary. A fairness index given by FI ¼ P n T i n Pn Ti ; ð1þ where T i is the average task completion time of player i, was discussed in [13] to quantify the fairness of loadbalancing schemes. If a load-balancing scheme is 100 percent fair, then FI is 1.0. A fairness index close to 1.0 indicates a relatively fair load-balancing scheme. In order to judge the relative fairness of the different schemes, we remove the effect of latency and communication bandwidth on the completion time of tasks. These communication delays are specific to each player in the sense that a player may have higher task completion time due to the low communication bandwidth available to that particular player. As such, for the purpose of evaluating fairness, we set all data transfers to instantaneous (that is, zero communication delays). We then use the same set of processors, players, and parameters as in the previous experiment. This means that the service time of tasks follow a Bounded Pareto distribution. In the first part of the experiment, we vary the average system load from 0.1 to 0.9. The results are shown in Fig. 13. Fig. 14. Fairness versus system size. As can be seen in the figure, the PS scheme has a fairness index of 1.0 across the entire utilization range from 0.1 to 0.9. This is the inherent advantage of the PS scheme even though it is a distributed, decentralized scheme, it guarantees the same average task completion time for each of the players. As shown in the figure, the GT scheme decreases in fairness as the system nears full capacity. However, the fairness index at 90 percent system load is still above 0.99, and depending on the requirement of the application, this value may well be above the minimum acceptable level. In the next set of experiments, we set the average system load to 60 percent and vary the number of processors in the system from 10 to 0. The parameters used for each of the processors are given in Table 3. The results are shown in Fig. 14. As in the previous experiment, the PS scheme gives a fairness index of 1.0 as the number of processors is varied from 10 to 0. The GT scheme shows some variations as the number of processors is varied. As before though, the fairness index in all of the cases is above 0.99, which, depending on the application, may well be above the minimum acceptable level. In this part of the experiment, we change the set of players from a highly heterogeneous set of players shown in Table, where the most active players generate 35 times more tasks than the least active players, to a less heterogeneous set of players, as shown in Table 6. The results are shown in Figs. 15 and 16. As can be seen in the figures, using a less heterogeneous set of players has improved the fairness of the GT scheme as compared to the previous set of experiments. 4 CONCLUSION This paper discussed a game-theoretic framework and algorithm to solve the grid load-balancing problem. The algorithm developed combines the inherent efficiency of the centralized approach and the fault-tolerant nature of the TABLE 6 Relative Task Arrival Rate of Each Player

10 10 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 19, NO., FEBRUARY 008 Fig. 15. Fairness versus system load. decentralized approach. The scheme is semistatic and responds to changes in system states during runtime. Further, the algorithm does not assume any particular distribution for service times of tasks, it only requires the first and second moments of the service times as input. Experiments were conducted to show the applicability of the proposed approaches. One advantage of our scheme is the relatively low overhead and robust performance against inaccuracies in performance prediction information. APPENDIX In this section, we show the correctness of the solution given by (1), (13), (14), and (15). Our objective is to minimize the objective function given by (11), subject to the constraints given by (3), (4), and (5). First, within the constraints, it can be shown i;j 0 D i 0. i;j such, (11) is a convex function; similarly, the constraint functions are all convex functions. Therefore, the first-order Karush-Kuhn-Tucker conditions are necessary and sufficient for optimality [16], []. First, we minimize (11) subject to the equality constraint (4) only, as it is clear that constraint (5) is inactive, and we will treat the inequality constraint (3) as a special case. The Lagrangian is given by L ¼ X m r i;j h j r 3 i;j i þ 1 h j;i 4r i;j h j þ j þ r i;j L i;j r i;j 5þ : h j j;i r i;j i ðþ The necessary i;j ¼ 0; ð3þ Fig. 16. Fairness versus system size. Solving for r i;j in (5) gives the partial answer for (13): qffiffiffiffiffiffiffiffiffiffiffi r i;j ¼ h j;i j j;i r ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi i i h j h j þ h : ð6þ j h j L i;j The Lagrange multiplier needs to be chosen so that the equality constraint (4) is satisfied. Using (6), (4), and some manipulations, we get (14). We now need to satisfy the inequality constraint (3). Using the above results so far, we know that one or more r i;j will be positive due to constraint (4). At the crossroad, we have r i;j ¼ 0. Setting r i;j ¼ 0 into (5) gives ¼ h j h j þ þ L i;j : ð7þ h j h j j;i Notice that the right-hand side of (7) is the expected task completion time of processor j. If we include all m processors in (6), then we may get r i;j < 0; these are due to processors with a high expected completion time as given by the righthand side of (7). Therefore, some processors may need to be excluded, that is, we set r i;j ¼ 0 for these processors. We therefore sort the processors according to their expected task completion time as given by (1) and exclude processors having an expected task completion time greater than the necessary threshold. Simply put, one can start with all the processors and gradually decrease the number of processors (threshold index) until we have all r i;j 0. The threshold index is given in a compact form by (15). ACKNOWLEDGMENTS This work is funded by the Smart Internet Technology CRC ( ¼ 0: Substituting () into (3), we get the following: h j h j þ j;i h j þ L i;j ¼ : h j j;i r i;j i h j ð4þ ð5þ REFERENCES [1] M. Avvenuti, L. Rizzo, and L. Vicisano, A Hybrid Approach to Adaptive Load Sharing and Its Performance, J. Systems Architecture, vol. 4, pp , [] N. Bansal and M. Harchol-Balter, Analysis of SRPT Scheduling: Investigating Unfairness, Proc. ACM Int l Conf. Measurement and Modeling of Computer Systems (SIGMETRICS 01), pp , 001.

11 SUBRATA ET AL.: GAME-THEORETIC APPROACH FOR LOAD BALANCING IN COMPUTATIONAL GRIDS 11 [3] C. Boeres, A. Lima, and V.E.F. Rebello, Hybrid Task Scheduling: Integrating Static and Dynamic Heuristics, Proc. 15th Symp. Computer Architecture and High Performance Computing, pp , 003. [4] T. Boulogne, E. Altman, and O. Pourtallier, On the Convergence to Nash Equilibrium in Problems of Distributed Computing, Annals of Operation Research, pp , 00. [5] H. Casanova and L. Marchal, A Network Model for Simulation of Grid Application, Research Report 00-40, 00. [6] Y.C. Chow and W.H. Kohler, Models for Dynamic Load Balancing in a Heterogeneous Multiple Processor System, IEEE Trans. Computers, vol. 8, pp , [7] R.B. Cooper, Introduction to Queueing Theory, second ed. Elsevier, [8] M.E. Crovella, M. Harchol-Balter, and C.D. Murta, Task Assignment in a Distributed System: Improving Performance by Unbalancing Load, Technical Report BUCS-TR , [9] I. Forster and C. Kesselman, The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann, [10] D. Grosu and A.T. Chronopoulos, Noncooperative Load Balancing in Distributed Systems, J. Parallel and Distributed Computing, vol. 65, pp , 005. [11] M. Harchol-Balter, Job Placement with Unknown Duration and No Preemption, Performance Evaluation Rev., vol. 8, pp. 3-5, 001. [1] M. Harchol-Balter and A.B. Downey, Exploiting Process Lifetime Distributions for Dynamic Load Balancing, ACM Trans. Computer Systems, vol. 15, pp , [13] R. Jain, D. Chiu, and W. Hawe, A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems, DEC Research Report TR-301, [14] H. Kameda, J. Li, C. Kim, and Y. Zhang, Optimal Load Balancing in Distributed Computer Systems. Springer, [15] C. Kim and H. Kameda, An Algorithm for Optimal Static Load Balancing in Distributed Computer Systems, IEEE Trans. Computers, vol. 41, pp , 199. [16] H.W. Kuhn and A.W. Tucker, Nonlinear Programming, Proc. Second Berkeley Symp. Math. Statistics and Probability, pp , [17] A. Legrand, L. Marchal, and H. Casanova, Scheduling Distributed Applications: The SimGrid Simulation Framework, Proc. Third IEEE/ACM Int l Symp. Cluster Computing and the Grid, pp , 003. [18] H.-C. Lin and C.S. Raghavendra, A Dynamic Load-Balancing Policy with a Central Job Dispatcher (LBC), IEEE Trans. Software Eng., pp , 199. [19] M.O. Lorenz, Methods of Measuring the Concentration of Wealth, Publications of the Am. Statistical Assoc., vol. 9, pp , [0] K. Lu, R. Subrata, and A.Y. Zomaya, An Efficient Load Balancing Algorithm for Heterogeneous Grid Systems Considering Desirability of Grid Sites, Proc. 5th IEEE Int l Performance Computing and Comm. Conf. (IPCCC 06), 006. [1] K. Lu, R. Subrata, and A.Y. Zomaya, Towards Decentralized Load Balancing in a Computational Grid Environment, Proc. First Int l Conf. Grid and Pervasive Computing (GPC 06), 006. [] D.G. Luenberger, Linear and Nonlinear Programming, second ed. Addison-Wesley, [3] A. Orda, R. Rom, and N. Shimkin, Competitive Routing in Multiuser Communication Networks, IEEE/ACM Trans. Networking, vol. 1, pp , [4] I.A. Rai, G. Urvoy-Keller, and E.W. Biersack, Analysis of LAS Scheduling for Job Size Distributions with High Variance, Proc. ACM Int l Conf. Measurement and Modeling of Computer Systems (SIGMETRICS 03), pp. 18-8, 003. [5] A. Riska, E. Smirni, and G. Ciardo, Analytic Modeling of Load Balancing Policies for Tasks with Heavy-Tailed Distributions, Proc. Second ACM Int l Workshop Software and Performance (WOSP 00), pp , 000. [6] N.G. Shivaratri, P. Krueger, and M. Singhal, Load Distributing for Locally Distributed Systems, Computer, pp , 199. [7] R. Wolski, N.T. Spring, and J. Hayes, The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing, J. Future Generation Computer Systems, vol. 15, pp , Riky Subrata received the BE degree (Hons 1) in electrical and electronic engineering, the BCom degree in 000, and the PhD degree from the School of Electrical, Electronic, and Computer Engineering, University of Western Australia, in 003. He is currently a research fellow in the School of Information Technologies, University of Sydney. His current research interests include high-performance computing, distributed algorithms, and mobile computing. He is a member of the IEEE. Albert Y. Zomaya is currently the head of school and the CISCO Systems chair professor of internetworking in the School of Information Technologies, University of Sydney. Prior to joining Sydney University, he was a full professor in the Electrical and Electronic Engineering Department, University of Western Australia, where he also led the Parallel Computing Research Laboratory during 1990 to 00. His research interests are in the areas of highperformance computing, parallel algorithms, mobile computing, and bioinformatics. He is the author or coauthor of six books and more than 300 publications in technical journals and conference proceedings and the editor of seven books and eight conference proceedings volumes. He is currently an associate editor of 15 journals, the founding editor of the Wiley Book Series on Parallel and Distributed Computing and a founding coeditor of the Wiley Book Series on Bioinformatics. He was the chair the IEEE Technical Committee on Parallel Processing ( ) and currently serves on its executive committee. He has been actively involved in the organization of national and international conferences. He is a chartered engineer (CEng), a fellow of the American Association for the Advancement of Science, the IEEE, the IEEE Computer Society, and the Institution of Electrical Engineers (UK), and a distinguished engineer of the ACM. He received the 1997 Edgeworth David Medal from the Royal Society of New South Wales for outstanding contributions to Australian science. He is also the recipient of the Meritorious Service Award in 000 and the Golden Core Recognition in 006, both from the IEEE Computer Society. Bjorn Landfeldt received the BSc-equivalent degree from the Royal Institute of Technology, Sweden, and the PhD degree from University of New South Wales in 000. In parallel with his studies in Sweden, he was running a mobile computing consultancy company, and after his studies, he joined Ericsson Research, Stockholm, as a senior researcher, where he worked on mobility management and quality-of-service (QoS) issues. In November 001, he took up a position as a CISCO senior lecturer in Internet technologies at the University of Sydney with the Schools of Electrical and Information Engineering and the School of Information Technologies. His research interests include wireless systems, systems modeling, mobility management, QoS, and service provisioning. He has been awarded eight patents in the US and globally. He has published more than 50 publications in international conferences, journals, and books and been awarded many competitive grants, such as Australian Research Council (ARC) discovery and linkage grants. He is also a research associate of National ICT Australia (NICTA) and the Smart Internet CRC. Currently, he is serving on the editorial boards of several international journals, is a program member of many international conferences, and is supervising eight PhD students. He is a senior member of the IEEE.. For more information on this or any other computing topic, please visit our Digital Library at