Two-Level Scheduling Technique for Mixed Best-Effort and QoS Job Arrays on Cluster Systems

Transcription

1 Two-Level Scheduling Technique for Mixed Best-Effort and QoS Job Arrays on Cluster Systems Ekasit Kijsipongse, Suriya U-ruekolan, Sornthep Vannarat Large Scale Simulation Research Laboratory National Electronics and Computer Technology Center ABSTRACT We consider the problem of dynamic scheduling of mixed best-effort and QoS job arrays on cluster computers. Each job array consists of many independent jobs that arrive into the cluster over time. The information of job arrays, i.e. the number of jobs and the processing time of each job, are not known to the scheduler until they arrive. A user submits a job array in either the best-effort or QoS type. In case of the best-effort type, the scheduler accepts the job array and allocates approximately equal amount of computing resources to all users under fair sharing policy. For a QoS job array, which all jobs in the job array must be finished by the deadline, the scheduler performs the online admission control to immediately notify the user if the newly QoS job array is accepted or rejected. The scheduler must reorganize the existing schedule accordingly to ensure that the fairness of the best-effort and the deadlines of all accepted QoS job arrays are always guaranteed. We propose the online two-level scheduling technique to schedule best-effort and QoS job arrays on shared cluster computing resources to meet diverse requirements from users. Keywords: Job Scheduling, Cluster Computing, Admission Control. 1. INTRODUCTION The continuous expansion of user groups in High Performance Computing (HPC) has given a significant impact on the design of the job scheduling and management module, an essential part of an HPC system. The job scheduler must be able to response to the diversity of the user groups and their applications in finding an efficient resource allocation so that users can gain the most profit out of the HPC resources while being fair to other users. At present, there exists another type of applications called the parameter sweep applications which the applications are composed of a number of independent sequential jobs. Each job performs the same task on different parameters or subsets of data. Since the jobs are independent, there is no communication or dependencies among them, and thus makes this type of applications suitable to be executed on a loosely-coupled distributed HPC such as cluster computing system. These applications can often be found in data mining, simulation, computation biology or animation rendering; each of which may consist of thousands of independent jobs. In the context of job scheduler, relevant jobs in the parameter sweep applications are submitted in group as a job array. Therefore, the new job scheduler should support these parameter sweep applications in addition to the sequential and parallel jobs found in typical HPCs. The diversity of the user groups imposes a problem to the job scheduling and management on HPC as well since their requirements for the HPC are often different. In general, two main user groups can be identified at any HPC centers. The first group consists of users who need their jobs to be executed with QoS guarantee, a primary requirement from industrial users. In most cases, a QoS job is submitted with a given deadline. The scheduler must find ways to allocate HPC resources to complete the QoS jobs by the deadlines and this must be transparent to the users. The

2 immediate notification is also another required feature of a job scheduler. The faster the users know if the job will or will not be completed by its deadline, the better they can make further decision. The second group consists of users who use HPC for academic or testing purpose. They are provided with the best-effort policy such that their jobs will be executed if the computing resources are not occupied by the QoS jobs. The scheduler should prevent a particular best-effort user from dominating the HPC resources and it should allocate resources equally to all of them in this group. As there exist both QoS and best-effort jobs waiting for resources, the job scheduler must select an appropriate policy to allocate the computing resources to satisfy the requirements of all users. This paper presents the design and implementation of an two-level scheduler with admission control to support the resource allocation for aforementioned QoS and best-effort job arrays. It is a dynamic scheduler that works in the online mode which the arrival of jobs is not known in advance. The rest of the paper is organized as follows. Section 2 gives an overview of relevant work in this area. Section 3 presents the design and implementation of the two-level scheduler with admission control. Section 4 draws conclusions and discusses the future work. In this paper, the terms job and job array will be used interchangeably unless they needs to be specific. 2. RELATED WORKS Online deadline scheduling on a single machine has been introduced in [1] where they emphasized on the immediate notification ability of job admission control. Similarly, scheduling jobs with equal processing time on multiple identical machines is discussed in [2, 3, 4]. Kim and Chwa [5] considered the online deadline scheduling on multiple resources but without admission control, which makes the job able to be rejected anytime before its deadline. The study of job array scheduling also appears in the context of scheduling Bag-of-Task (BoT) applications [6, 7, 8]. However, they dedicated to the BoT scheduling to meet a single objective such as minimizing makespan or minimizing the maximum completion time rather than to meet distinct requirements for different user groups as in our work. Anderson et al. [9] studied the offline scheduling for job arrays in animation rendering to maximize the utility value and meet their deadlines. The problem of job scheduling with mixed workload containing a steady incoming stream of best-effort and real-time jobs on clusters is presented in [10]. Libra [11] is a job scheduler on cluster systems that can provide immediate notification to users but it requires that at least a node is available to execute the job immediately. SGE [12], Condor [13], and Maui [14] are open source schedulers that support running job arrays on clusters or network of work stations; yet the admission control with immediate notification has not been implemented. The commercial Moab scheduler [15] provides a limited support for deadline scheduling 3. DESIGN AND IMPLEMENTATION OF THE TWO-LEVEL SCHEDULER Cluster computing systems typically consist of one frontend node and a number of compute nodes. Users submit jobs into the frontend node where a job scheduler interacts with the cluster resource manager to distribute jobs to compute nodes for execution. We use Torque [16] for the resource manager to provide the primitive job manipulation commands such as start, hold, cancel, and monitor jobs. Torque is an open source variant of PBS batch system and it has been widely used in many HPC centers over years. The decision on which job is started on which compute node is made by an external scheduler (Torque is bundled with a built-in but simple scheduler pbs_sched). Torque consists of two main components, pbs_server and pbs_mom, as illustrated in Figure 1. The pbs_server is a central server running on the frontend node to receive new jobs into job queues. It communicates with pbs_mom to collect the status of all compute nodes and jobs. The pbs_mom runs on each compute node to monitor status of the compute node and reports it to the pbs_server. When the pbs_server needs job scheduling, it sends a trigger message to an external scheduler. Trigger messages are sent according to several

3 events such as new jobs arrive, executing jobs terminate, or scheduling time is reached. After receiving a trigger message, the external scheduler module interacts with the pbs_server to perform job scheduling. We implement our two-level scheduler on PluS scheduler [17], a Java-based scheduler that can communicate with Torque. The two-level scheduler acts as a matchmaker between jobs and computing resources. When computing resources become available, the scheduler selects the jobs to be executed in the resources to meet our objective: the QoS jobs are always guaranteed to be completed within the deadlines while the best-effort jobs equally share the computing resources. To decide if the QoS jobs are accepted or rejected, we rely on the admission control module. For a QoS Job, the modified qsub command takes additional two arguments, -l deadline=[[[[cc]yy]mm]dd]hhmm[.ss] and -l walltime=hh:mm:ss for the deadline and the processing time of the submitted job, respectively. The qsub command first contacts the admission control for the immediate notification of acceptance. The job is rejected if resources are not enough for the job and all previously accepted QoS jobs to finish by their deadlines; otherwise the job is accepted and waits in the job queue for scheduling later. For a best-effort job, the qsub command works as usual, i.e. the job is put into the job queue. We assume in this work that the system is a homogeneous cluster, or in other words, all compute nodes are identical. Figure 1. Two-level scheduler with Torque resource manager. 3.1 ONLINE ADMISSION CONTROL The online admission control needs to ensure that the new QoS job can be completed by its deadline without violating the deadline of previously accepted jobs. The feasibility testing algorithm to accept a new job is given in Listing 1. The available time f(m) of the machine m is the earliest time that the machine is free to be used. When a job is dispatched to execute on a machine, its available time is updated accordingly. The p(j) and d(j) are the processing time and the deadline of job j given by the user, respectively. To test an incoming job arrays, every job in it

4 and all previously job arrays has to be checked if it can be completed by the deadline. We use the Earliest Deadline First (EDF) algorithm to select the job to be tested. If all QoS jobs can pass the test, the incoming job array is then admitted into the job queue of Torque. Algorithm IsFeasible Input : The incoming job arrays Output : True or False Listing 1. Feasible testing algorithm. let E be the list of previously accepted and the incoming QoS job arrays sorted by their deadlines for each job array A E do for each job j job array A do let m be the machine with the minimum available time f(m) if f(m) + p(j) > d(j) then return false else f(m) = f(m) + p(j) end if return true 3.2 FIRST LEVEL: DYNAMIC QoS SCHEDULING The jobs in the job queue are mixed with QoS and best-effort jobs. As QoS jobs takes higher priority than best-effort jobs, all cluster resources are always allocated to the QoS jobs first if possible. Thus, when there is an available compute node, we apply the EDF algorithm to select the next job to be dispatched so that the job with the earliest deadline always starts first. Though using the EDF algorithm can meet our requirement, it is a research challenge to see if other algorithms can be more efficient. If there is no QoS job existed in the job queue, fairshare scheduling is invoked for the best-effort jobs as illustrated in Figure 2. Figure 2. QoS and fairshare schedulers 3.3 SECOND LEVEL: DYNAMIC FAIRSHARE SCHEDULING Fairshare scheduling allows users to use cluster resources equally. If user A has done twice as much work as user B over a period of time, then user B should thereafter be able to do twice as much work as user A. The fairshare scheduler records the CPU usage of each user for his besteffort jobs executed so far and selects the next job belonging to the user who has the smallest CPU usage. The time in fairshare scheduling is discretized into a number of time frames. At the end of each time frame, the CPU usage of each individual job in a job array is calculated from the resources_used.cput field of Torque s job status which represents the CPU time used by this job since started. To limit the impact of usage data which are too old, the CPU usages are

5 exponentially decayed by time so that higher weight is put on more recent data when calculating the accumulative CPU usages over multiple time frames. The overall algorithm of the fairshare scheduling is shown in Listing 2. The CPU usage history is denoted by cpuusage, the set of running jobs at the previous time frame by R t-1, and the set of running jobs at the current time frame by R t. The used CPU times of each job are stored in cput t and cput t-1 for the current and previous time frame, respectively. The parameter FSDECAY is the exponential decay factor, and FSINTERVAL is the duration of each time frame. The values of these parameters can be set to suit each specific need. For more sophisticated fairshare scheduling such as group or project based fairshare, readers may refer to [18] Listing 2. Fairshare scheduling algorithm. Algorithm FairshareScheduling Input : A set of best-effort job arrays J Output : The selected job k for each user u do cpuusage{u} = FSDECAY * cpuusage{u} for each running job j R t-1 and j is a best-effort job do if j R t continue let u be the owner of job j cpuusage{u} = cpuusage{u} + ( cput t(j) - cput t-1(j) ) let A be the best-effort job array ( J) of the user with the smallest cpuusage let k be an arbitrary job in job array A return k 4. CONCLUSIONS We have proposed the two-level scheduling with admission control to support resource allocation in cluster computing systems for mixed QoS and best-effort job arrays. The implementation is based on the combination between two well-known job scheduling algorithms. Job admission control and the dynamic scheduling of QoS job arrays are accomplished by the EDF algorithm to ensure the completion time by their deadlines. The fairshare algorithm is used in scheduling of best-effort job arrays so that the computing resources are equally occupied by all users. The proposed scheduler can enrich the HPC systems to support more diverse user requirements in this complex world. To the users, the system is more responsive and versatile. To the resource owners, the new scheduler allows the resources to be used in more efficient manner. Furthermore, the scheduler can interface with many resource managers such as Torque, SGE or condor. In the future work, we plan to extend our scheduler for cost-based admission control and add a global resource sharing policy between the entire QoS and best-effort user groups. REFERENCES 1. M. H. Goldwasser and B. Kerbikov, Admission control with immediate notification, J. of Scheduling, vol. 6, no. 3, pp , J. Ding and G. Zhang, Online scheduling with hard deadlines on parallel machines, in Algorithmic Aspects in Information and Management, Second International Conference, Proceedings, Lecture Notes in Computer Science, S.-W. Cheng and C. K. Poon, Eds., vol Springer, pp , 2006.

6 3. J. Ding, T. Ebenlendr, J. Sgall, and G. Zhang, Online scheduling of equal-length jobs on parallel machines, in Algorithms - ESA 2007, 15th Annual European Symposium, Proceedings, Lecture Notes in Computer Science, vol Springer, pp , M. H. Goldwasser and M. Pedigo, Online nonpreemptive scheduling of equal-length jobs on two identical machines, ACM Trans. Algorithms, vol. 5, no. 1, pp. 1-18, J.-H. Kim and K.-Y. Chwa, On-line deadline scheduling on multiple resources, in COCOON '01: Proceedings of the 7th Annual International Conference on Computing and Combinatorics. Springer-Verlag, pp , W. Cirne, D. Paranhos, L. Costa, E. Santos-Neto, F. Brasileiro, J. Sauve, F. A. B. Silva, C. O. Barros, C. Silveira, and C. Silveira, Running bag-of-tasks applications on computational grids: the mygrid approach, in Parallel Processing, Proceedings International Conference on, pp , Y. C. Lee and A. Y. Zomaya, Practical scheduling of bag-of-tasks applications on grids with dynamic resilience, IEEE Trans. Comput., vol. 56, no. 6, pp , A. Benoit, L. Marchal, J.-F. Pineau, Y. Robert, and F. Vivien, Offline and online masterworker scheduling of concurrent bags-of-tasks on heterogeneous platforms, in 22nd IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2008, pp. 1-8, E. Anderson, D. Beyer, K. Chaudhuri, T. Kelly, N. Salazar, C. Santos, R. Swaminathan, R. Tarjan, J. Wiener, and Y. Zhou, Value-maximizing deadline scheduling and its application to animation rendering, in SPAA '05: Proceedings of the seventeenth annual ACM symposium on Parallelism in algorithms and architectures. ACM, pp , Y. Zhang and A. Sivasubramaniam, Scheduling best-effort and real-time pipelined applications on time-shared clusters, in SPAA '01: Proceedings of the thirteenth annual ACM symposium on Parallel algorithms and architectures. ACM, pp , J. Sherwani, N. Ali, N. Lotia, Z. Hayat, and R. Buyya, Libra: a computational economybased job scheduling system for clusters, Softw. Pract. Exper., vol. 34, no. 6, pp , Sun grid engine, Condor project, Maui scheduler, Moab workload manager, Torque resource manager, H. Nakada, A. Takefusa, K. Ookubo, M. Kishimoto, T. Kudoh, Y. Tanaka, and S. Sekiguchi, Design and implementation of a local scheduling system with advance reservation for coallocation on the grid, in CIT '06: Proceedings of the Sixth IEEE International Conference on Computer and Information Technology. p. 65, D. B. Jackson, Q. Snell, and M. J. Clement, Core algorithms of the maui scheduler, in JSSPP '01: Revised Papers from the 7th International Workshop on Job Scheduling Strategies for Parallel Processing. Springer-Verlag, pp , 2001.