Experimenting OAR in a virtual cluster environment for batch schedulers comparative evaluation

Transcription

1 Experimenting OAR in a virtual cluster environment for batch schedulers comparative evaluation Joseph Emeras (Joseph.Emeras@imag.fr) Yiannis Georgiou (Yiannis.Georgiou@imag.fr) 1 Context OAR [3] is the Grid 5000 Resources and Jobs Management System (RJMS). RJMS are composed of two parts: a batch scheduler and a resources manager. Their role is to plan and manage jobs executions on the resources and to manage the resources themselves. OAR is a tool for both production or research clusters. Its mechanism principle is the following: a user requests some resources associated to a task. The system schedules this request depending on availabilities and returns either a date when the task will start or a shell on one of the attributed resources. The OAR software is developed by MESCAL and MOAIS teams in the LIG [1] laboratory. Developments are also supported by the Aladdin/Grid 5000 community and the CIMENT [2] project, that are currently its two majors users. In addition, other installations were also listed in Brazil, Slovaquia, China and in France, where the BRGM in Orleans uses it. Currently, OAR handles succesfully up to several hundreds of resources on each site but we know that its scheduler may not be as efficient with thousands or hundreds of thousands resources. Since the number of cores and cpus per machine is increasing considerably, we must look into the future and be prepared to face the question of large scale resource management that could involve deep modifications in the data structures representing manipulated resources and scheduling algorithms. This is why we want to study: How OAR behaves comparing to the other RJMS depending on the number of resources through a comparative valuation How OAR will handle large scale resources An other unconfessed goal of such experiments in a virtual context is to ease OAR testing and tuning by helping the development team to test the new features they just coded and to set up non-regression tests in a real cluster context and researchers to test their new ideas and to play with OAR s tuning parameters. But since this is not the main goal, we will not range over too much this subject in this article. 1

2 2 Experiment goals We ve seen in the previous section that we want to study two aspects. For the comparative evaluation between OAR and other RJMS, several series of benchmarks have been made on several RJMS thanks to Grid 5000 and the benchmark suite ESP [6]. Amongst them, were OAR, Torque/MAUI and SLURM. The work was around the comparison between these systems depending on the number of managed resources. The problem here, is that such tests need a lot of resources on the Grid 5000 platform and it is not always easy to obtain the resources if a lot of jobs are running, and certainly not nice to the other users to monopolize the platform. The second aspect joins here the first one, if we want to study how OAR will handle large scale resources, we will need such large scale resources, and for the moment, this is not the case in Grid In fact, what we want to do, is to increase the number of resources in Grid 5000 to serve our goals. One response to this problem is Virtualization. If we manage to create several virtual machines per node on our cluster, we have solved our two problems. The next figure presents the concept of the virtualization in the experiment. In fact, if we manage to create a tool that will build automatically an OAR powered cluster inside Grid 5000, and if this tool is convenient enough to tune parameters such as the number of real nodes, the number of virtual machines per node, the virtual machines memory and the number of virtual cpus (vcpus) on the virtual machines, we will be able to play with the benchmarking tool and make very complete behaviour graphs of the different RJMS but also with few resources, we will be able to create a lot of virtual resources, that will be very usefull to test OAR in a large scale resources context. An other benefit of Virtualization will be to speed up the number of experiments results by allowing to run more experiments in parallel or to save energy by reducing the number of physical machines used for a same amount of experiments. It still remains a problem that we have to solve: will the use of virtual machines be transparent for benchmarking, in other words, if the use of virtualization slows down too much the system, the benchmark can be distorded. We have to take into account this problem in our choices. 2

3 Figure 1: Virtualization experiment concept. 3

4 3 Tools used We saw that we will need a lot of virtual machines, and that the global overhead due to virtualization has to be as small as possible. An other aspect here, is that by using many virtual machines on one host, this will induce a large amount of disk space usage. If one virtual machine takes about 400MiB, having dozens not to say hundreds of them could be problematic. To solve these two problems we chose to use Xen + LVM. Two studies of such problematics ([4] and [5]) shown that Xen is a good, reliable, and efficient virtualization tool. Furthermore, in several Grid 5000 sites, Xen powered images are available or will be soon. LVM and snapshot system will allow to minimize the disk space usage. By using this method, only the base image of the guests OS will take about 400MiB but the snapshots themselves will take only few KiB. At the end, having one virtual machine or hundreds of virtual machines on the same host will use approximately the same disk space. The benchmark tool used will be ESP (Effective System Performance). It is designed for RJMS testing. It launches jobs of different durations that will take from one resource to the entire cluster. These jobs can be (depending on how was parametrized the ESP run) simple sleeps or sort of cpuburn jobs. Here, we do not focus on the jobs themselves, their result is useless in a benchmark case, the only important thing is their run time, which is a parameter of the job itself and do not depend on the cpu speed nor on the current cpu charge. This is why in our case, it is not a problem to use virtualization and to run X jobs on one real core (X beeing the number of virtual machines on the host), because these jobs will still last the same duration as if there where only one job on the core. Since the goal of the experiment is to launch a lot of benchmark tests on different situations and then, study the results depending on the number of real machines, virtual machines, memory and vcpus allocated per vm, the process have to be automated. Thus, the Kapharnaum tool has been created. Currently it is more a prototype then a real production tool but it is reliable enough for experimenting the automated process. It takes as parameter these variables and then automatically creates the OAR powered virtual cluster on top of Xen and LVM snapshots. Everything is automatic, from the OAR server set-up and virtual computing nodes register to the benchmark launching and the results collect. It is to be said that when creating the nodes and server, the script installs OAR from the latest generated packages, this allowing, coupled to a smart automated package generation tool, to test the latest code commits. This tool, very easy to handle and mostly highly customizable will allow us to study deeply OAR s behaviour. But first, let s see if Xen handles well virtualization without a big overhead. 4

5 4 Results Now that we have a tool to test OAR in a virtualized context, it is important to compare its effectiveness in a regular vs. virtualized environment. By regular, we mean in a standard Grid 5000 cluster context. For now, we don t have a lot of results since the virtual cluster creation tool is very new, but some of them are very interesting. They are shown in the table below. Comparison of OAR ranks in virtualized vs. non-virtualized modes depending on the number of resources and the experiment parameters. Griffon cluster, non-virtualized mode Total resources number Number of real nodes 8 64 Number of cores per node 8 8 ESP rank (percent) Grelon cluster, virtualized mode Total resources number Number of real nodes Number of cores per node Virtual machines per node cores/vcpus used ESP rank (percent) Variance Griffon cluster, virtualized mode Total resources number Number of real nodes Number of cores per node Virtual machines per node cores/vcpus used ESP rank (percent) Variance Experiments realised in the Nancy Grid 5000 site on the griffon cluster (nodes: 92, cpuarch: x86-64, cputype: xeon-hapertown, nodecpu: 2, cpucore: 4, memnode: 16384, disktype: sata, ethnb: 1) and grelon cluster (nodes: 120, cpuarch: x86-64, cputype: xeon-woodcrest, nodecpu: 2, cpucore: 2, memnode: 2048, disktype: sata, ethnb: 2). The Total resources number is the Number of real nodes Virtual machines per node cores/vcpus used. It is important to say that the Kapharnaum tool is really new and is still in testing state, so we don t have enough results to make a valuable comparison. However, regarding the preliminary results we can see a tendency here. A benchmark with ESP will give comparable results if the number of total vcpus per host (number of vm on the host number of vcpus per vm) is less or equal 5

6 to eight time the number of real cores on the physical node. If this total number is bigger, we begin to observe some performance lost. Thus we can multiply by 8 the number of resources with XEN without impacting our benchmark result. These preliminary results encourage us to continue the work done here and to add more smartness and error control in Kapharnaum in order to observe more finely the XEN behaviour in this case. In conclusion, we can say that the Kapharnaum tool is almost ready for automated OAR benchmarks runs but it is still a work in progress if we want more precise knowledge of the interactions between XEN, ESP and OAR and if we want to improve the results. 6

7 References [1] Grenoble Computing Laboratory - [2] The ciment project - [3] N. Capit. A batch scheduler with high level components. CCGrid 05. [4] L. Nussbaum. Contributions à l expérimentation sur les systèmes distribués de grande taille - [5] B. Quetier. Etude de mécanismes de virtualisation pour l émulation conforme de Grilles à grande échelle. [6] A. T. Wong. ESP: A System Utilization Benchmark. 7