Provisioning and Resource Management at Large Scale (Kadeploy and OAR)

Transcription

1 Provisioning and Resource Management at Large Scale (Kadeploy and OAR) Olivier Richard Laboratoire d Informatique de Grenoble (LIG) Projet INRIA Mescal 31 octobre 2007 Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

2 Outline 1 Resource Management Systems (RMS) 2 OAR 3 Features 4 Provisioning 5 Issues at large scale 6 Conclusion livier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

3 Resource Management Systems (RMS) Job and Resource Management System Also called Batch Scheduler, they can be divided in 2 main parts Job Management. Resource (node) Management. Example of systems : Slurm, Torque/MAUI, Sun Grid Engine, condor, LSF, PBSPro, Moab Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

4 OAR OAR : a Job and Resource Management System History : Developed after the use of OpenPBS/PBSPrp on Icluster1 (225 PC cluster (ranked 385 Top500/2001 june)) [ ] Need of a scalable, robust and flexible batch scheduler with short development cycle. Need of a RMS to test new ideas, abstractions, schedulers. OAR version 1.6 was managed production clusters and clusters of Grid 5000 (more than 3 million of jobs had been executed). [ ] New branch : OAR 2.X (2.2.8 on Grid 5000 platform) [2007-] Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

5 OAR OAR : General Architecture Central Server which orchestrates all operations between modules. Client programs to interact with users or automatic job agents. Lot of parameters to setup. Users Submission Scheduler Matching of resource Launching and control of execution Monitoring Client Log, Accounting Server Computing nodes Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

6 OAR OAR : Design Philosophy Based on high level software components : relational database engine (MySql) to store all states (job, resources, internal state. Scripting Language (mainly Perl, ) to implement the execution engine, modules. ssh as remote command on resource. Taktuk to execute parallel operation on resources. Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

7 Features Key Features Interactive and Batch modes Advance Reservation Suspend/Resume Job Checkpoint support Job Dependencies Resources Matching First-Fit scheduling (backfilling) FairSharing Resources Hierarchy Multiple type of resources (as Licence, Switch, Opitical Network, virtual resources and so on) Multiple types of jobs (deploy, best-effort, co-system,...) Moldable Jobs support Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

8 Provisioning Provisioning (Deploying) Provisioning or deploying complete computing environment on compute nodes is an important issue : to administrators to simplify maintenance operations (installation, upgrade). to users to dispose of their own environment and typically root privileges. Computing Environment is all necessary softwares : OS Libraries Runtime, Middleware Applications Data Several tools exists : Freesbee (Emulab), Dolly+, Kadeploy, Xcat, SystemImager... Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

9 Provisioning Kadeploy : a low level deploying tool Features : It allows to quick deploy and manage multiple computing environments on cluster and multi-cluster platform. It permits also the deployment of multiple Operating Systems on every node without compromising the installed partitions. Kadeploy makes it possible for users to concurrently deploy computing environments on different sets of nodes on the same clusters. Application : On Grid 5000 platform, Kadeploy is used by users to easily setup their own environment to do more deep and controllable experiments. It s also used to homogenize environment between different clusters. Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and OAR) / 22

10 Kadeploy : Principles Provisioning A relational database is used to store all information about environments (except image of environment) Scalable thanks to pipelined chain for environment diffusion Deployment is composed of 14 steps (reboots, node checks, diffusion, pre- and post- installations...). During process when a fault occurs concerned nodes are excluded. 10 minutes for 400 nodes Physical Network n D D Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 10 / 22

11 Provisioning Deployment and Resource Management Links between Deployment and Resource Management : Rights Management : Resource Management System gives/removes the right to a job (or users or his script) to execute deploy operations. RMS must restore previous system (reboot and specific partition or deploy a reference environment). Scheduling : the overhead time of deployment can be factorized if several jobs have the same environment requirements and leave RMS schedule deployments. livier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 11 / 22

12 Issues at large scale Deployment and Resource Management At Large Scale At large scale x10000 of resources Platform diversity different kind of clusters partitioned, multi-cluster (Grid 5000) multi-site, grid Software Diversity surely will increase. Users want simple access to resource and their environment. Administrators want simple infrastructure (HW & SW) to manage and customize. Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 12 / 22

13 Issues at large scale Deployment and Resource Management At Large Scale Main issues are Scalability (again) Performances Reactivity (scheduling and deployment time) Fault-tolerance... Complexity Configurability (become hard). Usability (users/administrators). Tractability (monitoring, logging, accounting, reporting, power consumption...) livier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 13 / 22

14 Issues at large scale Resource Management Scalability We use Taktuk for parallel operations in OAR. Need to optimize scheduler to large number of resources jobs Complexity Classical approaches to add features Add new abstraction/concept can be very hard to introduce in legacy RMS (like Torque). Add wrapper(s) or script(s) (see report on SunGridEngine for TSUBAME). Introduce new abstraction can be very profitable Introduce a new way to address some issues which can be more understandable. Can help to simply resource management OAR is good candidate to test/introduce new abstraction livier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 14 / 22

15 Issues at large scale Example : Resource Hierarchy in OAR A key abstraction to address large scale platform Can describe all levels of architecture (or logical) hierarchy : site/cluster/node/cpu/core/thread (as instance on Grid 5000 : cluster/node/cpu/core) Allows powerful resource requests Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 15 / 22

16 Issues at large scale OAR : Resource Hierarchy The Price : Scheduler Complexity Increases!!! Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 16 / 22

17 Issues at large scale Future works (collaboration interests) Resource Management related Dynamicity Power management and power-saving (green computing) Study global management (all resources : cpu, network, storage, licences...) Develop Scheduler (scalable scheduler w matching ressource, fair-sharing, first-fit, resource hierarchy, job depencies, dynamic job, moldable job Is it feasible? (surely not in Perl language) Service Level Agreement (SLA) and Quality of Service (QoS) Simplify Administration Interactions External Agents (Applications, Middleware, others RMS, Grid Agents) Internal (multitype job, multitype resource, multiple policies their combinaison raise several issue (as incoherence issue, inefficiency,...) livier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 17 / 22

18 Issues at large scale Issues of Deploying at Large Scale At large scale Scalability Orchestrate deployment on several infrastructure which can don t have the same low level deployment tool. Typical user want : to develop on his multi-cores laptop, to test on local cluster / large server (100-1K cores), to run on large infrastructure ( 1k cores). All these previous steps must as transparent or easy as possible. livier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 18 / 22

19 Issues at large scale Virtualization (everwhere?) A new try for a new global solution Virtual Cluster/Grid Elastic Compute Cloud (Amazon EC2) Virtual Workspace (Globus) Some limitations and issues with virtualization in HPC Lot of virtual machine solutions (see wikipedia) : Xen, Vserver, Vmware, Qemu, KVM... Performance degradations (for instance I/O, system calls). Physical resource sharing issues : fair sharing and performance isolation (CPU/memory/network/storage) specialized hardware sharing (as high performance network) New element to consider to global resource management (optimisation, scheduling impact,...) Virtualization must be considered in large scale infrastructure but Low level deployment remains an important need. Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 19 / 22

20 Issues at large scale The next deploying system? Need a very flexible deploying system Platform diversity : low level deployment tools virtual machines Extend execution environment definition Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 20 / 22

21 Conclusion Conclusion OAR : ready for production uses and research studies Kadeploy : needs deep modifications (it lacks of flexibility) Scalability issues Balance between performance and number of available features. Scheduling need to optimize for large scale issues. Need tests at large scale (1K to 100K resources) (on Grid 5000) Complexity issue at large scale become hard to master need to find good/right abstractions livier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 21 / 22

22 Links Conclusion OAR http ://oar.imag.fr Kadeploy (last version up on request) http ://kadeploy.imag.fr Olivier Richard ( Laboratoire d Informatique deprovisioning Grenoble (LIG) and Resource Projet INRIA Management Mescal) at Large Scale (Kadeploy 31 octobre and2007 OAR) 22 / 22