Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst

Transcription

1 Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst

2 Outline General Description of Virtualization / Virtualization Solutions Shared HPC Infrastructure Virtualization in High Performance Computing Dynamic Partitioning of a shared Computing Cluster Concept KIT Implementation (KVM and Maui/Torque) DESY Implementation (Xen, SGE and vmimagemanger) 2

3 Virtualization Possible Definition (Computer Virtualization) Sharing resources of one physical machine between independent Operating Systems (OS) in Virtual Machines (VM) Requirements: Support of multiple OS in the VMs Different Linux Distributions Microsoft Products, a.s.o. VMs have to be isolated Acceptable performance overhead 3

4 Virtualization - Products KVM and many more 4

5 Shared HPC Infrastructure Two possibilities to run a high performance computing cluster: Departments or user groups are running their own computing infrastructure Pros: 1. Setup will cope all their needs in hardware and software Cons: Overhead in administration, administration has to be done by each group No load balancing between the isolated ( private ) computing clusters 5 Department or user groups run a shared computing infrastructure: Pros: Administration can be centralised (e.g. at the Computing Centre of a University) Shared funding may lead to a favourable hardware price Load-balancing Cons: Set-up has to be a compromise! 2.

6 Shared HPC Infrastructure Compromise is not desirable/possible in some cases: Incompatibilities between software and operating systems (OS) within the needs of the different groups. Some groups want to participate in a Grid environment (may lead to point above). The Grid environment should be isolated from the local users (security...) Department or user groups run a shared computing infrastructure: Pros: Administration can be centralised (e.g. at the Computing Centre of a University) Shared funding may lead to a favourable hardware price Load-balancing Cons: Set-up has to be a compromise! 2. Virtualization 6

7 Dynamic Partitioned Cluster A cluster where multiple OS are needed can be partitioned dynamically with virtualization Dynamic Partitioned Cluster: Virtual machines are deployed just as needed Scheduling with fair-share Virtualization techniques hidden from the user, sees only standard batch system and partition queues 7

8 Dynamic Partitioned Cluster User Interface Realization of the virtualization system: Wrapper script (Prologue/Epilogue) around the actual computing job inside the batch system Virtual machines are deployed e.g. on the hardware node discs Wrapper script starts VM and checks status of the VM Starts computing job inside VM Destroys or halts VM after the job has finished 8 WMS CE & Batch server Worker node & Batch client

9 Dynamic Partitioned Cluster User Interface Realization of the virtualization system: Wrapper script (Prologue/Epilogue) around the actual computing job inside the batch system Virtual machines are deployed e.g. on the hardware node discs Wrapper script starts VM and checks status of the VM Starts computing job inside VM Destroys or halts VM after the job has finished WMS CE & Batch server - Start VM - Run job in VM Virtual (wait ) Worker Node - Stop VM Physical Host & Batch client 9

10 KIT Solution Hardware: Test nodes 2x Intel Quadcore Xeon X5355 Diskspace per node: 750GB ClusterFS Lustre (~350TB) Software Host OS SuSE Enterprise 10 sp2 (SLES10sp2) 64 Bit Kernel smp Virtualization: KVM-84 VM OS SLC Bit / 32 Bit Batch System MAUI/Torque MAUI 3.2.6p19 Torque

11 KIT Set-Up of the Clusters Script languages: Perl, Bash One VM Partition with SLC

12 DESY Solution Main goal: Bringing virtualization to Grid/gLite users Using the Sun Grid Engine as back-end SGE support by the glite middleware Version 6.1u3 used lcg-ce as grid front-end used (3.1.18) Host: SL 5.1, el5xen kernel No middleware installed, runs SGE execd and shepherd VM: SL 4.7, ELxenU kernel Has middleware installed, but no batch client 12

13 Implementation Details Short prologue, epilogue and started scripts Basically run vmimagemanager And execute job payload via ssh on VM Proof of principle implementation:1 CE, 1 SGE master, 1 host, 2 VM Stress test with > 100 jobs submitted simultaneously via WMS successful! No changes to CE, minor SGE configuration changes. 13

14 VM Deployment Simple layered solution Sun Grid Engine Image-manager Import/export of VM images Stores OS Restores OS SGE prefix Job SGE postfix Batch Queue integration Prefix: Initiates image manager. Run: initiates job on VM. Postfix: Halts VM. State management is simple Very little to deploy/state vmimagemanager.py Virtualization Layer Xen (KVM planned) Virtual Job 14

15 vmimagemanager.py Deployment Scripted Configuration LVM image management Low dependencies Mostly automated (YAIM) Functions Fresh OS per job Easy to change to just reboot the OS Easy to extend to user defined jobs. Hooks available History Deployed for debugging vmimagemanger.py Years of DESY Test Glite Batch Queue Plans (in progress) KVM, Qeum, Xen Through libvirt Kpartx: full disk handling Mounting virtual hosts SA3 certification WLCG Mass Deployment NIKEF interested in testing 15

16 Problem: Different virtualization interfaces (Xen, KVM) libvirt Toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes) Long term stable C API Supports: Xen on Linux and Solaris hosts, QEMU, KVM, LXC Linux container system, OpenVZ, User Mode Linux Storage on IDE/SCSI/USB disks, FibreChannel, LVM, iscsi, NFS and filesystems Provides: Remote management using TLS encryption and x509 certificates Remote management authenticating with Kerberos and SASL Local access control using PolicyKit Zero-conf discovery using Avahi multicast-dns Management of virtual machines, virtual networks and storage Portable client API for Linux, Solaris and Windows Hompage: 16

17 Conclusion There are use cases where a bare-metal use of worker nodes in a shared cluster is not possible Virtualization Allows dynamic partitioning of a Cluster customised software environments for all user groups load-balancing Performance overhead is acceptable Our approaches (DESY/KIT): Do NOT need a modification of the used batch system to be VM aware (VM is seen as job) Light-weight and transparent Intelligent scripts and the standard-batch-system configuration do the job 17

18 Outlook DESY/KIT: Test Scalability Planned test at KIT: up to 1600 VMs on core worker nodes KIT: Production use at UNI-KARLSRUHE WLCG Tier3 site DESY: Different VM images test possibility of user defined images DESY: CREAM CE Offers advanced virtualization possibilities due to new GLUE scheme and availability of JDL file DESY/KIT: Change scripts to use libvirt enable parallel KVM and XEN usage 18

19 People & Contributors KIT (IEKP/SCC): Volker Büge, Marcel Kunze, Oliver Oberst, Günter Quast, Armin Scheurer DESY: Yves Kemp, Owen Synge HTW Berlin: Hermann Hessling Sun / SGE: Andreas Haas 19

20 Backup Slides 20

21 Security and password-less login Prologue and epilogue run as user, but need root privileges to start XEN VM. E.g. scripts in sudoers Starter method needs password-less login from Adequate shosts.equiv e.a. configuration for ssh ssh is required because of glite middleware No other ssh connection from necessary Connection to WMS secured via GSI User has no possibility to run code on the host! 21