Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil

Transcription

1 Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil Volker Büge 1, Marcel Kunze 2, OIiver Oberst 1,2, Günter Quast 1, Armin Scheurer 1 1) Institut für Experimentelle Kernphysik, KIT 2) Steinbuch Centre for Computing, KIT

2 What is Particle Physics? - Dimensions Macrocosm Crystal Molecule 10 1 m 10-2 m 10-9 m Atom Quark / Electron Proton Nucleus m <10-18 m m m 2

3 Our Instruments Accelerators - LHC Lake Geneva Large Hadron Collider Circumference: 27 km Beam energy: 7 TeV Below surface: 100 m Temperature: -271 C Energy use: 1 TWh/a 4 large experiments: CERN Airport CMS LHCb ATLAS ALICE 3

4 Our Instruments Event Display The collision of 2 high-energetic hydrogen nuclei (protons) produces several thousands of particles 4

5 Our Instruments Detectors - CMS Compact Muon Solenoid - CMS Specifications: total weight: T overall diameter: overall length: magnetic field: 15 m 21,5 m 4 Tesla read out channels: > read out rate: 40 MHz 5

6 Our Instruments Detectors - Data Rates Reduction with ASICs ~ 60 TB/sec Level 1 Trigger Collision Rate: ~ 40 MHz Event size: ~1.5 MB for Offline- Analysis Tape & HDD Storage ~ 225 MB/sec ~ 150 GB/sec High Level Trigger Software Data Reduction (PC Farm) CMS will produce more than 1,5 Petabyte per year 6

7 Peculiarities of HEP Data Processing e.g. meteorology, geography, finance Static program and dynamic data: Set of fixed programs used to analyse new data in short intervals Same approach for e.g. in High Energy Physics Static data and dynamic code : Data acquisition very expensive Events are independent! Data is analysed repeatedly with iteratively improved code e.g. ~1000 publications from 500 people with ~1TB preprocessed data! 7

8 CMS High Energy Physics Collaboration CMS 38 Nations 182 Institutions > 2000 Scientists & Engineers 8

9 Data Storage and Access Constraints and Approaches: HEP experiments very expensive! redundant storage of 1.5 PetaByte of detector data per year only for CMS! not possible at one single computing centre (funding constraints) distribution of data to participating computing centres all over the world huge datasets (~TeraByte) cannot be transferred to each user the analysis job goes where the data set is ensure access to these data for more thousands of physicists from hundreds of institutes all over the world access to data and computing resources without local login for the user The LHC experiments cope with these challenges using grid technologies The Worldwide LHC Computing Grid 9

10 The Worldwide LHC Computing Grid (WLCG) The WLCG Computing Model: Computing centres are organised in a hierarchical structure Different policies concerning computing power and data storage The tiered architecture: 1 Tier 0 (at CERN): Accepts raw data from detectors Data transfer to Tier1 centres few Tier 1 centres: Secure data archiving Reconstruction & reprocessing Coordination of associated Tier2s Several Tier 2 centres: Capacity for data-intensive analysis Calibration & Monte Carlo simulation 10

11 GridView The WLCG Real Time Monitor Connecting computing resources to the WLCG is realised through grid middleware services which have to be deployed at the site The EGEE middleware glite 11

12 Software Constraints Grid Middleware glite: Required services at a site: Computing Element: portal to the local batch system of a site Storage Element: portal to the local storage resources Monitoring Box: collects and publishes information on grid jobs executed at a site Worker Node package: to be deployed on each WN to provide grid functionality CMS Software Environment CMSSW: Framework is continuously improved by collaborators and offers: Interfaces to standard Monte Carlo generators Modules for the simulation of the detector response for MC data Modules for the reconstruction of physics objects Development of code for user analyses Both, glite and CMSSW are only validated on Scientific Linux 4! LHC experiments require this OS on all grid worker nodes 12

13 Virtualisation A possible definition of virtualisation: Possibility to share resources of one physical machine between different independent operating systems (OS) in Virtual Machines (VM) Requirements: Support of multiple commodity OS in virtual machines Different Linux distributions Microsoft Windows Virtual Machines have to be isolated Acceptable performance overhead 13

14 Virtualisation Products There are plenty of existing products for virtualisation: KVM and many more 14

15 Applications - Hardware Consolidation Typical situation at smaller grid sites: For reasons of stability and maintainability, the different grid services should run on different machines Varying load on the different machines Resources not fully exploited Recycling of older machines leads to a heterogeneous hardware structure High administrative effort for installation and maintenance of the system CE SE MON CE SE MON Host (XEN) Virtualisation of these machines leads to few machines to be maintained and to homogenous OS installations 15

16 Shared Computing Infrastructures 1 Isolated cluster Each institute maintains its own cluster Clusters optimised for the individual particularities of each group: Hardware Layout Operating system Security policies Software Each institute has to provide experts for administration of the cluster services and hardware Multiple installations of small cooling an power supply infrastructures Isolated clusters are flexible but have drawbacks! Clusters have to be designed to cover peak loads Non optimal usage of the investment 16

17 Shared Computing Infrastructures 2 Shared cluster Departments or user groups share one computing infrastructure Administration can be organised centrally by few experts ( e.g. from the computing centre) Shared funding may lead to a better price of the tender Set-up has to be a compromise Hardware setup must be homogeneous OS of choice has to be supported by administration Load balancing only possible in case of a homogenous cluster setup Shared cluster offer advantages BUT set-up will be a compromise! 17

18 Shared Computing Infrastructures 3 Dynamic Partitioned Cluster (using virtual machines) Combining all advantages of a shared cluster without the need of a homogenous OS set-up Decoupling the needs of all user groups and the administration Load-balancing through fair-share functionality of the batch system Virtualisation layer hidden from the user, who only sees a standard batch system Virtualisation helps overcoming the limitations of shared clusters 18

19 Institute SCC Institute Cluster at the SCC: Multiple institutes from different departments of the University of Karlsruhe Each institute has particular requirements concerning hardware and OS Deployed and maintained by SCC Hardware: 200 worker nodes (1600 cores): 2xIntel Quadcore 2.66GHz Xeons 2GB RAM per core 16 GB per node 750 GB per node 370 TB Lustre file system Access from WN via Infiniband Software: Host OS SuSE Enterprise 10 sp2 (SLES10sp2) 64 Bit Kernel: smp Virtualisation: KVM-84 Virtual Machines Scientific Linux Cern Bit / 32 Bit Batch System Torque MAUI 3.2.6p19 19

20 Wrap-Job Batch System Virtualisation ViBatch Approach does not require modifications of standard batch systems like Maui/Torque VM image files are deployed on each physics Worker Node Wrapper script around the actual computing job inside the batch system Starts VM on the hardware node assigned to the job Checks status of the VM As soon as VM is ready, the job is forwarded to the VM using ssh and executed inside the VM Having finished the job, the VM is deleted (for security reasons) 20

21 Implementation of ViBatch 21

22 Conclusion & Outlook Virtualisation techniques well suited for server consolidation Extendable to the field of batch systems to offer a dynamic partitioning of a cluster: Customised operating system for each user group Load balancing becomes possible in heterogeneous OS environments Small performance overhead for HEP application acceptable Security / privacy aspects Presented approach: Offers batch system virtualisation without modifications of standard batch systems Not restricted to a dedicated virtualisation technique Selected publications: V. Büge, Y. Kemp, M. Kunze and G. Quast: Application of Virtualization Techniques at a University Grid Center e-science'06 - Second IEEE International Conference on e-science and Grid Computing, ISBN , 2006 V. Büge, Y. Kemp, M. Kunze, O. Oberst and G. Quast: Virtualizing a Batch Queuing System at a University Grid Center Lecture Notes in Computer Science, ISBN~ , Springer 2006 Volker Büge: Virtualisation of Grid Resources and Prospects of the Measurement of Z Boson Production in Association with Jets at the Large Hadron Collider Vdm Verlag, ISBN , November 2008 V. Büge, O. Oberst et. al: Integration of Virtualized Worker Nodes in Standard-Batch-Systems Proceedings of CHEP 2009, Prague, March 2009 (to be published) 22

23 BackUp 23

24 What is Particle Physics? - Big Bang Today Heavy Atoms... Timescale Light Atoms Nuclei Nucleons Energy... Elementary Particles Big Bang 24

25 Our Instruments Detectors - CMS 25