Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil"

Transcription

1 Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil Volker Büge 1, Marcel Kunze 2, OIiver Oberst 1,2, Günter Quast 1, Armin Scheurer 1 1) Institut für Experimentelle Kernphysik, KIT 2) Steinbuch Centre for Computing, KIT

2 What is Particle Physics? - Dimensions Macrocosm Crystal Molecule 10 1 m 10-2 m 10-9 m Atom Quark / Electron Proton Nucleus m <10-18 m m m 2

3 Our Instruments Accelerators - LHC Lake Geneva Large Hadron Collider Circumference: 27 km Beam energy: 7 TeV Below surface: 100 m Temperature: -271 C Energy use: 1 TWh/a 4 large experiments: CERN Airport CMS LHCb ATLAS ALICE 3

4 Our Instruments Event Display The collision of 2 high-energetic hydrogen nuclei (protons) produces several thousands of particles 4

5 Our Instruments Detectors - CMS Compact Muon Solenoid - CMS Specifications: total weight: T overall diameter: overall length: magnetic field: 15 m 21,5 m 4 Tesla read out channels: > read out rate: 40 MHz 5

6 Our Instruments Detectors - Data Rates Reduction with ASICs ~ 60 TB/sec Level 1 Trigger Collision Rate: ~ 40 MHz Event size: ~1.5 MB for Offline- Analysis Tape & HDD Storage ~ 225 MB/sec ~ 150 GB/sec High Level Trigger Software Data Reduction (PC Farm) CMS will produce more than 1,5 Petabyte per year 6

7 Peculiarities of HEP Data Processing e.g. meteorology, geography, finance Static program and dynamic data: Set of fixed programs used to analyse new data in short intervals Same approach for e.g. in High Energy Physics Static data and dynamic code : Data acquisition very expensive Events are independent! Data is analysed repeatedly with iteratively improved code e.g. ~1000 publications from 500 people with ~1TB preprocessed data! 7

8 CMS High Energy Physics Collaboration CMS 38 Nations 182 Institutions > 2000 Scientists & Engineers 8

9 Data Storage and Access Constraints and Approaches: HEP experiments very expensive! redundant storage of 1.5 PetaByte of detector data per year only for CMS! not possible at one single computing centre (funding constraints) distribution of data to participating computing centres all over the world huge datasets (~TeraByte) cannot be transferred to each user the analysis job goes where the data set is ensure access to these data for more thousands of physicists from hundreds of institutes all over the world access to data and computing resources without local login for the user The LHC experiments cope with these challenges using grid technologies The Worldwide LHC Computing Grid 9

10 The Worldwide LHC Computing Grid (WLCG) The WLCG Computing Model: Computing centres are organised in a hierarchical structure Different policies concerning computing power and data storage The tiered architecture: 1 Tier 0 (at CERN): Accepts raw data from detectors Data transfer to Tier1 centres few Tier 1 centres: Secure data archiving Reconstruction & reprocessing Coordination of associated Tier2s Several Tier 2 centres: Capacity for data-intensive analysis Calibration & Monte Carlo simulation 10

11 GridView The WLCG Real Time Monitor Connecting computing resources to the WLCG is realised through grid middleware services which have to be deployed at the site The EGEE middleware glite 11

12 Software Constraints Grid Middleware glite: Required services at a site: Computing Element: portal to the local batch system of a site Storage Element: portal to the local storage resources Monitoring Box: collects and publishes information on grid jobs executed at a site Worker Node package: to be deployed on each WN to provide grid functionality CMS Software Environment CMSSW: Framework is continuously improved by collaborators and offers: Interfaces to standard Monte Carlo generators Modules for the simulation of the detector response for MC data Modules for the reconstruction of physics objects Development of code for user analyses Both, glite and CMSSW are only validated on Scientific Linux 4! LHC experiments require this OS on all grid worker nodes 12

13 Virtualisation A possible definition of virtualisation: Possibility to share resources of one physical machine between different independent operating systems (OS) in Virtual Machines (VM) Requirements: Support of multiple commodity OS in virtual machines Different Linux distributions Microsoft Windows Virtual Machines have to be isolated Acceptable performance overhead 13

14 Virtualisation Products There are plenty of existing products for virtualisation: KVM and many more 14

15 Applications - Hardware Consolidation Typical situation at smaller grid sites: For reasons of stability and maintainability, the different grid services should run on different machines Varying load on the different machines Resources not fully exploited Recycling of older machines leads to a heterogeneous hardware structure High administrative effort for installation and maintenance of the system CE SE MON CE SE MON Host (XEN) Virtualisation of these machines leads to few machines to be maintained and to homogenous OS installations 15

16 Shared Computing Infrastructures 1 Isolated cluster Each institute maintains its own cluster Clusters optimised for the individual particularities of each group: Hardware Layout Operating system Security policies Software Each institute has to provide experts for administration of the cluster services and hardware Multiple installations of small cooling an power supply infrastructures Isolated clusters are flexible but have drawbacks! Clusters have to be designed to cover peak loads Non optimal usage of the investment 16

17 Shared Computing Infrastructures 2 Shared cluster Departments or user groups share one computing infrastructure Administration can be organised centrally by few experts ( e.g. from the computing centre) Shared funding may lead to a better price of the tender Set-up has to be a compromise Hardware setup must be homogeneous OS of choice has to be supported by administration Load balancing only possible in case of a homogenous cluster setup Shared cluster offer advantages BUT set-up will be a compromise! 17

18 Shared Computing Infrastructures 3 Dynamic Partitioned Cluster (using virtual machines) Combining all advantages of a shared cluster without the need of a homogenous OS set-up Decoupling the needs of all user groups and the administration Load-balancing through fair-share functionality of the batch system Virtualisation layer hidden from the user, who only sees a standard batch system Virtualisation helps overcoming the limitations of shared clusters 18

19 Institute SCC Institute Cluster at the SCC: Multiple institutes from different departments of the University of Karlsruhe Each institute has particular requirements concerning hardware and OS Deployed and maintained by SCC Hardware: 200 worker nodes (1600 cores): 2xIntel Quadcore 2.66GHz Xeons 2GB RAM per core 16 GB per node 750 GB per node 370 TB Lustre file system Access from WN via Infiniband Software: Host OS SuSE Enterprise 10 sp2 (SLES10sp2) 64 Bit Kernel: smp Virtualisation: KVM-84 Virtual Machines Scientific Linux Cern Bit / 32 Bit Batch System Torque MAUI 3.2.6p19 19

20 Wrap-Job Batch System Virtualisation ViBatch Approach does not require modifications of standard batch systems like Maui/Torque VM image files are deployed on each physics Worker Node Wrapper script around the actual computing job inside the batch system Starts VM on the hardware node assigned to the job Checks status of the VM As soon as VM is ready, the job is forwarded to the VM using ssh and executed inside the VM Having finished the job, the VM is deleted (for security reasons) 20

21 Implementation of ViBatch 21

22 Conclusion & Outlook Virtualisation techniques well suited for server consolidation Extendable to the field of batch systems to offer a dynamic partitioning of a cluster: Customised operating system for each user group Load balancing becomes possible in heterogeneous OS environments Small performance overhead for HEP application acceptable Security / privacy aspects Presented approach: Offers batch system virtualisation without modifications of standard batch systems Not restricted to a dedicated virtualisation technique Selected publications: V. Büge, Y. Kemp, M. Kunze and G. Quast: Application of Virtualization Techniques at a University Grid Center e-science'06 - Second IEEE International Conference on e-science and Grid Computing, ISBN , 2006 V. Büge, Y. Kemp, M. Kunze, O. Oberst and G. Quast: Virtualizing a Batch Queuing System at a University Grid Center Lecture Notes in Computer Science, ISBN~ , Springer 2006 Volker Büge: Virtualisation of Grid Resources and Prospects of the Measurement of Z Boson Production in Association with Jets at the Large Hadron Collider Vdm Verlag, ISBN , November 2008 V. Büge, O. Oberst et. al: Integration of Virtualized Worker Nodes in Standard-Batch-Systems Proceedings of CHEP 2009, Prague, March 2009 (to be published) 22

23 BackUp 23

24 What is Particle Physics? - Big Bang Today Heavy Atoms... Timescale Light Atoms Nuclei Nucleons Energy... Elementary Particles Big Bang 24

25 Our Instruments Detectors - CMS 25

Integration of Virtualized Worker Nodes in Batch Systems

Integration of Virtualized Worker Nodes in Batch Systems Integration of Virtualized Worker Nodes Dr. A. Scheurer, Dr. V. Büge, O. Oberst, P. Krauß Linuxtag 2010, Berlin, Session: Cloud Computing, Talk ID: #16197 KIT University of the State of Baden-Wuerttemberg

More information

Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst

Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst Outline General Description of Virtualization / Virtualization Solutions Shared HPC Infrastructure Virtualization

More information

Virtualization of a Cluster Batch System

Virtualization of a Cluster Batch System Virtualization of a Cluster Batch System Christian Baun, Volker Büge, Benjamin Klein, Jens Mielke, Oliver Oberst and Armin Scheurer Die Kooperation von Cluster Batch System Batch system accepts computational

More information

Computing in High- Energy-Physics: How Virtualization meets the Grid

Computing in High- Energy-Physics: How Virtualization meets the Grid Computing in High- Energy-Physics: How Virtualization meets the Grid Yves Kemp Institut für Experimentelle Kernphysik Universität Karlsruhe Yves Kemp Barcelona, 10/23/2006 Outline: Problems encountered

More information

Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012

Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Thomas Hauth,, Günter Quast IEKP KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz

More information

Integration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept

Integration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept Integration of Virtualized Workernodes in Batch Queueing Systems, Dr. Armin Scheurer, Oliver Oberst, Prof. Günter Quast INSTITUT FÜR EXPERIMENTELLE KERNPHYSIK FAKULTÄT FÜR PHYSIK KIT University of the

More information

Virtualization Infrastructure at Karlsruhe

Virtualization Infrastructure at Karlsruhe Virtualization Infrastructure at Karlsruhe HEPiX Fall 2007 Volker Buege 1),2), Ariel Garcia 1), Marcus Hardt 1), Fabian Kulla 1),Marcel Kunze 1), Oliver Oberst 1),2), Günter Quast 2), Christophe Saout

More information

Basics of Virtualisation

Basics of Virtualisation Basics of Virtualisation Volker Büge Institut für Experimentelle Kernphysik Universität Karlsruhe Die Kooperation von The x86 Architecture Why do we need virtualisation? x86 based operating systems are

More information

Integrating a heterogeneous and shared Linux cluster into grids

Integrating a heterogeneous and shared Linux cluster into grids Integrating a heterogeneous and shared Linux cluster into grids 1,2 1 1,2 1 V. Büge, U. Felzmann, C. Jung, U. Kerzel, 1 1 1 M. Kreps, G. Quast, A. Vest 1 2 DPG Frühjahrstagung March 28 31, 2006 Dortmund

More information

Solving the Mysteries of the Universe with Big Data

Solving the Mysteries of the Universe with Big Data Solving the Mysteries of the Universe with Big Data Sverre Jarp CERN openlab CTO Big Data Innovation Summit, Boston, 12 th September 2013 Accelerating Science and Innovation 1 What is CERN? The European

More information

Shared Computing Driving Discovery: From the Large Hadron Collider to Virus Hunting. Frank Würthwein

Shared Computing Driving Discovery: From the Large Hadron Collider to Virus Hunting. Frank Würthwein Shared Computing Driving Discovery: From the Large Hadron Collider to Virus Hunting Frank Würthwein Professor of Physics University of California San Diego February 14th, 2015 The Science of the LHC The

More information

Collaboration, Big Data and the search for the Higgs Boson

Collaboration, Big Data and the search for the Higgs Boson Collaboration, Big Data and the search for the Higgs Boson Intel European Research and Innovation Conference October 23 rd 2012 Andrzej Nowak, CERN openlab Andrzej.Nowak@cern.ch The European Particle Physics

More information

Solving the Mysteries of the Universe with Big Data

Solving the Mysteries of the Universe with Big Data Solving the Mysteries of the Universe with Big Data Sverre Jarp Former CERN openlab CTO Big Data Innovation Summit, Stockholm, 8 th May 2014 Accelerating Science and Innovation 1 What is CERN? The European

More information

Big Data and Storage Management at the Large Hadron Collider

Big Data and Storage Management at the Large Hadron Collider Big Data and Storage Management at the Large Hadron Collider Dirk Duellmann CERN IT, Data & Storage Services Accelerating Science and Innovation CERN was founded 1954: 12 European States Science for Peace!

More information

E-mail: guido.negri@cern.ch, shank@bu.edu, dario.barberis@cern.ch, kors.bos@cern.ch, alexei.klimentov@cern.ch, massimo.lamanna@cern.

E-mail: guido.negri@cern.ch, shank@bu.edu, dario.barberis@cern.ch, kors.bos@cern.ch, alexei.klimentov@cern.ch, massimo.lamanna@cern. *a, J. Shank b, D. Barberis c, K. Bos d, A. Klimentov e and M. Lamanna a a CERN Switzerland b Boston University c Università & INFN Genova d NIKHEF Amsterdam e BNL Brookhaven National Laboratories E-mail:

More information

(Possible) HEP Use Case for NDN. Phil DeMar; Wenji Wu NDNComm (UCLA) Sept. 28, 2015

(Possible) HEP Use Case for NDN. Phil DeMar; Wenji Wu NDNComm (UCLA) Sept. 28, 2015 (Possible) HEP Use Case for NDN Phil DeMar; Wenji Wu NDNComm (UCLA) Sept. 28, 2015 Outline LHC Experiments LHC Computing Models CMS Data Federation & AAA Evolving Computing Models & NDN Summary Phil DeMar:

More information

NT1: An example for future EISCAT_3D data centre and archiving?

NT1: An example for future EISCAT_3D data centre and archiving? March 10, 2015 1 NT1: An example for future EISCAT_3D data centre and archiving? John White NeIC xx March 10, 2015 2 Introduction High Energy Physics and Computing Worldwide LHC Computing Grid Nordic Tier

More information

Status of Grid Activities in Pakistan. FAWAD SAEED National Centre For Physics, Pakistan

Status of Grid Activities in Pakistan. FAWAD SAEED National Centre For Physics, Pakistan Status of Grid Activities in Pakistan FAWAD SAEED National Centre For Physics, Pakistan 1 Introduction of NCP-LCG2 q NCP-LCG2 is the only Tier-2 centre in Pakistan for Worldwide LHC computing Grid (WLCG).

More information

HIGH ENERGY PHYSICS EXPERIMENTS IN GRID COMPUTING NETWORKS EKSPERYMENTY FIZYKI WYSOKICH ENERGII W SIECIACH KOMPUTEROWYCH GRID. 1.

HIGH ENERGY PHYSICS EXPERIMENTS IN GRID COMPUTING NETWORKS EKSPERYMENTY FIZYKI WYSOKICH ENERGII W SIECIACH KOMPUTEROWYCH GRID. 1. Computer Science Vol. 9 2008 Andrzej Olszewski HIGH ENERGY PHYSICS EXPERIMENTS IN GRID COMPUTING NETWORKS The demand for computing resources used for detector simulations and data analysis in High Energy

More information

Scalable stochastic tracing of distributed data management events

Scalable stochastic tracing of distributed data management events Scalable stochastic tracing of distributed data management events Mario Lassnig mario.lassnig@cern.ch ATLAS Data Processing CERN Physics Department Distributed and Parallel Systems University of Innsbruck

More information

Accelerating Experimental Elementary Particle Physics with the Gordon Supercomputer. Frank Würthwein Rick Wagner August 5th, 2013

Accelerating Experimental Elementary Particle Physics with the Gordon Supercomputer. Frank Würthwein Rick Wagner August 5th, 2013 Accelerating Experimental Elementary Particle Physics with the Gordon Supercomputer Frank Würthwein Rick Wagner August 5th, 2013 The Universe is a strange place! 67% of energy is dark energy We got no

More information

Status and Evolution of ATLAS Workload Management System PanDA

Status and Evolution of ATLAS Workload Management System PanDA Status and Evolution of ATLAS Workload Management System PanDA Univ. of Texas at Arlington GRID 2012, Dubna Outline Overview PanDA design PanDA performance Recent Improvements Future Plans Why PanDA The

More information

Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware

Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware R. Goranova University of Sofia St. Kliment Ohridski,

More information

CMS Tier-3 cluster at NISER. Dr. Tania Moulik

CMS Tier-3 cluster at NISER. Dr. Tania Moulik CMS Tier-3 cluster at NISER Dr. Tania Moulik What and why? Grid computing is a term referring to the combination of computer resources from multiple administrative domains to reach common goal. Grids tend

More information

Virtualization. (and cloud computing at CERN)

Virtualization. (and cloud computing at CERN) Virtualization (and cloud computing at CERN) Ulrich Schwickerath Special thanks: Sebastien Goasguen Belmiro Moreira, Ewan Roche, Romain Wartel See also related presentations: CloudViews2010 conference,

More information

Tier0 plans and security and backup policy proposals

Tier0 plans and security and backup policy proposals Tier0 plans and security and backup policy proposals, CERN IT-PSS CERN - IT Outline Service operational aspects Hardware set-up in 2007 Replication set-up Test plan Backup and security policies CERN Oracle

More information

Long term analysis in HEP: Use of virtualization and emulation techniques

Long term analysis in HEP: Use of virtualization and emulation techniques Long term analysis in HEP: Use of virtualization and emulation techniques Yves Kemp DESY IT First Workshop on Data Preservation and Long Term Analysis in HEP, DESY 26.1.2009 Outline Why virtualization

More information

LHC Databases on the Grid: Achievements and Open Issues * A.V. Vaniachine. Argonne National Laboratory 9700 S Cass Ave, Argonne, IL, 60439, USA

LHC Databases on the Grid: Achievements and Open Issues * A.V. Vaniachine. Argonne National Laboratory 9700 S Cass Ave, Argonne, IL, 60439, USA ANL-HEP-CP-10-18 To appear in the Proceedings of the IV International Conference on Distributed computing and Gridtechnologies in science and education (Grid2010), JINR, Dubna, Russia, 28 June - 3 July,

More information

Computing at the HL-LHC

Computing at the HL-LHC Computing at the HL-LHC Predrag Buncic on behalf of the Trigger/DAQ/Offline/Computing Preparatory Group ALICE: Pierre Vande Vyvre, Thorsten Kollegger, Predrag Buncic; ATLAS: David Rousseau, Benedetto Gorini,

More information

Virtualization across the organization

Virtualization across the organization Virtualization across the organization Server Virtualization Desktop Virtualization Application Virtualization Presentation Virtualization Consolidate workloads for more efficient resource utilization

More information

SUPPORT FOR CMS EXPERIMENT AT TIER1 CENTER IN GERMANY

SUPPORT FOR CMS EXPERIMENT AT TIER1 CENTER IN GERMANY The 5th InternaEonal Conference Distributed CompuEng and Grid technologies in Science and EducaEon SUPPORT FOR CMS EXPERIMENT AT TIER1 CENTER IN GERMANY N. Ratnikova, J. Berger, C. Böser, O. Oberst, G.

More information

Site specific monitoring of multiple information systems the HappyFace Project

Site specific monitoring of multiple information systems the HappyFace Project Home Search Collections Journals About Contact us My IOPscience Site specific monitoring of multiple information systems the HappyFace Project This content has been downloaded from IOPscience. Please scroll

More information

OSG Hadoop is packaged into rpms for SL4, SL5 by Caltech BeStMan, gridftp backend

OSG Hadoop is packaged into rpms for SL4, SL5 by Caltech BeStMan, gridftp backend Hadoop on HEPiX storage test bed at FZK Artem Trunov Karlsruhe Institute of Technology Karlsruhe, Germany KIT The cooperation of Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH) www.kit.edu

More information

Open access to data and analysis tools from the CMS experiment at the LHC

Open access to data and analysis tools from the CMS experiment at the LHC Open access to data and analysis tools from the CMS experiment at the LHC Thomas McCauley (for the CMS Collaboration and QuarkNet) University of Notre Dame, USA thomas.mccauley@cern.ch! 5 Feb 2015 Outline

More information

Beyond High Performance Computing: What Matters to CERN

Beyond High Performance Computing: What Matters to CERN Beyond High Performance Computing: What Matters to CERN Pierre VANDE VYVRE for the ALICE Collaboration ALICE Data Acquisition Project Leader CERN, Geneva, Switzerland 2 CERN CERN is the world's largest

More information

Report from SARA/NIKHEF T1 and associated T2s

Report from SARA/NIKHEF T1 and associated T2s Report from SARA/NIKHEF T1 and associated T2s Ron Trompert SARA About SARA and NIKHEF NIKHEF SARA High Energy Physics Institute High performance computing centre Manages the Surfnet 6 network for the Dutch

More information

Data sharing and Big Data in the physical sciences. 2 October 2015

Data sharing and Big Data in the physical sciences. 2 October 2015 Data sharing and Big Data in the physical sciences 2 October 2015 Content Digital curation: Data and metadata Why consider the physical sciences? Astronomy: Video Physics: LHC for example. Video The Research

More information

Single Sign-In User Centered Computing for High Energy Physics

Single Sign-In User Centered Computing for High Energy Physics Single Sign-In User Centered Computing for High Energy Physics Max Fischer, Oliver Oberst, Günter Quast, Marian Zvada ISGC 2013: March 19-22, 2013 Taipei, Taiwan INSTITUT FÜR EXPERIMENTELLE KERNPHYSIK

More information

Server Virtualisation with VMware. Philipp Heckel, University of Mannheim, 4/12/08 1

Server Virtualisation with VMware. Philipp Heckel, University of Mannheim, 4/12/08 1 Server Virtualisation with VMware Philipp Heckel, University of Mannheim, 4/12/08 1 Virtualisation 1. Virtualisation Overview 2. Impact on Service Providers 3. Virtualisation Technology 4. VMware Infrastructure

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details Use cases By scientist By HEP experiment System requirements and installation How to get it? 2 What

More information

Performance Monitoring of the Software Frameworks for LHC Experiments

Performance Monitoring of the Software Frameworks for LHC Experiments Proceedings of the First EELA-2 Conference R. mayo et al. (Eds.) CIEMAT 2009 2009 The authors. All rights reserved Performance Monitoring of the Software Frameworks for LHC Experiments William A. Romero

More information

Data analysis in Par,cle Physics

Data analysis in Par,cle Physics Data analysis in Par,cle Physics From data taking to discovery Tuesday, 13 August 2013 Lukasz Kreczko - Bristol IT MegaMeet 1 $ whoami Lukasz (Luke) Kreczko Par,cle Physicist Graduated in Physics from

More information

Moving Beyond the Web, a Look at the Potential Benefits of Grid Computing for Future Power Networks

Moving Beyond the Web, a Look at the Potential Benefits of Grid Computing for Future Power Networks Moving Beyond the Web, a Look at the Potential Benefits of Grid Computing for Future Power Networks by Malcolm Irving, Gareth Taylor, and Peter Hobson 1999 ARTVILLE, LLC. THE WORD GRID IN GRID-COMPUTING

More information

Linux and the Higgs Particle

Linux and the Higgs Particle Linux and the Higgs Particle Dr. Bernd Panzer-Steindel Computing Fabric Area Manager, CERN/IT Linux World, Frankfurt 27.October 2004 Outline What is CERN The Physics The Physics Tools The Accelerator The

More information

Estonian Scientific Computing Infrastructure (ETAIS)

Estonian Scientific Computing Infrastructure (ETAIS) Estonian Scientific Computing Infrastructure (ETAIS) Week #7 Hardi Teder hardi@eenet.ee University of Tartu March 27th 2013 Overview Estonian Scientific Computing Infrastructure Estonian Research infrastructures

More information

Virtualization with Windows

Virtualization with Windows Virtualization with Windows at CERN Juraj Sucik, Emmanuel Ormancey Internet Services Group Agenda Current status of IT-IS group virtualization service Server Self Service New virtualization features in

More information

Parallels VDI Solution

Parallels VDI Solution Parallels VDI Solution White Paper VDI SIZING A Competitive Comparison of VDI Solution Sizing between Parallels VDI versus VMware VDI www.parallels.com Parallels VDI Sizing. 29 Table of Contents Overview...

More information

Virtual Environments for Prototyping Tier-3 Clusters

Virtual Environments for Prototyping Tier-3 Clusters Virtual Environments for Prototyping Tier-3 Clusters M Mambelli, R Gardner University of Chicago, 5735 S Ellis Ave, Chicago, IL 60637, USA marco@hep.uchicago.edu Abstract. The deployed hierarchy of Tier-1

More information

Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft. Global Grid User Support - GGUS - within the LCG & EGEE environment

Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft. Global Grid User Support - GGUS - within the LCG & EGEE environment Global Grid User Support - GGUS - within the LCG & EGEE environment Abstract: For very large projects like the LHC Computing Grid Project (LCG) involving some 8,000 scientists from universities and laboratories

More information

PES. Batch virtualization and Cloud computing. Part 1: Batch virtualization. Batch virtualization and Cloud computing

PES. Batch virtualization and Cloud computing. Part 1: Batch virtualization. Batch virtualization and Cloud computing Batch virtualization and Cloud computing Batch virtualization and Cloud computing Part 1: Batch virtualization Tony Cass, Sebastien Goasguen, Belmiro Moreira, Ewan Roche, Ulrich Schwickerath, Romain Wartel

More information

High Throughput WAN Data Transfer with Hadoop-based Storage

High Throughput WAN Data Transfer with Hadoop-based Storage High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wüerthwein 1 1 University of California, San

More information

An Integrated CyberSecurity Approach for HEP Grids. Workshop Report. http://hpcrd.lbl.gov/hepcybersecurity/

An Integrated CyberSecurity Approach for HEP Grids. Workshop Report. http://hpcrd.lbl.gov/hepcybersecurity/ An Integrated CyberSecurity Approach for HEP Grids Workshop Report http://hpcrd.lbl.gov/hepcybersecurity/ 1. Introduction The CMS and ATLAS experiments at the Large Hadron Collider (LHC) being built at

More information

GRID computing at LHC Science without Borders

GRID computing at LHC Science without Borders GRID computing at LHC Science without Borders Kajari Mazumdar Department of High Energy Physics Tata Institute of Fundamental Research, Mumbai. Disclaimer: I am a physicist whose research field induces

More information

CMS: Challenges in Advanced Computing Techniques (Big Data, Data reduction, Data Analytics)

CMS: Challenges in Advanced Computing Techniques (Big Data, Data reduction, Data Analytics) CMS: Challenges in Advanced Computing Techniques (Big Data, Data reduction, Data Analytics) With input from: Daniele Bonacorsi, Ian Fisk, Valentin Kuznetsov, David Lange Oliver Gutsche CERN openlab technical

More information

Virtualization, Grid, Cloud: Integration Paths for Scientific Computing

Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Or, where and how will my efficient large-scale computing applications be executed? D. Salomoni, INFN Tier-1 Computing Manager Davide.Salomoni@cnaf.infn.it

More information

Global Grid User Support - GGUS - in the LCG & EGEE environment

Global Grid User Support - GGUS - in the LCG & EGEE environment Global Grid User Support - GGUS - in the LCG & EGEE environment Torsten Antoni (torsten.antoni@iwr.fzk.de) Why Support? New support groups Network layer Resource centers CIC / GOC / etc. more to come New

More information

HAMBURG ZEUTHEN. DESY Tier 2 and NAF. Peter Wegner, Birgit Lewendel for DESY-IT/DV. Tier 2: Status and News NAF: Status, Plans and Questions

HAMBURG ZEUTHEN. DESY Tier 2 and NAF. Peter Wegner, Birgit Lewendel for DESY-IT/DV. Tier 2: Status and News NAF: Status, Plans and Questions DESY Tier 2 and NAF Peter Wegner, Birgit Lewendel for DESY-IT/DV Tier 2: Status and News NAF: Status, Plans and Questions Basics T2: 1.5 average Tier 2 are requested by CMS-groups for Germany Desy commitment:

More information

Shoal: IaaS Cloud Cache Publisher

Shoal: IaaS Cloud Cache Publisher University of Victoria Faculty of Engineering Winter 2013 Work Term Report Shoal: IaaS Cloud Cache Publisher Department of Physics University of Victoria Victoria, BC Mike Chester V00711672 Work Term 3

More information

Grid Computing in Aachen

Grid Computing in Aachen GEFÖRDERT VOM Grid Computing in Aachen III. Physikalisches Institut B Berichtswoche des Graduiertenkollegs Bad Honnef, 05.09.2008 Concept of Grid Computing Computing Grid; like the power grid, but for

More information

A quantitative comparison between xen and kvm

A quantitative comparison between xen and kvm Home Search Collections Journals About Contact us My IOPscience A quantitative comparison between xen and kvm This content has been downloaded from IOPscience. Please scroll down to see the full text.

More information

ATLAS job monitoring in the Dashboard Framework

ATLAS job monitoring in the Dashboard Framework ATLAS job monitoring in the Dashboard Framework J Andreeva 1, S Campana 1, E Karavakis 1, L Kokoszkiewicz 1, P Saiz 1, L Sargsyan 2, J Schovancova 3, D Tuckett 1 on behalf of the ATLAS Collaboration 1

More information

U-LITE: a proposal for scientific computing at LNGS. S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011

U-LITE: a proposal for scientific computing at LNGS. S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011 U-LITE: a proposal for scientific computing at LNGS S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011 20 years of Scientific Computing at LNGS Early 90s: highly centralized structure based on VMS cluster

More information

Alternative models to distribute VO specific software to WLCG sites: a prototype set up at PIC

Alternative models to distribute VO specific software to WLCG sites: a prototype set up at PIC EGEE and glite are registered trademarks Enabling Grids for E-sciencE Alternative models to distribute VO specific software to WLCG sites: a prototype set up at PIC Elisa Lanciotti, Arnau Bria, Gonzalo

More information

ZEN LOAD BALANCER EE v3.04 DATASHEET The Load Balancing made easy

ZEN LOAD BALANCER EE v3.04 DATASHEET The Load Balancing made easy ZEN LOAD BALANCER EE v3.04 DATASHEET The Load Balancing made easy OVERVIEW The global communication and the continuous growth of services provided through the Internet or local infrastructure require to

More information

Object Database Scalability for Scientific Workloads

Object Database Scalability for Scientific Workloads Object Database Scalability for Scientific Workloads Technical Report Julian J. Bunn Koen Holtman, Harvey B. Newman 256-48 HEP, Caltech, 1200 E. California Blvd., Pasadena, CA 91125, USA CERN EP-Division,

More information

A High-Performance Storage System for the LHCb Experiment Juan Manuel Caicedo Carvajal, Jean-Christophe Garnier, Niko Neufeld, and Rainer Schwemmer

A High-Performance Storage System for the LHCb Experiment Juan Manuel Caicedo Carvajal, Jean-Christophe Garnier, Niko Neufeld, and Rainer Schwemmer 658 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 57, NO. 2, APRIL 2010 A High-Performance Storage System for the LHCb Experiment Juan Manuel Caicedo Carvajal, Jean-Christophe Garnier, Niko Neufeld, and Rainer

More information

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb Batch and Cloud overview Andrew McNab University of Manchester GridPP and LHCb Overview Assumptions Batch systems The Grid Pilot Frameworks DIRAC Virtual Machines Vac Vcycle Tier-2 Evolution Containers

More information

wu.cloud: Insights Gained from Operating a Private Cloud System

wu.cloud: Insights Gained from Operating a Private Cloud System wu.cloud: Insights Gained from Operating a Private Cloud System Stefan Theußl, Institute for Statistics and Mathematics WU Wirtschaftsuniversität Wien March 23, 2011 1 / 14 Introduction In statistics we

More information

Distributed Computing for CEPC. YAN Tian On Behalf of Distributed Computing Group, CC, IHEP for 4 th CEPC Collaboration Meeting, Sep.

Distributed Computing for CEPC. YAN Tian On Behalf of Distributed Computing Group, CC, IHEP for 4 th CEPC Collaboration Meeting, Sep. Distributed Computing for CEPC YAN Tian On Behalf of Distributed Computing Group, CC, IHEP for 4 th CEPC Collaboration Meeting, Sep. 12-13, 2014 1 Outline Introduction Experience of BES-DIRAC Distributed

More information

Database Monitoring Requirements. Salvatore Di Guida (CERN) On behalf of the CMS DB group

Database Monitoring Requirements. Salvatore Di Guida (CERN) On behalf of the CMS DB group Database Monitoring Requirements Salvatore Di Guida (CERN) On behalf of the CMS DB group Outline CMS Database infrastructure and data flow. Data access patterns. Requirements coming from the hardware and

More information

Big Data and Cloud Computing for GHRSST

Big Data and Cloud Computing for GHRSST Big Data and Cloud Computing for GHRSST Jean-Francois Piollé (jfpiolle@ifremer.fr) Frédéric Paul, Olivier Archer CERSAT / Institut Français de Recherche pour l Exploitation de la Mer Facing data deluge

More information

CMS Experience with Online and Offline Databases

CMS Experience with Online and Offline Databases CMS Experience with Online and Offline Databases Dr. for the CMS experiment CHEP 2012, New York (NY), USA 1 Outline Overview The Challenge Conditions data: what and how DB Evolution and Performance Monitoring

More information

Building a Private Cloud with Eucalyptus

Building a Private Cloud with Eucalyptus Building a Private Cloud with Eucalyptus 5th IEEE International Conference on e-science Oxford December 9th 2009 Christian Baun, Marcel Kunze KIT The cooperation of Forschungszentrum Karlsruhe GmbH und

More information

Distributed Database Services - a Fundamental Component of the WLCG Service for the LHC Experiments - Experience and Outlook

Distributed Database Services - a Fundamental Component of the WLCG Service for the LHC Experiments - Experience and Outlook Distributed Database Services - a Fundamental Component of the WLCG Service for the LHC Experiments - Experience and Outlook Maria Girone CERN, 1211 Geneva 23, Switzerland Maria.Girone@cern.ch Abstract.

More information

Peer-to-peer cooperative scheduling architecture for National Grid Infrastructure

Peer-to-peer cooperative scheduling architecture for National Grid Infrastructure Peer-to-peer cooperative scheduling architecture for National Grid Infrastructure L. Matyska, M. Ruda, S. Toth CESNET Czech Republic 10 th March 2010 ISGC2010 (Taipei, Taiwan) Cooperative scheduling 10

More information

Summer Student Project Report

Summer Student Project Report Summer Student Project Report Dimitris Kalimeris National and Kapodistrian University of Athens June September 2014 Abstract This report will outline two projects that were done as part of a three months

More information

Intro to Virtualization

Intro to Virtualization Cloud@Ceid Seminars Intro to Virtualization Christos Alexakos Computer Engineer, MSc, PhD C. Sysadmin at Pattern Recognition Lab 1 st Seminar 19/3/2014 Contents What is virtualization How it works Hypervisor

More information

KIT Site Report. Andreas Petzold. www.kit.edu STEINBUCH CENTRE FOR COMPUTING - SCC

KIT Site Report. Andreas Petzold. www.kit.edu STEINBUCH CENTRE FOR COMPUTING - SCC KIT Site Report Andreas Petzold STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association www.kit.edu GridKa Tier 1 - Batch

More information

Online Monitoring in the CDF II experiment

Online Monitoring in the CDF II experiment Online Monitoring in the CDF II experiment 3t, Tetsuo Arisawa 4, Koji Ikado 4, Kaori Maeshima 1, Hartmut Stadie 3, Greg Veramendi 2, Hans Wenzel 1 1 Fermi National Accelerator Laboratory, Batavia, U.S.A.

More information

Status and Integration of AP2 Monitoring and Online Steering

Status and Integration of AP2 Monitoring and Online Steering Status and Integration of AP2 Monitoring and Online Steering Daniel Lorenz - University of Siegen Stefan Borovac, Markus Mechtel - University of Wuppertal Ralph Müller-Pfefferkorn Technische Universität

More information

Distributed analysis for the ATLAS Experiment in the S.Co.P.E Project

Distributed analysis for the ATLAS Experiment in the S.Co.P.E Project FINAL WORKSHOP OF GRID PROJECTS, PON RICERCA 2000-2006, AVVISO 1575 1 Distributed analysis for the ATLAS Experiment in the S.Co.P.E Project A. Aloisio 1,2, M. Alviggi 1,2, M. Biglietti 1,2, V. Canale 1,2,

More information

Evolution of Database Replication Technologies for WLCG

Evolution of Database Replication Technologies for WLCG Home Search Collections Journals About Contact us My IOPscience Evolution of Database Replication Technologies for WLCG This content has been downloaded from IOPscience. Please scroll down to see the full

More information

The CMS analysis chain in a distributed environment

The CMS analysis chain in a distributed environment The CMS analysis chain in a distributed environment on behalf of the CMS collaboration DESY, Zeuthen,, Germany 22 nd 27 th May, 2005 1 The CMS experiment 2 The CMS Computing Model (1) The CMS collaboration

More information

Lecture 2 Cloud Computing & Virtualization. Cloud Application Development (SE808, School of Software, Sun Yat-Sen University) Yabo (Arber) Xu

Lecture 2 Cloud Computing & Virtualization. Cloud Application Development (SE808, School of Software, Sun Yat-Sen University) Yabo (Arber) Xu Lecture 2 Cloud Computing & Virtualization Cloud Application Development (SE808, School of Software, Sun Yat-Sen University) Yabo (Arber) Xu Outline Introduction to Virtualization The Major Approaches

More information

Software, Computing and Analysis Models at CDF and D0

Software, Computing and Analysis Models at CDF and D0 Software, Computing and Analysis Models at CDF and D0 Donatella Lucchesi CDF experiment INFN-Padova Outline Introduction CDF and D0 Computing Model GRID Migration Summary III Workshop Italiano sulla fisica

More information

HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk

HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk HPC and Big Data EPCC The University of Edinburgh Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk EPCC Facilities Technology Transfer European Projects HPC Research Visitor Programmes Training

More information

From Distributed Computing to Distributed Artificial Intelligence

From Distributed Computing to Distributed Artificial Intelligence From Distributed Computing to Distributed Artificial Intelligence Dr. Christos Filippidis, NCSR Demokritos Dr. George Giannakopoulos, NCSR Demokritos Big Data and the Fourth Paradigm The two dominant paradigms

More information

High Availability Databases based on Oracle 10g RAC on Linux

High Availability Databases based on Oracle 10g RAC on Linux High Availability Databases based on Oracle 10g RAC on Linux WLCG Tier2 Tutorials, CERN, June 2006 Luca Canali, CERN IT Outline Goals Architecture of an HA DB Service Deployment at the CERN Physics Database

More information

ZEN LOAD BALANCER EE v3.02 DATASHEET The Load Balancing made easy

ZEN LOAD BALANCER EE v3.02 DATASHEET The Load Balancing made easy ZEN LOAD BALANCER EE v3.02 DATASHEET The Load Balancing made easy OVERVIEW The global communication and the continuous growth of services provided through the Internet or local infrastructure require to

More information

Building a Volunteer Cloud

Building a Volunteer Cloud Building a Volunteer Cloud Ben Segal, Predrag Buncic, David Garcia Quintas / CERN Daniel Lombrana Gonzalez / University of Extremadura Artem Harutyunyan / Yerevan Physics Institute Jarno Rantala / Tampere

More information

PoS(EGICF12-EMITC2)110

PoS(EGICF12-EMITC2)110 User-centric monitoring of the analysis and production activities within the ATLAS and CMS Virtual Organisations using the Experiment Dashboard system Julia Andreeva E-mail: Julia.Andreeva@cern.ch Mattia

More information

JUROPA Linux Cluster An Overview. 19 May 2014 Ulrich Detert

JUROPA Linux Cluster An Overview. 19 May 2014 Ulrich Detert Mitglied der Helmholtz-Gemeinschaft JUROPA Linux Cluster An Overview 19 May 2014 Ulrich Detert JuRoPA JuRoPA Jülich Research on Petaflop Architectures Bull, Sun, ParTec, Intel, Mellanox, Novell, FZJ JUROPA

More information

Performance monitoring of the software frameworks for LHC experiments

Performance monitoring of the software frameworks for LHC experiments Performance monitoring of the software frameworks for LHC experiments William A. Romero R. wil-rome@uniandes.edu.co J.M. Dana Jose.Dana@cern.ch First EELA-2 Conference Bogotá, COL OUTLINE Introduction

More information

Global Grid User Support - GGUS - start up schedule

Global Grid User Support - GGUS - start up schedule Global Grid User Support - GGUS - start up schedule GDB Meeting 2004-07 07-13 Concept Target: 24 7 support via time difference and 3 support teams Currently: GGUS FZK GGUS ASCC Planned: GGUS USA Support

More information

Data Management Plan (DMP) for Particle Physics Experiments prepared for the 2015 Consolidated Grants Round. Detailed Version

Data Management Plan (DMP) for Particle Physics Experiments prepared for the 2015 Consolidated Grants Round. Detailed Version Data Management Plan (DMP) for Particle Physics Experiments prepared for the 2015 Consolidated Grants Round. Detailed Version The Particle Physics Experiment Consolidated Grant proposals now being submitted

More information

Strategies for increasing data centre power efficiency

Strategies for increasing data centre power efficiency Strategies for increasing data centre power efficiency An overview of CERN s approach to energy efficient computing Dr. Andreas Hirstius, Sverre Jarp, Andrzej Nowak CERN openlab 19 February 2008 1 Table

More information

Implementing a Digital Video Archive Using XenData Software and a Spectra Logic Archive

Implementing a Digital Video Archive Using XenData Software and a Spectra Logic Archive Using XenData Software and a Spectra Logic Archive With the Video Edition of XenData Archive Series software on a Windows server and a Spectra Logic T-Series digital archive, broadcast organizations have

More information

CMS Software Deployment on OSG

CMS Software Deployment on OSG CMS Software Deployment on OSG Bockjoo Kim 1, Michael Thomas 2, Paul Avery 1, Frank Wuerthwein 3 1. University of Florida, Gainesville, FL 32611, USA 2. California Institute of Technology, Pasadena, CA

More information

Red Hat Enterprise Linux 6. Stanislav Polášek ELOS Technologies sp@elostech.cz

Red Hat Enterprise Linux 6. Stanislav Polášek ELOS Technologies sp@elostech.cz Stanislav Polášek ELOS Technologies sp@elostech.cz Red Hat - an Established Global Leader Compiler Development Identity & Authentication Storage & File Systems Middleware Kernel Development Virtualization

More information

Cloud Computing PES. (and virtualization at CERN) Cloud Computing. GridKa School 2011, Karlsruhe. Disclaimer: largely personal view of things

Cloud Computing PES. (and virtualization at CERN) Cloud Computing. GridKa School 2011, Karlsruhe. Disclaimer: largely personal view of things PES Cloud Computing Cloud Computing (and virtualization at CERN) Ulrich Schwickerath et al With special thanks to the many contributors to this presentation! GridKa School 2011, Karlsruhe CERN IT Department

More information