DYNAMIC ALLOCATION AND CLOUD COMPUTING IN CMS

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "DYNAMIC ALLOCATION AND CLOUD COMPUTING IN CMS"

Transcription

1 DYNAMIC ALLOCATION AND CLOUD COMPUTING IN CMS Massimo Sgaravatto INFN Padova Claudio Grandi INFN Bologna

2 CMS VIEW ON CLOUDS From January GDB meeting CMS is reasonably happy with the current resource allocation system in use on CMS-owned resources Including shared sites where CMS owns a fraction of the resources Any change, including the use of a Cloud interface, is acceptable provided there is no significant efficiency degradation Commercial or in general opportunistic Clouds may be interesting for absorbing usage peaks CMS is active in adapting the job submission framework to Cloud interfaces 2

3 CMS VIEW ON CLOUDS (CONT.ED) From Spring offline and computing week We should not go to heavy in-house engineering in order to solve problems already solved by current (old?) technologies. Either our requirements are addressed by the "cloud community" or we try to exploit what we have. In the end going to cloud for us is a way to use commonly used software and thus reduce the need to develop specific solutions. 3

4 CMS JOB SUBMISSION FRAMEWORK Mainly based on the pilot job approach Separate resource provisioning from resource scheduling Pilot jobs (not user jobs) are submitted as normal Grid jobs to the available sites to create an overlay batch system User jobs are then run on this overlay batch system, whose size changes according to the resource availability in the Grid Implemented via GlideinWMS Heavily HTCondor based The overlay batch system is HTCondor The pilot, called glidein, basically installs and configure the batch slot as a Condor executing machine Implemented by USCMS. Support currently provided by FNAL 4

5 GLIDEINWMS OVERVIEW Pilots are submitted to all Grid sites by GlideinWMS factories As soon as a pilot (glidein) starts running, that allocated job slot joins the HTCondor user pool User jobs are Condor jobs, submitted to HTCondor user pool Even if users don t realize to use HTCondor, since this is hidden by the experiment software framework Grid site GlideinWMS factory Grid site GlideIn Grid site HTCondor user job repo User Job 5

6 DYNAMIC PROVISIONING ON GRID & CLOUD Cloud support introduced with GlideinWMS v3 Same architecture: submission of Glideins vs instantation of VMs where glideins run job User pool (HTCondor) VO frontend Glidein factory Condor-G job User pool (HTCondor) VO frontend Glidein factory Condor-G Condor startd Grid Site Worker node glidein Condor startd glidein bootstrap Virtual machine 6 Cloud Site

7 GLIDEINWMS &CLOUDS HTCondor (and GlideinWMS) support Clouds providing an EC2 interface This does not mean that GlideinWMS works out of the box with any EC2 based cloud Tested with Amazon EC2, Openstack, Eucalyptus Actually HTCondor supports also DeltaCloud (Cloud translation service) But no experiences with DeltaCloud with GlideinWMS AFAIK no plans to support OCCI in HTCondor (and therefore in GlideinWMS) 7

8 IMAGES VMs instantiated by GlideinWMS factory Images must be registered in advance in the relevant Clouds Glidein bootstrap RPMs must be part of the VM image Image id, flavor, and EC2 credentials specified in the GlideinWMS Frontend or Factory configuration file GlideinWMS creates a different SSH key for each requested VM If a key is compromised, only a VM is affected But this also means that troubleshooting problems in the VM is not straightforward, unless you have access to the GlideinWMS factory 8

9 CONTEXTUALIZATION User-data used only for Glidein configuration Tarball with glidein startup script, proxy, details about factory and frontend Normal site contextualization is not possible Difficult to have a single certified image usable everywhere + specific local customization However some specific site (entry) customizations scripts can be defined at the GlideinWMS level Issue planned to be addressed in GlideinWMS after V3.1 9

10 GLIDEIN IN CLOUD: HOW IT WORKS Glidein started as service when the VM is created Everything runs as an unprivileges user (created by glidein RPM prescript) Glidein configuration in user-data Download startup script Perform checks defined in the Factory and/or Frontend glideinwms Download and execute HTCondor startd Startd runs multiple single core jobs and/or multicore jobs to fill the machine Glidein stops its execution in case of error or when there is no more work VM stopped and deleted 10

11 VM LIFETIME Glidein lifetime is VM lifetime Configurable at GlideinWMS level in the very same way as normal glideins Possibility to configure Maxixum time the VM can be idle before shutting down Retire time: no new user jobs are accepted after this time Job Max Time: after Retire time is exceeded, user jobs can run at most for this time otherwise they are killed Max wall time: maximum overall time for the VM 11

12 TESTING AT OpenStack based IaaS cloud CERNVM based images CVMFS for experiment software Xrootd for I/O from/to EOS First test with VM instantiation and job submission manually handled MonteCarlo and analysis jobs Very high job efficiency 12

13 TESTING AT (CONT.ED) Second test using GlideinWMS 800 cores Started with 8 cores -16 GB images; then moved to 4 cores - 8 GB images for faster allocation time Montecarlo jobs (WMAgent) Workflows completed without major problems Analysis jobs (CRAB2 + RemoteGlidein) 9581 jobs, 96% efficiency Issue with high load in Clod Controller Addressed in GlideinWMS with bulk queries and decreasing polling rate 13

14 THE CMS HLT GETTING CMSOOOOOCLOUD CMS High Level Trigger Farm 13.3K cores, ~ 195 KHS06 Not shared with other experiments Turning this farm into a Cloud To use it during LS1 and in general whenever not used Overlay infrastructure: Minimal changes to convert HLT nodes in Cloud compute nodes Cloud middleware OpenStack Essex, moving to Grizzly 1 Cloud controller 1 Glance server (images) 1300 Compute nodes KVM virtualization Openvswitch used to virtualize network CMS Openstack, OpenSwitch-ed, Opportunistic, Overlay, Online-cluster Cloud (CMSoooooCloud) 14

15 HLT-CLOUD TESTING Images Created initially using CERNVM, then moved to BoxGrinder CERNVM uses Conary instead of RPMs SL 5.x + CA certificates + CVMFS + Glidein bootstrap + xrootd client CMS software through CVMFS 16 cores VMs I/O from/to EOS using xrootd Glexec not used But there are no technical problems to use it, provided that the VM is properly configure to use it VM instantiations and job submissions managed using GlideinWMS v3 Jobs submitted using WMAgent 15

16 HLT CLOUD TESTING (CONT.ED) Many issues, now basically addressed, e.g.: CVMFS not available fast enough Buggy xrootd client in the EPEL repo installed in the VM image Termination of stopped VMs was not performed Openstack ignoring the InstanceInstantiatedShutdownBehavior EC2 setting Network saturation preventing to scale up Network update in progress Reprocessing of 2011 data as validation task to start soon 16

17 OTHER CMS CLOUD RELATED ACTIVITIES GridPP Cloud Pilot in UK Replicated HLT setup in smaller scale RAL Openstack IC (7 compute nodes, 200 cores) Being tested with CRAB2 analysis jobs The relevant people are the same active in the HLT-cloud testing activities In Italy Also replicating (in smaller scale) the CMS-HLT setup Glidein WMS instance at T2 Padova-Legnaro for Cloud testing Small Folsom Openstack installation at T2 Padova-Legnaro 1 controller node + 2 Compute nodes Starting using also a small Grizzly Openstack installation at CNAF Testing with CRAB2 & RemoteGlidein but the final goal is to evaluate something different (see later) 17

18 TESTING IN ITALY CRAB2 jobs using T2 Padova-Legnaro Data read using xrootd, write using SRM 18

19 CAN A CLOUD REPLACE A GRID CE? E.g. for a T2? Some pros Big (and increasing) Cloud community In most of the cases we can hopefully see implemented (instead of implementing) what is needed Less resources available for Grid middleware maintenance and evolution now Easier to manage different requirements in terms of OS and libraries by the different VOs Site admins relieved from lot of work They have just to install and maintain an OpenStack (or whatever) installation No middleware installation/update, no experiment software to install/configure, etc. Some cons Site admins give up a degree of control on their resources It is even not straightforward to access the VMs Fair-share management in sites supporting multiple VOs 19

20 SCHEDULING In Grid fair-share among the VOs is implemented by the batch system This is something which doesn t exist in the IaaS clouds that were not designed as job submission facilities There is not a queuing system for VM requests If there are no more resources, the request to instantiate a new VM simply fails With the current implementations the only solution would be to statically partition the resources, which is something we don t want 20

21 SCHEDULING (CONT.ED) Some solutions proposed in the context of the WLCG GDB cloud working group, but they would require significant efforts for their implementations: Implementation via graceful termination of VMs VMs can be shutdown by the site if/when needed SLA specifying X minimum hours of shutdown notice for VMs VO A let the site knows they would like more resources Graceful termination of VMs for VO B that were using unused resources Economic model VOs are given credits The price of a VM increases with its duration an the number of VMs owned by a VO 21

22 WACK WACK supposed to address this issue Wnodes + OpenStack integration Exposing (also) EC2 interface therefore usable by GlideinWMS Wnodes engine used to instantiate VMs Using the batch system where fair-share can be implemented See Davide s talk Agreed to try GlideinWMS WACK interactions as soon as some WACK prototype appears 22

23 Wack - Davide WS CCR, Salomoni 27-31/5/

24 CONCLUSIONS Overall job submission architecture not changed to support Clouds Dynamic provisioning of VM for Clouds Dynamic provisioning of job slots for Grids Not far to be production ready for the use of opportunistic Clouds where scheduling is not needed E.g. HLT farm Old technology (i.e. Grid) is probably still better for sites supporting multi-vos Not straitghforward to assure fair-share and efficient resource usage with the existing implementations 24

25 25 BACKUP SLIDES

26 Frontend GLIDEINWMS ARCHITECTURE Knows about user jobs and requests Glideins Constant pressure policy : keep a certain number of idle glideins all time Operated by VO admins Factory Knowns about sites and submit Glideins as requested by the frontend Not a VO specific service Glidein Validate environment (installed software, disk space, etc.) Download, configure and run HTCondor daemon Does cleanup at the end 26

27 GLIDEINWMS AT A GLANCE 27

CHEP 2013. Cloud Bursting with glideinwms Means to satisfy ever increasing computing needs for Scientific Workflows

CHEP 2013. Cloud Bursting with glideinwms Means to satisfy ever increasing computing needs for Scientific Workflows CHEP 2013 Cloud Bursting with glideinwms Means to satisfy ever increasing computing needs for Scientific Workflows by I. Sfiligoi 1, P. Mhashilkar 2, A. Tiradani 2, B. Holzman 2, K. Larson 2 and M. Rynge

More information

WM Technical Evolution Group Report

WM Technical Evolution Group Report WM Technical Evolution Group Report Davide Salomoni, INFN for the WM TEG February 7, 2012 The WM TEG, organization Mailing list, wlcg-teg-workload-mgmt@cern.ch 48 people currently subscribed; representation

More information

CMS Experience Provisioning Cloud Resources with GlideinWMS. Anthony Tiradani HTCondor Week 2015 20 May 2015

CMS Experience Provisioning Cloud Resources with GlideinWMS. Anthony Tiradani HTCondor Week 2015 20 May 2015 CMS Experience Provisioning Cloud Resources with GlideinWMS Anthony Tiradani Week 2015 20 May 2015 glideinwms Quick Facts glideinwms is an open- source Fermilab CompuJng Sector product driven by CMS Heavy

More information

The CMS Tier0 goes Cloud and Grid for LHC Run 2. Dirk Hufnagel (FNAL) for CMS Computing

The CMS Tier0 goes Cloud and Grid for LHC Run 2. Dirk Hufnagel (FNAL) for CMS Computing The CMS Tier0 goes Cloud and Grid for LHC Run 2 Dirk Hufnagel (FNAL) for CMS Computing CHEP, 13.04.2015 Overview Changes for the Tier0 between Run 1 and Run 2 CERN Agile Infrastructure (in GlideInWMS)

More information

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb Batch and Cloud overview Andrew McNab University of Manchester GridPP and LHCb Overview Assumptions Batch systems The Grid Pilot Frameworks DIRAC Virtual Machines Vac Vcycle Tier-2 Evolution Containers

More information

A Web-based Portal to Access and Manage WNoDeS Virtualized Cloud Resources

A Web-based Portal to Access and Manage WNoDeS Virtualized Cloud Resources A Web-based Portal to Access and Manage WNoDeS Virtualized Cloud Resources Davide Salomoni 1, Daniele Andreotti 1, Luca Cestari 2, Guido Potena 2, Peter Solagna 3 1 INFN-CNAF, Bologna, Italy 2 University

More information

Virtualisation Cloud Computing at the RAL Tier 1. Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013

Virtualisation Cloud Computing at the RAL Tier 1. Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013 Virtualisation Cloud Computing at the RAL Tier 1 Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013 Virtualisation @ RAL Context at RAL Hyper-V Services Platform Scientific Computing Department

More information

An objective comparison test of workload management systems

An objective comparison test of workload management systems An objective comparison test of workload management systems Igor Sfiligoi 1 and Burt Holzman 1 1 Fermi National Accelerator Laboratory, Batavia, IL 60510, USA E-mail: sfiligoi@fnal.gov Abstract. The Grid

More information

HTCondor at the RAL Tier-1

HTCondor at the RAL Tier-1 HTCondor at the RAL Tier-1 Andrew Lahiff, Alastair Dewhurst, John Kelly, Ian Collier, James Adams STFC Rutherford Appleton Laboratory HTCondor Week 2014 Outline Overview of HTCondor at RAL Monitoring Multi-core

More information

Single Sign-In User Centered Computing for High Energy Physics

Single Sign-In User Centered Computing for High Energy Physics Single Sign-In User Centered Computing for High Energy Physics Max Fischer, Oliver Oberst, Günter Quast, Marian Zvada ISGC 2013: March 19-22, 2013 Taipei, Taiwan INSTITUT FÜR EXPERIMENTELLE KERNPHYSIK

More information

The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)

The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud) 15-19 April 2013, Bologna Italy HEPiX Spring 2013 Workshop Wojciech Ozga Faculty of Computer Science, Electronics and Telecommunication AGH University of Science and Technology in Krakow, Poland CERN,

More information

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases NASA Ames NASA Advanced Supercomputing (NAS) Division California, May 24th, 2012 Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases Ignacio M. Llorente Project Director OpenNebula Project.

More information

CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment

CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment George Lestaris - Ioannis Charalampidis D. Berzano, J. Blomer, P. Buncic, G. Ganis and R. Meusel PH-SFT /

More information

Running real jobs in virtual machines. Andrew McNab University of Manchester

Running real jobs in virtual machines. Andrew McNab University of Manchester Running real jobs in virtual machines Andrew McNab University of Manchester Overview The Grid we have now Strategies for starting VMs Fabric / Cloud / Vac Hypervisors Disk images Contextualization Pilot

More information

Optimization of Italian CMS Computing Centers via MIUR funded Research Projects

Optimization of Italian CMS Computing Centers via MIUR funded Research Projects Optimization of Italian CMS Computing Centers via MIUR funded Research Projects T.Boccali 1, G.Donvito 2, A.Pompili 2, G.Della Ricca 3, E.Mazzoni 1, S.Argiro 4, C.Grandi 5, D.Bonacorsi 6, L.Lista 12, F.Fabozzi

More information

CONDOR CLUSTERS ON EC2

CONDOR CLUSTERS ON EC2 CONDOR CLUSTERS ON EC2 Val Hendrix, Roberto A. Vitillo Lawrence Berkeley National Lab ATLAS Cloud Computing R & D 1 INTRODUCTION This is our initial work on investigating tools for managing clusters and

More information

The CMS analysis chain in a distributed environment

The CMS analysis chain in a distributed environment The CMS analysis chain in a distributed environment on behalf of the CMS collaboration DESY, Zeuthen,, Germany 22 nd 27 th May, 2005 1 The CMS experiment 2 The CMS Computing Model (1) The CMS collaboration

More information

Virtualization, Grid, Cloud: Integration Paths for Scientific Computing

Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Or, where and how will my efficient large-scale computing applications be executed? D. Salomoni, INFN Tier-1 Computing Manager Davide.Salomoni@cnaf.infn.it

More information

Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012

Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Thomas Hauth,, Günter Quast IEKP KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz

More information

Welcome. Grid on Demand. Willem Toorop and Alain van Hoof. {wtoorop,ahoof}@os3.nl. June 30, 2010

Welcome. Grid on Demand. Willem Toorop and Alain van Hoof. {wtoorop,ahoof}@os3.nl. June 30, 2010 Welcome Grid on Demand Willem Toorop and Alain van Hoof {wtoorop,ahoof}@os3.nl June 30, 2010 Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 1 / 39 Research Question Introduction Research

More information

Computing in High- Energy-Physics: How Virtualization meets the Grid

Computing in High- Energy-Physics: How Virtualization meets the Grid Computing in High- Energy-Physics: How Virtualization meets the Grid Yves Kemp Institut für Experimentelle Kernphysik Universität Karlsruhe Yves Kemp Barcelona, 10/23/2006 Outline: Problems encountered

More information

Cloud-pilot.doc 12-12-2010 SA1 Marcus Hardt, Marcin Plociennik, Ahmad Hammad, Bartek Palak E U F O R I A

Cloud-pilot.doc 12-12-2010 SA1 Marcus Hardt, Marcin Plociennik, Ahmad Hammad, Bartek Palak E U F O R I A Identifier: Date: Activity: Authors: Status: Link: Cloud-pilot.doc 12-12-2010 SA1 Marcus Hardt, Marcin Plociennik, Ahmad Hammad, Bartek Palak E U F O R I A J O I N T A C T I O N ( S A 1, J R A 3 ) F I

More information

Adding IaaS Clouds to the ATLAS Computing Grid

Adding IaaS Clouds to the ATLAS Computing Grid Adding IaaS Clouds to the ATLAS Computing Grid Ashok Agarwal, Frank Berghaus, Andre Charbonneau, Mike Chester, Asoka de Silva, Ian Gable, Joanna Huang, Colin Leavett-Brown, Michael Paterson, Randall Sobie,

More information

Dynamic Resource Provisioning with HTCondor in the Cloud

Dynamic Resource Provisioning with HTCondor in the Cloud Dynamic Resource Provisioning with HTCondor in the Cloud Ryan Taylor Frank Berghaus 1 Overview Review of Condor + Cloud Scheduler system Condor job slot configuration Dynamic slot creation Automatic slot

More information

Mobile Cloud Computing T-110.5121 Open Source IaaS

Mobile Cloud Computing T-110.5121 Open Source IaaS Mobile Cloud Computing T-110.5121 Open Source IaaS Tommi Mäkelä, Otaniemi Evolution Mainframe Centralized computation and storage, thin clients Dedicated hardware, software, experienced staff High capital

More information

OSG PUBLIC STORAGE. Tanya Levshina

OSG PUBLIC STORAGE. Tanya Levshina PUBLIC STORAGE Tanya Levshina Motivations for Public Storage 2 data to use sites more easily LHC VOs have solved this problem (FTS, Phedex, LFC) Smaller VOs are still struggling with large data in a distributed

More information

University of Messina, Italy

University of Messina, Italy University of Messina, Italy IEEE MoCS 2011 Kerkyra - Greece June 28, 2011 Dr. Massimo Villari mvillari@unime.it Cross Cloud Federation Federated Cloud Scenario Cloud Middleware Model: the Stack The CLEVER

More information

An Efficient Use of Virtualization in Grid/Cloud Environments. Supervised by: Elisa Heymann Miquel A. Senar

An Efficient Use of Virtualization in Grid/Cloud Environments. Supervised by: Elisa Heymann Miquel A. Senar An Efficient Use of Virtualization in Grid/Cloud Environments. Arindam Choudhury Supervised by: Elisa Heymann Miquel A. Senar Index Introduction Motivation Objective State of Art Proposed Solution Experimentations

More information

Dynamic Resource Distribution Across Clouds

Dynamic Resource Distribution Across Clouds University of Victoria Faculty of Engineering Winter 2010 Work Term Report Dynamic Resource Distribution Across Clouds Department of Physics University of Victoria Victoria, BC Michael Paterson V00214440

More information

Infrastructure as a Service

Infrastructure as a Service Infrastructure as a Service Jose Castro Leon CERN IT/OIS Cloud Computing On-Demand Self-Service Scalability and Efficiency Resource Pooling Rapid elasticity 2 Infrastructure as a Service Objectives 90%

More information

Long term analysis in HEP: Use of virtualization and emulation techniques

Long term analysis in HEP: Use of virtualization and emulation techniques Long term analysis in HEP: Use of virtualization and emulation techniques Yves Kemp DESY IT First Workshop on Data Preservation and Long Term Analysis in HEP, DESY 26.1.2009 Outline Why virtualization

More information

GRID workload management system and CMS fall production. Massimo Sgaravatto INFN Padova

GRID workload management system and CMS fall production. Massimo Sgaravatto INFN Padova GRID workload management system and CMS fall production Massimo Sgaravatto INFN Padova What do we want to implement (simplified design) Master chooses in which resources the jobs must be submitted Condor-G

More information

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida Plug-and-play Virtual Appliance Clusters Running Hadoop Dr. Renato Figueiredo ACIS Lab - University of Florida Advanced Computing and Information Systems laboratory Introduction You have so far learned

More information

CUMULUX WHICH CLOUD PLATFORM IS RIGHT FOR YOU? COMPARING CLOUD PLATFORMS. Review Business and Technology Series www.cumulux.com

CUMULUX WHICH CLOUD PLATFORM IS RIGHT FOR YOU? COMPARING CLOUD PLATFORMS. Review Business and Technology Series www.cumulux.com ` CUMULUX WHICH CLOUD PLATFORM IS RIGHT FOR YOU? COMPARING CLOUD PLATFORMS Review Business and Technology Series www.cumulux.com Table of Contents Cloud Computing Model...2 Impact on IT Management and

More information

glideinwms monitoring from a VO Frontend point of view

glideinwms monitoring from a VO Frontend point of view VO Forum glideinwms monitoring from a VO Frontend point of view by Igor Sfiligoi VO Forum, 3/24/2011 Frontend monitoring 1 glideinwms architecture Central manager Submit node Schedd Collector Negotiator

More information

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH CONTENTS Introduction... 4 System Components... 4 OpenNebula Cloud Management Toolkit... 4 VMware

More information

Elastic Management of Cluster based Services in the Cloud

Elastic Management of Cluster based Services in the Cloud First Workshop on Automated Control for Datacenters and Clouds (ACDC09) June 19th, Barcelona, Spain Elastic Management of Cluster based Services in the Cloud Rafael Moreno Vozmediano, Ruben S. Montero,

More information

Computing at the HL-LHC

Computing at the HL-LHC Computing at the HL-LHC Predrag Buncic on behalf of the Trigger/DAQ/Offline/Computing Preparatory Group ALICE: Pierre Vande Vyvre, Thorsten Kollegger, Predrag Buncic; ATLAS: David Rousseau, Benedetto Gorini,

More information

Data Centers and Cloud Computing

Data Centers and Cloud Computing Data Centers and Cloud Computing CS377 Guest Lecture Tian Guo 1 Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing Case Study: Amazon EC2 2 Data Centers

More information

IaaS Cloud Architectures: Virtualized Data Centers to Federated Cloud Infrastructures

IaaS Cloud Architectures: Virtualized Data Centers to Federated Cloud Infrastructures IaaS Cloud Architectures: Virtualized Data Centers to Federated Cloud Infrastructures Dr. Sanjay P. Ahuja, Ph.D. 2010-14 FIS Distinguished Professor of Computer Science School of Computing, UNF Introduction

More information

Context-aware cloud computing for HEP

Context-aware cloud computing for HEP Department of Physics and Astronomy, University of Victoria, Victoria, British Columbia, Canada V8W 2Y2 E-mail: rsobie@uvic.ca The use of cloud computing is increasing in the field of high-energy physics

More information

Ezilla - WebOS Toward the Private Cloud & Possibility of Virtual Classroom

Ezilla - WebOS Toward the Private Cloud & Possibility of Virtual Classroom Ezilla - WebOS Toward the Private Cloud & Possibility of Virtual Classroom C.H. Wu, Y.L. Serena Pan, H.E. Max Yu, H.S. CHen, Weicheng Huang National Center for High-performance Computing, Taiwan 2012/04/18

More information

U-LITE: a proposal for scientific computing at LNGS. S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011

U-LITE: a proposal for scientific computing at LNGS. S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011 U-LITE: a proposal for scientific computing at LNGS S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011 20 years of Scientific Computing at LNGS Early 90s: highly centralized structure based on VMS cluster

More information

Potential of Virtualization Technology for Long-term Data Preservation

Potential of Virtualization Technology for Long-term Data Preservation Potential of Virtualization Technology for Long-term Data Preservation J Blomer on behalf of the CernVM Team jblomer@cern.ch CERN PH-SFT 1 / 12 Introduction Potential of Virtualization Technology Preserve

More information

Interoperating Cloud-based Virtual Farms

Interoperating Cloud-based Virtual Farms Stefano Bagnasco, Domenico Elia, Grazia Luparello, Stefano Piano, Sara Vallero, Massimo Venaruzzo For the STOA-LHC Project Interoperating Cloud-based Virtual Farms The STOA-LHC project 1 Improve the robustness

More information

WINDOWS AZURE EXECUTION MODELS

WINDOWS AZURE EXECUTION MODELS WINDOWS AZURE EXECUTION MODELS Windows Azure provides three different execution models for running applications: Virtual Machines, Web Sites, and Cloud Services. Each one provides a different set of services,

More information

The OpenNebula Cloud Platform for Data Center Virtualization

The OpenNebula Cloud Platform for Data Center Virtualization CloudOpen 2012 San Diego, USA, August 29th, 2012 The OpenNebula Cloud Platform for Data Center Virtualization Carlos Martín Project Engineer Acknowledgments The research leading to these results has received

More information

Cloud Computing PES. (and virtualization at CERN) Cloud Computing. GridKa School 2011, Karlsruhe. Disclaimer: largely personal view of things

Cloud Computing PES. (and virtualization at CERN) Cloud Computing. GridKa School 2011, Karlsruhe. Disclaimer: largely personal view of things PES Cloud Computing Cloud Computing (and virtualization at CERN) Ulrich Schwickerath et al With special thanks to the many contributors to this presentation! GridKa School 2011, Karlsruhe CERN IT Department

More information

GESTIONE DI APPLICAZIONI DI CALCOLO ETEROGENEE CON STRUMENTI DI CLOUD COMPUTING: ESPERIENZA DI INFN-TORINO

GESTIONE DI APPLICAZIONI DI CALCOLO ETEROGENEE CON STRUMENTI DI CLOUD COMPUTING: ESPERIENZA DI INFN-TORINO GESTIONE DI APPLICAZIONI DI CALCOLO ETEROGENEE CON STRUMENTI DI CLOUD COMPUTING: ESPERIENZA DI INFN-TORINO S.Bagnasco, D.Berzano, R.Brunetti, M.Concas, S.Lusso 2 Motivations During the last years, the

More information

SUSE Cloud 2.0. Pete Chadwick. Douglas Jarvis. Senior Product Manager pchadwick@suse.com. Product Marketing Manager djarvis@suse.

SUSE Cloud 2.0. Pete Chadwick. Douglas Jarvis. Senior Product Manager pchadwick@suse.com. Product Marketing Manager djarvis@suse. SUSE Cloud 2.0 Pete Chadwick Douglas Jarvis Senior Product Manager pchadwick@suse.com Product Marketing Manager djarvis@suse.com SUSE Cloud SUSE Cloud is an open source software solution based on OpenStack

More information

Virtual Data Centre Public Cloud Simplicity Private Cloud Security

Virtual Data Centre Public Cloud Simplicity Private Cloud Security Virtual Data Centre Public Cloud Simplicity Private Cloud Security www.interoute.com Interoute Virtual Data Centre Virtual Data Centre (VDC) is Interoute s Enterprise class Infrastructure as a Service

More information

The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)"

The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud) The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)" J.A. Coarasa " CERN, Geneva, Switzerland" for the CMS TriDAS group." " CHEP2013, 14-18 October 2013, Amsterdam, The Netherlands

More information

Florida Site Report. US CMS Tier-2 Facilities Workshop. April 7, 2014. Bockjoo Kim University of Florida

Florida Site Report. US CMS Tier-2 Facilities Workshop. April 7, 2014. Bockjoo Kim University of Florida Florida Site Report US CMS Tier-2 Facilities Workshop April 7, 2014 Bockjoo Kim University of Florida Outline Site Overview Computing Resources Site Status Future Plans Summary 2 Florida Tier-2 Paul Avery

More information

HTCondor within the European Grid & in the Cloud

HTCondor within the European Grid & in the Cloud HTCondor within the European Grid & in the Cloud Andrew Lahiff STFC Rutherford Appleton Laboratory HEPiX 2015 Spring Workshop, Oxford The Grid Introduction Computing element requirements Job submission

More information

HEP Data-Intensive Distributed Cloud Computing System Requirements Specification Document

HEP Data-Intensive Distributed Cloud Computing System Requirements Specification Document HEP Data-Intensive Distributed Cloud Computing System Requirements Specification Document CANARIE NEP-101 Project University of Victoria HEP Computing Group December 18, 2013 Version 1.0 1 Revision History

More information

Measurement of BeStMan Scalability

Measurement of BeStMan Scalability Measurement of BeStMan Scalability Haifeng Pi, Igor Sfiligoi, Frank Wuerthwein, Abhishek Rana University of California San Diego Tanya Levshina Fermi National Accelerator Laboratory Alexander Sim, Junmin

More information

Integration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept

Integration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept Integration of Virtualized Workernodes in Batch Queueing Systems, Dr. Armin Scheurer, Oliver Oberst, Prof. Günter Quast INSTITUT FÜR EXPERIMENTELLE KERNPHYSIK FAKULTÄT FÜR PHYSIK KIT University of the

More information

Cloud infrastructure for the on demand provisioning of Worker Nodes

Cloud infrastructure for the on demand provisioning of Worker Nodes Cloud infrastructure for the on demand provisioning of Worker Nodes A. Andronidis, P. Korosoglou, G. Fergadis and P. Argyrakis Outline Observation Idea Implementation 2 Our observations (last Spring) Scientific

More information

Understanding ArcGIS in Virtualization and Cloud Environments

Understanding ArcGIS in Virtualization and Cloud Environments Esri Middle East and Africa User Conference December 10 12 Abu Dhabi, UAE Understanding ArcGIS in Virtualization and Cloud Environments Marwa Mabrouk Powerful GIS capabilities Delivered as Web services

More information

Simulation and user analysis of BaBar data in a distributed cloud

Simulation and user analysis of BaBar data in a distributed cloud Simulation and user analysis of BaBar data in a distributed cloud A. Agarwal, University of Victoria M. Anderson, University of Victoria P. Armstrong, University of Victoria A. Charbonneau, National Research

More information

Provisioning and Resource Management at Large Scale (Kadeploy and OAR)

Provisioning and Resource Management at Large Scale (Kadeploy and OAR) Provisioning and Resource Management at Large Scale (Kadeploy and OAR) Olivier Richard Laboratoire d Informatique de Grenoble (LIG) Projet INRIA Mescal 31 octobre 2007 Olivier Richard ( Laboratoire d Informatique

More information

Chapter 10 Transparency

Chapter 10 Transparency Chapter 10 1 2 Statement Complexity Distributed systems consist of many interacting components. Given the connectivity and even the existence of many components may vary during operation. The system is

More information

High Throughput WAN Data Transfer with Hadoop-based Storage

High Throughput WAN Data Transfer with Hadoop-based Storage High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wüerthwein 1 1 University of California, San

More information

Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst

Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst Outline General Description of Virtualization / Virtualization Solutions Shared HPC Infrastructure Virtualization

More information

Disk images for virtual machines

Disk images for virtual machines Disk images for virtual machines November 12, 2013 Disk images for virtual machines November 12, 2013 1 / 14 Virtual machines have disks The host (on which the virtual machines are running) needs to provide

More information

OpenStack Introduction. November 4, 2015

OpenStack Introduction. November 4, 2015 OpenStack Introduction November 4, 2015 Application Platforms Undergoing A Major Shift What is OpenStack Open Source Cloud Software Launched by NASA and Rackspace in 2010 Massively scalable Managed by

More information

glideinwms Training HTCondor scalability testing hints by Igor Sfiligoi, UC San Diego Aug 2014 Glidein scalability hints 1

glideinwms Training HTCondor scalability testing hints by Igor Sfiligoi, UC San Diego Aug 2014 Glidein scalability hints 1 glideinwms Training HTCondor scalability testing hints by Igor Sfiligoi, UC San Diego Aug 2014 Glidein scalability hints 1 Overview These slides provide a few hints on how to do a HTCondor scalability

More information

Using ssh as portal CHEP The CMS CRAB over glideinwms experience

Using ssh as portal CHEP The CMS CRAB over glideinwms experience CHEP 2013 Using ssh as portal The CMS CRAB over glideinwms experience by I Sfiligoi 1, S Belforte 2, J Letts 1, T Martin 1, M D Saiz Santos 1 and F Fanzago 3 1 University of California San Diego 2 Università

More information

In Cloud, Do MTC or HTC Service Providers Benefit from the Economies of Scale?

In Cloud, Do MTC or HTC Service Providers Benefit from the Economies of Scale? In Cloud, Do MTC or HTC Service Providers Benefit from the Economies of Scale? Lei Wang, Jianfeng Zhan, Weisong Shi, Yi Liang, Lin Yuan Institute of Computing Technology, Chinese Academy of Sciences Department

More information

Cloud and Virtualization to Support Grid Infrastructures

Cloud and Virtualization to Support Grid Infrastructures ESAC GRID Workshop '08 ESAC, Villafranca del Castillo, Spain 11-12 December 2008 Cloud and Virtualization to Support Grid Infrastructures Distributed Systems Architecture Research Group Universidad Complutense

More information

Auto-Scaling Model for Cloud Computing System

Auto-Scaling Model for Cloud Computing System Auto-Scaling Model for Cloud Computing System Che-Lun Hung 1*, Yu-Chen Hu 2 and Kuan-Ching Li 3 1 Dept. of Computer Science & Communication Engineering, Providence University 2 Dept. of Computer Science

More information

DSS. Data & Storage Services. Huawei Cloud Storage Evaluation and Testing with Prototype Services. Seppo S. Heikkila CERN IT

DSS. Data & Storage Services. Huawei Cloud Storage Evaluation and Testing with Prototype Services. Seppo S. Heikkila CERN IT Data & Storage Huawei Cloud Storage Evaluation and Testing with Prototype Seppo S. Heikkila CERN IT Openlab Minor Review 29th of October 2013 CERN, Geneva Introduction Motivation Cloud storage market is

More information

OpenNebula. Sándor Ács

OpenNebula. Sándor Ács OpenNebula Sándor Ács acs.sandor@sztaki.mta.hu http://www.lpds.sztaki.hu/cloudresearch This presentation is heavily based on multiple presentations of the following people: Gábor Kecskeméti, Ignacio M.

More information

New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud

New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud CISCO NerdLunch Series November 7, 2008 San Jose, CA New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud Ruben Santiago Montero Distributed Systems Architecture Research

More information

Science Days 2010. Stefan Freitag. 03. November 2010. Robotics Research Institute Dortmund University of Technology. Cloud Computing in D-Grid

Science Days 2010. Stefan Freitag. 03. November 2010. Robotics Research Institute Dortmund University of Technology. Cloud Computing in D-Grid in at and in Science Days 2010 Stefan Freitag Robotics Research Institute Dortmund University of Technology 03. November 2010 Collaboration of Mad Rocket Scientists in Site B at Site A and Site D Site

More information

Incident Response & Forensics In The Cloud 2013 SANS

Incident Response & Forensics In The Cloud 2013 SANS MCP+I, MCSE, CCSA, CCSE, CISSP-ISSAP, CISM, CISA, CIFI, CCE, ACE, GCFE, GCFA, GSEC, VCP4/5, vexpert Senior SANS Instructor - phenry@sans.org 1 A Lot To Cover In ½ An Hour We simply can not cover all cloud

More information

Cloud Accounting. Laurence Field IT/SDC 22/05/2014

Cloud Accounting. Laurence Field IT/SDC 22/05/2014 Cloud Accounting Laurence Field IT/SDC 22/05/2014 Helix Nebula Pathfinder project Development and exploitation Cloud Computing Infrastructure Divided into supply and demand Three flagship applications

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details Use cases By scientist By HEP experiment System requirements and installation How to get it? 2 What

More information

Cloud computing is a marketing term that means different things to different people. In this presentation, we look at the pros and cons of using

Cloud computing is a marketing term that means different things to different people. In this presentation, we look at the pros and cons of using Cloud computing is a marketing term that means different things to different people. In this presentation, we look at the pros and cons of using Amazon Web Services rather than setting up a physical server

More information

Scaling Analysis Services in the Cloud

Scaling Analysis Services in the Cloud Our Sponsors Scaling Analysis Services in the Cloud by Gerhard Brückl gerhard@gbrueckl.at blog.gbrueckl.at About me Gerhard Brückl Working with Microsoft BI since 2006 Windows Azure / Cloud since 2013

More information

Managing a tier-2 computer centre with a private cloud infrastructure

Managing a tier-2 computer centre with a private cloud infrastructure Managing a tier-2 computer centre with a private cloud infrastructure Stefano Bagnasco 1, Dario Berzano 1,2,3, Riccardo Brunetti 1,4, Stefano Lusso 1 and Sara Vallero 1,2 1 Istituto Nazionale di Fisica

More information

Virtualization Infrastructure at Karlsruhe

Virtualization Infrastructure at Karlsruhe Virtualization Infrastructure at Karlsruhe HEPiX Fall 2007 Volker Buege 1),2), Ariel Garcia 1), Marcus Hardt 1), Fabian Kulla 1),Marcel Kunze 1), Oliver Oberst 1),2), Günter Quast 2), Christophe Saout

More information

DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2

DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2 DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing Slide 1 Slide 3 A style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet.

More information

BMC Control-M for Cloud. BMC Control-M Workload Automation

BMC Control-M for Cloud. BMC Control-M Workload Automation BMC Control-M for Cloud BMC Control-M Workload Automation Virtualization & Cloud Computing Are Top Priorities Cloud Computing is a catalyst for improving IT maturity and moving virtualization to another

More information

Dennis Waldron, CERN IT/DM/DA CASTOR Face-to-Face Meeting, Feb 19 th 2009. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.

Dennis Waldron, CERN IT/DM/DA CASTOR Face-to-Face Meeting, Feb 19 th 2009. CERN IT Department CH-1211 Genève 23 Switzerland www.cern. Load Testing Dennis Waldron, CERN IT/DM/DA CASTOR Face-to-Face Meeting, Feb 19 th 2009 Outline Certification (functionality) setup Stress testing ti setup Current stress tests Certification Setup (CERT2)

More information

Big Data and Cloud Computing for GHRSST

Big Data and Cloud Computing for GHRSST Big Data and Cloud Computing for GHRSST Jean-Francois Piollé (jfpiolle@ifremer.fr) Frédéric Paul, Olivier Archer CERSAT / Institut Français de Recherche pour l Exploitation de la Mer Facing data deluge

More information

Managing a Tier-2 Computer Centre with a Private Cloud Infrastructure

Managing a Tier-2 Computer Centre with a Private Cloud Infrastructure Managing a Tier-2 Computer Centre with a Private Cloud Infrastructure Stefano Bagnasco, Riccardo Brunetti, Stefano Lusso (INFN-Torino), Dario Berzano (CERN) ACAT2013 Beijing, May 16-21, 2013 motivation

More information

Cloud Server. Parallels. Key Features and Benefits. White Paper. www.parallels.com

Cloud Server. Parallels. Key Features and Benefits. White Paper. www.parallels.com Parallels Cloud Server White Paper Key Features and Benefits www.parallels.com Table of Contents Introduction... 3 Key Features... 3 Distributed Cloud Storage (Containers and Hypervisors)... 3 Rebootless

More information

Ad hoc Cloud Computing

Ad hoc Cloud Computing Ad hoc Cloud Computing Gary A. McGilvary, Adam Barker, Malcolm Atkinson Edinburgh Data-Intensive Research Group, School of Informatics, The University of Edinburgh Email: gary.mcgilvary@ed.ac.uk, mpa@staffmail.ed.ac.uk

More information

Cloud services for the Fermilab scientific stakeholders

Cloud services for the Fermilab scientific stakeholders Cloud services for the Fermilab scientific stakeholders S Timm 1, G Garzoglio 1, P Mhashilkar 1*, J Boyd 1, G Bernabeu 1, N Sharma 1, N Peregonow 1, H Kim 1, S Noh 2,, S Palur 3, and I Raicu 3 1 Scientific

More information

An Experimental Study of Load Balancing of OpenNebula Open-Source Cloud Computing Platform

An Experimental Study of Load Balancing of OpenNebula Open-Source Cloud Computing Platform An Experimental Study of Load Balancing of OpenNebula Open-Source Cloud Computing Platform A B M Moniruzzaman 1, Kawser Wazed Nafi 2, Prof. Syed Akhter Hossain 1 and Prof. M. M. A. Hashem 1 Department

More information

Implementing and using IaaS cloud within the Flexible Services for the Support of Research project

Implementing and using IaaS cloud within the Flexible Services for the Support of Research project Implementing and using IaaS cloud within the Flexible Services for the Support of Research project Dr David Wallom, Associate Director - Innovation (Oxford e-research Centre) Technical Director (UK NGS)

More information

UZH Experiences with OpenStack

UZH Experiences with OpenStack GC3: Grid Computing Competence Center UZH Experiences with OpenStack What we did, what went well, what went wrong. Antonio Messina 29 April 2013 Setting up Hardware configuration

More information

An Introduction to Virtualization and Cloud Technologies to Support Grid Computing

An Introduction to Virtualization and Cloud Technologies to Support Grid Computing New Paradigms: Clouds, Virtualization and Co. EGEE08, Istanbul, September 25, 2008 An Introduction to Virtualization and Cloud Technologies to Support Grid Computing Distributed Systems Architecture Research

More information

Client/Server Grid applications to manage complex workflows

Client/Server Grid applications to manage complex workflows Client/Server Grid applications to manage complex workflows Filippo Spiga* on behalf of CRAB development team * INFN Milano Bicocca (IT) Outline Science Gateways and Client/Server computing Client/server

More information

A Gentle Introduction to Cloud Computing

A Gentle Introduction to Cloud Computing A Gentle Introduction to Cloud Computing Source: Wikipedia Platform Computing, Inc. Platform Clusters, Grids, Clouds, Whatever Computing The leader in managing large scale shared environments o 18 years

More information

Evolution of the ATLAS PanDA Production and Distributed Analysis System

Evolution of the ATLAS PanDA Production and Distributed Analysis System Evolution of the ATLAS PanDA Production and Distributed Analysis System T. Maeno 1, K. De 2, T. Wenaus 1, P. Nilsson 2, R. Walker 3, A. Stradling 2, V. Fine 1, M. Potekhin 1, S. Panitkin 1, G. Compostella

More information

Denis Caromel, CEO Ac.veEon. Orchestrate and Accelerate Applica.ons. Open Source Cloud Solu.ons Hybrid Cloud: Private with Burst Capacity

Denis Caromel, CEO Ac.veEon. Orchestrate and Accelerate Applica.ons. Open Source Cloud Solu.ons Hybrid Cloud: Private with Burst Capacity Cloud computing et Virtualisation : applications au domaine de la Finance Denis Caromel, CEO Ac.veEon Orchestrate and Accelerate Applica.ons Open Source Cloud Solu.ons Hybrid Cloud: Private with Burst

More information

Getting Started Hacking on OpenNebula

Getting Started Hacking on OpenNebula LinuxTag 2013 Berlin, Germany, May 22nd Getting Started Hacking on OpenNebula Carlos Martín Project Engineer Acknowledgments The research leading to these results has received funding from Comunidad de

More information

Experiences with the GLUE information schema in the LCG/EGEE production Grid

Experiences with the GLUE information schema in the LCG/EGEE production Grid Experiences with the GLUE information schema in the LCG/EGEE production Grid Stephen Burke, Sergio Andreozzi and Laurence Field CHEP07, Victoria, Canada www.eu-egee.org EGEE and glite are registered trademarks

More information