Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb

Size: px
Start display at page:

Download "Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb"

Transcription

1 Batch and Cloud overview Andrew McNab University of Manchester GridPP and LHCb

2 Overview Assumptions Batch systems The Grid Pilot Frameworks DIRAC Virtual Machines Vac Vcycle Tier-2 Evolution Containers Event servers 2

3 Basic assumptions You have a series of pieces of computation that need doing From more than one person in your collaboration? With different priorities? With different requirements? You want to execute these pieces of computation on remote resources Remote not the machine you edit files on Accessing remote resources means you can use what is available May allow you to scale your computation up by orders of magnitude These pieces of computation need to read or write data Where they run? Somewhere remote from that too? Piece of computation sounds like a job, but not necessarily 3

4 Batch systems Traditionally all this has been done with batch queuing systems Long and noble history going back to first mainframes, when operators physically loaded people s stacks of punched cards into readers Many systems descended from or influenced by NASA NQS have the command qsub to submit a file that defines the job to be run. Simplest way of running remote batch jobs is to do something like: scp job.sh ; ssh qsub job.sh In the 1990s CERN built a system called SHIFT that did just that If you have access to one big system somewhere else, you may be doing that yourself today However, it s very very unscalable, as it relies on setting up ssh access knowing that somewhere.ac.uk has resources you can run on 4

5 The Grid (Here The Grid is the infrastructure of European Grid Infrastructure (EGI) and Worldwide LHC Computing Grid (WLCG)) The Grid gives access to remote batch systems in a much more scalable fashion Uses X.509 and VOMS certificates for identifying users and services and for virtual organization (VO) membership An information system to discover suitable resources Lots of operational support (tickets, mailing lists, monitoring) Makes it possible to automate the submission of jobs Originally via a Broker or Workload Management System Now via the client submitting the jobs directly (That sounds like a backward step but it s not) 5

6 The Grid CREAM or ARC CE & batch queues Underlies the infrastructure run by EGI/WLCG Job Grid Site Originally proposed by EU DataGrid in 2000 WMS Broker (deprecated) Central agents & services Job factory User and production jobs 6

7 The Grid with Pilot Jobs The Grid + pilot jobs is the dominant model for running HEP jobs. Well established, and gives access to resources around the world. CREAM or ARC CE & batch queues Pilot Job. Runs Job Agent to fetch from TQ Pilot jobs Grid Site This late binding model allows the virtual organization to decide priorities itself. It sidelines the batch system s scheduling of time slots: each type of pilot job are all the same irrespective of the contents. Requests for real jobs Central agents & services Director (pilot factory) Matcher & Task Queue User and production jobs 7

8 Pilot frameworks Pilot job concept was introduced by LHCb s DIRAC project in 2003 DIRAC now spun-off as a separate project and used by other experiments, and provided by infrastructures like EGI and GridPP GridPP DIRAC service is intended to support experiments and VOs without their own pilot framework ATLAS and CMS also use pilot frameworks and have done some work on using ATLAS s BigPanda Pilot framework becomes a higher level batch system Can be used to hide the details of the underlying batch system, or even whether jobs are running inside VMs at cloud sites The virtual organization can monitor and manage all of the resources available to it centrally 8

9 Virtual grid sites Cloud systems like OpenStack allow virtualization of fabric What s new is that they expose (defacto?) standard APIs for the creation and management of virtual machines by external users Bringing up a machine can be delegated towards user communities Rather than using PXE+Kickstart+Puppet on bare metal Zeroth order virtualization is just to provide a Grid with Pilot Jobs site, on VMs Potential to use staff more efficiently: one big pool of hardware and hypervisors managed all together rather than separate clusters However, can take this a step further and have the experiment managing the cloud resources as a virtual grid site 9

10 Virtual Grid with Pilot Jobs site Gatekeeper (ARC? Condor?) Cloud Site Batch VM. Runs Job Agent to fetch from TQ Experiment creates its own conventional grid site on the cloud resources. Transparent to existing central services, and user/production job submitters. (Or via VM factory) CloudScheduler (ATLAS) and elastiq (ALICE) are implementations of this model. Requests for real jobs Central agents & services VM factory Pilot factory Matcher & Task Queue User and production jobs 10

11 Experiment creates VMs directly? Cloud Site VM. Runs Job Agent to fetch from TQ Experiment creates VMs instead of pilot jobs. Job Agent or pilot client runs as normal inside. CMS glidein WMS works this way for cloud sites: looks at demand and creates VMs to join Condor pool Requests for real jobs Central agents & services VM factory Matcher & Task Queue User and production jobs 11

12 Further simplification: Vacuum model Following the CHEP 2013 paper: The Vacuum model can be defined as a scenario in which virtual machines are created and contextualized for experiments by the resource provider itself. The contextualization procedures are supplied in advance by the experiments and launch clients within the virtual machines to obtain work from the experiments' central queue of tasks. ( Running jobs in the vacuum, A McNab et al 2014 J. Phys.: Conf. Ser ) Loosely coupled, late binding approach in the spirit of pilot frameworks For the experiments, VMs appear by spontaneous production in the vacuum Like virtual particles in the physical vacuum: they appear, potentially interact, and then disappear Three implementations so far: Vac, Vcycle, HTCondor Vacuum 12

13 Vac - the first Vacuum system Pilot VM. Runs Job Agent to fetch from TQ Vacuum site Since we have the pilot framework, we can do something really simple Strip the system right down and have each physical host at the site create the VMs itself. Requests for real jobs Instead of being created by the experiments, the virtual machines appear spontaneously out of the vacuum at sites. Use same VMs as on Cloud sites Central agents & services Matcher & Task Queue User and production jobs 13

14 Vcycle Site Vcycle VM factory Cloud Site Pilot VM. Runs Job Agent to fetch from TQ Apply Vac ideas to Clouds Vcycle implements an external VM factory that manages VMs. Can be run centrally by experiment or by site itself or by a third party Third party Vcycle VM factory Requests for real jobs VMs started and monitored by Vcycle, but not managed in detail ( black boxes ) Central agents & services Vcycle VM factory No direct communication between Vcycle and task queue Matcher & Task Queue User and production jobs 14

15 Pilot VMs Vac, Vcycle, HTCondor Vacuum assume the VMs have a defined lifecycle Need a boot image and user_data file with contextualisation Provided by experiment centrally from an HTTPS web server Virtual disks and boot media defined and VM started The VM runs and its state is monitored VM executes shutdown -h when finished or if no more work available Maybe also update a heartbeat file and so stalled or overruning VMs are killed shutdown_message file can be used to say why the VM shut down Experiments VMs are a lot simpler for the site to handle than running worker nodes LHC experiments and GridPP use CernVM-FS wide-area filesystem to provide VMs with operating system files and experiment software 15

16 Some implications of virtualization Decouples operating system versioning requirements of experiments and sites Jobs see an identical environment across sites Simplifies many aspects of security model (escalation much harder) Many opportunities for other simplifications (eg Vacuum) For the sites, VMs make WLCG systems seem a lot more normal Especially if unusual middleware is only inside the VMs When the Grid started, several sites had problems (or in fact failed) trying to run shared clusters with local IT services eg shared PBS cluster for HEP and other sciences, with same WN configuration for everyone Virtualization gives us a second chance to try this where appropriate 16

17 GridPP s Tier-2 Evolution project In the plan for GridPP5, we proposed radically simplifying what s needed to run a university grid site (a Tier-2 site ) using VMs We re working through all the requirements to be able to run purely VM-based sites Getting VMs signed off by experiments for all Tier-2 workloads Making sure monitoring, accounting, ticketing etc all support VM-based sites and the software we re using to manage them (Also trying to simplify the storage side) This is partly in response to reduced staff funding But it also dramatically lowers the effort required to create and run a Tier-2 style site to run jobs in GridPP DIRAC VMs and the other VM flavours we support 17

18 Containers Linux containers can provide an alternative (often a drop-in alternative) to most of the places I ve talked about VMs Can use containers to provide a protected virtual environment of libraries, files etc to payload jobs Container as VM So we should talk about Logical Machines to be general Other more lightweight models are possible, more focussed on the environment seen by particular processes or sets of processes CernVM base-image we re using it being adapted to work as a Docker container Again using CernVM-FS to get OS and experiment files Support for this will be added to Vac 18

19 Micro jobs and event servers So far we ve talked about jobs lasting hours or days Push model of early Grid became the pull model of Pilot Jobs But much finer grained pull models also exist Notably the ATLAS experiment s Event Service Still have something that looks like a job that runs for hours or days But now takes smaller packages of work from a central service Every few minutes? Makes it much easier to solve the Masonry Problem of packing work into the available time and space provided by a fixed lifetime job But the cost is potentially a lot more network connections and opportunities bottlenecks 19

20 Summary Broad trends towards automation and decoupling jobs from details of the sites they run at Automation gives opportunities for accessing a lot more resources As easy to run at 300 sites once you can automatically pick 1 of 3 But running at 300 sites is more complicated when things go wrong Pilot job frameworks abstract the specifics of how the job got there GridPP DIRAC service intended for experiments and VOs without their own pilot framework A lot of activity in using virtualized and now containerized resources GridPP aims to enable sites to run purely using VMs if they choose Event servers and micro jobs offer new opportunities to make more efficient use of job slots 20

21 Extra slides 21

22 The Masonry Problem... 22

23 The Masonry Problem Maximum length job, starting just before advance notice given Max length Max length job Wasted resources Max length job Short job Short job Max length Shorter job Max length job Short job Start of advance notice Hard deadline for jobs to finish 23

24 OpenStack cloud site architecture 24

Running real jobs in virtual machines. Andrew McNab University of Manchester

Running real jobs in virtual machines. Andrew McNab University of Manchester Running real jobs in virtual machines Andrew McNab University of Manchester Overview The Grid we have now Strategies for starting VMs Fabric / Cloud / Vac Hypervisors Disk images Contextualization Pilot

More information

CHEP 2013. Cloud Bursting with glideinwms Means to satisfy ever increasing computing needs for Scientific Workflows

CHEP 2013. Cloud Bursting with glideinwms Means to satisfy ever increasing computing needs for Scientific Workflows CHEP 2013 Cloud Bursting with glideinwms Means to satisfy ever increasing computing needs for Scientific Workflows by I. Sfiligoi 1, P. Mhashilkar 2, A. Tiradani 2, B. Holzman 2, K. Larson 2 and M. Rynge

More information

HTCondor within the European Grid & in the Cloud

HTCondor within the European Grid & in the Cloud HTCondor within the European Grid & in the Cloud Andrew Lahiff STFC Rutherford Appleton Laboratory HEPiX 2015 Spring Workshop, Oxford The Grid Introduction Computing element requirements Job submission

More information

Status and Evolution of ATLAS Workload Management System PanDA

Status and Evolution of ATLAS Workload Management System PanDA Status and Evolution of ATLAS Workload Management System PanDA Univ. of Texas at Arlington GRID 2012, Dubna Outline Overview PanDA design PanDA performance Recent Improvements Future Plans Why PanDA The

More information

CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment

CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment George Lestaris - Ioannis Charalampidis D. Berzano, J. Blomer, P. Buncic, G. Ganis and R. Meusel PH-SFT /

More information

WM Technical Evolution Group Report

WM Technical Evolution Group Report WM Technical Evolution Group Report Davide Salomoni, INFN for the WM TEG February 7, 2012 The WM TEG, organization Mailing list, wlcg-teg-workload-mgmt@cern.ch 48 people currently subscribed; representation

More information

DYNAMIC ALLOCATION AND CLOUD COMPUTING IN CMS

DYNAMIC ALLOCATION AND CLOUD COMPUTING IN CMS DYNAMIC ALLOCATION AND CLOUD COMPUTING IN CMS Massimo Sgaravatto INFN Padova Claudio Grandi INFN Bologna CMS VIEW ON CLOUDS From January GDB meeting CMS is reasonably happy with the current resource allocation

More information

Context-aware cloud computing for HEP

Context-aware cloud computing for HEP Department of Physics and Astronomy, University of Victoria, Victoria, British Columbia, Canada V8W 2Y2 E-mail: rsobie@uvic.ca The use of cloud computing is increasing in the field of high-energy physics

More information

Cloud Accounting. Laurence Field IT/SDC 22/05/2014

Cloud Accounting. Laurence Field IT/SDC 22/05/2014 Cloud Accounting Laurence Field IT/SDC 22/05/2014 Helix Nebula Pathfinder project Development and exploitation Cloud Computing Infrastructure Divided into supply and demand Three flagship applications

More information

Integration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept

Integration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept Integration of Virtualized Workernodes in Batch Queueing Systems, Dr. Armin Scheurer, Oliver Oberst, Prof. Günter Quast INSTITUT FÜR EXPERIMENTELLE KERNPHYSIK FAKULTÄT FÜR PHYSIK KIT University of the

More information

The Virtualization Practice

The Virtualization Practice The Virtualization Practice White Paper: Managing Applications in Docker Containers Bernd Harzog Analyst Virtualization and Cloud Performance Management October 2014 Abstract Docker has captured the attention

More information

HTCondor at the RAL Tier-1

HTCondor at the RAL Tier-1 HTCondor at the RAL Tier-1 Andrew Lahiff, Alastair Dewhurst, John Kelly, Ian Collier, James Adams STFC Rutherford Appleton Laboratory HTCondor Week 2014 Outline Overview of HTCondor at RAL Monitoring Multi-core

More information

Single Sign-In User Centered Computing for High Energy Physics

Single Sign-In User Centered Computing for High Energy Physics Single Sign-In User Centered Computing for High Energy Physics Max Fischer, Oliver Oberst, Günter Quast, Marian Zvada ISGC 2013: March 19-22, 2013 Taipei, Taiwan INSTITUT FÜR EXPERIMENTELLE KERNPHYSIK

More information

Dynamic Resource Provisioning with HTCondor in the Cloud

Dynamic Resource Provisioning with HTCondor in the Cloud Dynamic Resource Provisioning with HTCondor in the Cloud Ryan Taylor Frank Berghaus 1 Overview Review of Condor + Cloud Scheduler system Condor job slot configuration Dynamic slot creation Automatic slot

More information

PoS(EGICF12-EMITC2)005

PoS(EGICF12-EMITC2)005 , Or: How One HEP Experiment Is Evaluating Strategies to Incorporate The Cloud into the Existing Grid Infrastructure Daniel Colin van der Ster 1 E-mail: daniel.colin.vanderster@cern.ch Fernando Harald

More information

The Definitive Guide To Docker Containers

The Definitive Guide To Docker Containers The Definitive Guide To Docker Containers EXECUTIVE SUMMARY THE DEFINITIVE GUIDE TO DOCKER CONTAINERS Executive Summary We are in a new technology age software is dramatically changing. The era of off

More information

A Web-based Portal to Access and Manage WNoDeS Virtualized Cloud Resources

A Web-based Portal to Access and Manage WNoDeS Virtualized Cloud Resources A Web-based Portal to Access and Manage WNoDeS Virtualized Cloud Resources Davide Salomoni 1, Daniele Andreotti 1, Luca Cestari 2, Guido Potena 2, Peter Solagna 3 1 INFN-CNAF, Bologna, Italy 2 University

More information

Clusters in the Cloud

Clusters in the Cloud Clusters in the Cloud Dr. Paul Coddington, Deputy Director Dr. Shunde Zhang, Compu:ng Specialist eresearch SA October 2014 Use Cases Make the cloud easier to use for compute jobs Par:cularly for users

More information

Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012

Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Thomas Hauth,, Günter Quast IEKP KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz

More information

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases NASA Ames NASA Advanced Supercomputing (NAS) Division California, May 24th, 2012 Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases Ignacio M. Llorente Project Director OpenNebula Project.

More information

Monitoring the Grid at local, national, and global levels

Monitoring the Grid at local, national, and global levels Home Search Collections Journals About Contact us My IOPscience Monitoring the Grid at local, national, and global levels This content has been downloaded from IOPscience. Please scroll down to see the

More information

Ryu SDN Framework What weʼ ve learned Where weʼ ll go

Ryu SDN Framework What weʼ ve learned Where weʼ ll go Ryu SDN Framework What weʼ ve learned Where weʼ ll go FUJITA Tomonori NTT Software Innovation Center Ryu Project lead 2014.11.14 NTT Ryu team goal Change the networking industry by Open Source Software

More information

PoS(EGICF12-EMITC2)110

PoS(EGICF12-EMITC2)110 User-centric monitoring of the analysis and production activities within the ATLAS and CMS Virtual Organisations using the Experiment Dashboard system Julia Andreeva E-mail: Julia.Andreeva@cern.ch Mattia

More information

ATLAS job monitoring in the Dashboard Framework

ATLAS job monitoring in the Dashboard Framework ATLAS job monitoring in the Dashboard Framework J Andreeva 1, S Campana 1, E Karavakis 1, L Kokoszkiewicz 1, P Saiz 1, L Sargsyan 2, J Schovancova 3, D Tuckett 1 on behalf of the ATLAS Collaboration 1

More information

Work Environment. David Tur HPC Expert. HPC Users Training September, 18th 2015

Work Environment. David Tur HPC Expert. HPC Users Training September, 18th 2015 Work Environment David Tur HPC Expert HPC Users Training September, 18th 2015 1. Atlas Cluster: Accessing and using resources 2. Software Overview 3. Job Scheduler 1. Accessing Resources DIPC technicians

More information

Building a Volunteer Cloud

Building a Volunteer Cloud Building a Volunteer Cloud Ben Segal, Predrag Buncic, David Garcia Quintas / CERN Daniel Lombrana Gonzalez / University of Extremadura Artem Harutyunyan / Yerevan Physics Institute Jarno Rantala / Tampere

More information

Managing a tier-2 computer centre with a private cloud infrastructure

Managing a tier-2 computer centre with a private cloud infrastructure Managing a tier-2 computer centre with a private cloud infrastructure Stefano Bagnasco 1, Dario Berzano 1,2,3, Riccardo Brunetti 1,4, Stefano Lusso 1 and Sara Vallero 1,2 1 Istituto Nazionale di Fisica

More information

Virtualisation Cloud Computing at the RAL Tier 1. Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013

Virtualisation Cloud Computing at the RAL Tier 1. Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013 Virtualisation Cloud Computing at the RAL Tier 1 Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013 Virtualisation @ RAL Context at RAL Hyper-V Services Platform Scientific Computing Department

More information

INTRODUCTION TO CLOUD MANAGEMENT

INTRODUCTION TO CLOUD MANAGEMENT CONFIGURING AND MANAGING A PRIVATE CLOUD WITH ORACLE ENTERPRISE MANAGER 12C Kai Yu, Dell Inc. INTRODUCTION TO CLOUD MANAGEMENT Oracle cloud supports several types of resource service models: Infrastructure

More information

Virtualization, Grid, Cloud: Integration Paths for Scientific Computing

Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Or, where and how will my efficient large-scale computing applications be executed? D. Salomoni, INFN Tier-1 Computing Manager Davide.Salomoni@cnaf.infn.it

More information

OASIS: a data and software distribution service for Open Science Grid

OASIS: a data and software distribution service for Open Science Grid OASIS: a data and software distribution service for Open Science Grid B. Bockelman 1, J. Caballero Bejar 2, J. De Stefano 2, J. Hover 2, R. Quick 3, S. Teige 3 1 University of Nebraska-Lincoln, Lincoln,

More information

International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan

International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud Ruben Santiago Montero Distributed

More information

Virtualization. (and cloud computing at CERN)

Virtualization. (and cloud computing at CERN) Virtualization (and cloud computing at CERN) Ulrich Schwickerath Special thanks: Sebastien Goasguen Belmiro Moreira, Ewan Roche, Romain Wartel See also related presentations: CloudViews2010 conference,

More information

CERN local High Availability solutions and experiences. Thorsten Kleinwort CERN IT/FIO WLCG Tier 2 workshop CERN 16.06.2006

CERN local High Availability solutions and experiences. Thorsten Kleinwort CERN IT/FIO WLCG Tier 2 workshop CERN 16.06.2006 CERN local High Availability solutions and experiences Thorsten Kleinwort CERN IT/FIO WLCG Tier 2 workshop CERN 16.06.2006 1 Introduction Different h/w used for GRID services Various techniques & First

More information

CMS Experience Provisioning Cloud Resources with GlideinWMS. Anthony Tiradani HTCondor Week 2015 20 May 2015

CMS Experience Provisioning Cloud Resources with GlideinWMS. Anthony Tiradani HTCondor Week 2015 20 May 2015 CMS Experience Provisioning Cloud Resources with GlideinWMS Anthony Tiradani Week 2015 20 May 2015 glideinwms Quick Facts glideinwms is an open- source Fermilab CompuJng Sector product driven by CMS Heavy

More information

HEP Data-Intensive Distributed Cloud Computing System Requirements Specification Document

HEP Data-Intensive Distributed Cloud Computing System Requirements Specification Document HEP Data-Intensive Distributed Cloud Computing System Requirements Specification Document CANARIE NEP-101 Project University of Victoria HEP Computing Group December 18, 2013 Version 1.0 1 Revision History

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details Use cases By scientist By HEP experiment System requirements and installation How to get it? 2 What

More information

Cloud Computing PES. (and virtualization at CERN) Cloud Computing. GridKa School 2011, Karlsruhe. Disclaimer: largely personal view of things

Cloud Computing PES. (and virtualization at CERN) Cloud Computing. GridKa School 2011, Karlsruhe. Disclaimer: largely personal view of things PES Cloud Computing Cloud Computing (and virtualization at CERN) Ulrich Schwickerath et al With special thanks to the many contributors to this presentation! GridKa School 2011, Karlsruhe CERN IT Department

More information

New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud

New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud CISCO NerdLunch Series November 7, 2008 San Jose, CA New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud Ruben Santiago Montero Distributed Systems Architecture Research

More information

Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst

Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst Outline General Description of Virtualization / Virtualization Solutions Shared HPC Infrastructure Virtualization

More information

Infrastructure as a Service

Infrastructure as a Service Infrastructure as a Service Jose Castro Leon CERN IT/OIS Cloud Computing On-Demand Self-Service Scalability and Efficiency Resource Pooling Rapid elasticity 2 Infrastructure as a Service Objectives 90%

More information

Cloud services for the Fermilab scientific stakeholders

Cloud services for the Fermilab scientific stakeholders Cloud services for the Fermilab scientific stakeholders S Timm 1, G Garzoglio 1, P Mhashilkar 1*, J Boyd 1, G Bernabeu 1, N Sharma 1, N Peregonow 1, H Kim 1, S Noh 2,, S Palur 3, and I Raicu 3 1 Scientific

More information

PanDA. A New Paradigm for Computing in HEP. Through the Lens of ATLAS and Other Experiments. Kaushik De

PanDA. A New Paradigm for Computing in HEP. Through the Lens of ATLAS and Other Experiments. Kaushik De PanDA A New Paradigm for Computing in HEP Through the Lens of ATLAS and Other Experiments Univ. of Texas at Arlington On behalf of the ATLAS Collaboration ICHEP 2014, Valencia Computing Challenges at the

More information

KVM, OpenStack, and the Open Cloud

KVM, OpenStack, and the Open Cloud KVM, OpenStack, and the Open Cloud Adam Jollans, IBM Southern California Linux Expo February 2015 1 Agenda A Brief History of VirtualizaJon KVM Architecture OpenStack Architecture KVM and OpenStack Case

More information

NT1: An example for future EISCAT_3D data centre and archiving?

NT1: An example for future EISCAT_3D data centre and archiving? March 10, 2015 1 NT1: An example for future EISCAT_3D data centre and archiving? John White NeIC xx March 10, 2015 2 Introduction High Energy Physics and Computing Worldwide LHC Computing Grid Nordic Tier

More information

Linux A first-class citizen in Windows Azure. Bruno Terkaly bterkaly@microsoft.com Principal Software Engineer Mobile/Cloud/Startup/Enterprise

Linux A first-class citizen in Windows Azure. Bruno Terkaly bterkaly@microsoft.com Principal Software Engineer Mobile/Cloud/Startup/Enterprise Linux A first-class citizen in Windows Azure Bruno Terkaly bterkaly@microsoft.com Principal Software Engineer Mobile/Cloud/Startup/Enterprise 1 First, I am software developer (C/C++, ASM, C#, Java, Node.js,

More information

Potential of Virtualization Technology for Long-term Data Preservation

Potential of Virtualization Technology for Long-term Data Preservation Potential of Virtualization Technology for Long-term Data Preservation J Blomer on behalf of the CernVM Team jblomer@cern.ch CERN PH-SFT 1 / 12 Introduction Potential of Virtualization Technology Preserve

More information

The Evolution of Cloud Computing in ATLAS

The Evolution of Cloud Computing in ATLAS The Evolution of Cloud Computing in ATLAS Ryan Taylor on behalf of the ATLAS collaboration CHEP 2015 Evolution of Cloud Computing in ATLAS 1 Outline Cloud Usage and IaaS Resource Management Software Services

More information

KVM, OpenStack, and the Open Cloud

KVM, OpenStack, and the Open Cloud KVM, OpenStack, and the Open Cloud Adam Jollans, IBM & Mike Kadera, Intel CloudOpen Europe - October 13, 2014 13Oct14 Open VirtualizaGon Alliance 1 Agenda A Brief History of VirtualizaGon KVM Architecture

More information

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida Plug-and-play Virtual Appliance Clusters Running Hadoop Dr. Renato Figueiredo ACIS Lab - University of Florida Advanced Computing and Information Systems laboratory Introduction You have so far learned

More information

YAN, Tian. On behalf of distributed computing group. Institute of High Energy Physics (IHEP), CAS, China. CHEP-2015, Apr. 13-17th, OIST, Okinawa

YAN, Tian. On behalf of distributed computing group. Institute of High Energy Physics (IHEP), CAS, China. CHEP-2015, Apr. 13-17th, OIST, Okinawa YAN, Tian On behalf of distributed computing group Institute of High Energy Physics (IHEP), CAS, China CHEP-2015, Apr. 13-17th, OIST, Okinawa Distributed computing for BESIII Other experiments wish to

More information

LSKA 2010 Survey Report Job Scheduler

LSKA 2010 Survey Report Job Scheduler LSKA 2010 Survey Report Job Scheduler Graduate Institute of Communication Engineering {r98942067, r98942112}@ntu.edu.tw March 31, 2010 1. Motivation Recently, the computing becomes much more complex. However,

More information

Data Centers and Cloud Computing. Data Centers. MGHPCC Data Center. Inside a Data Center

Data Centers and Cloud Computing. Data Centers. MGHPCC Data Center. Inside a Data Center Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises

More information

An Efficient Use of Virtualization in Grid/Cloud Environments. Supervised by: Elisa Heymann Miquel A. Senar

An Efficient Use of Virtualization in Grid/Cloud Environments. Supervised by: Elisa Heymann Miquel A. Senar An Efficient Use of Virtualization in Grid/Cloud Environments. Arindam Choudhury Supervised by: Elisa Heymann Miquel A. Senar Index Introduction Motivation Objective State of Art Proposed Solution Experimentations

More information

Experiences with the GLUE information schema in the LCG/EGEE production Grid

Experiences with the GLUE information schema in the LCG/EGEE production Grid Experiences with the GLUE information schema in the LCG/EGEE production Grid Stephen Burke, Sergio Andreozzi and Laurence Field CHEP07, Victoria, Canada www.eu-egee.org EGEE and glite are registered trademarks

More information

E-mail: guido.negri@cern.ch, shank@bu.edu, dario.barberis@cern.ch, kors.bos@cern.ch, alexei.klimentov@cern.ch, massimo.lamanna@cern.

E-mail: guido.negri@cern.ch, shank@bu.edu, dario.barberis@cern.ch, kors.bos@cern.ch, alexei.klimentov@cern.ch, massimo.lamanna@cern. *a, J. Shank b, D. Barberis c, K. Bos d, A. Klimentov e and M. Lamanna a a CERN Switzerland b Boston University c Università & INFN Genova d NIKHEF Amsterdam e BNL Brookhaven National Laboratories E-mail:

More information

Deploying a distributed data storage system on the UK National Grid Service using federated SRB

Deploying a distributed data storage system on the UK National Grid Service using federated SRB Deploying a distributed data storage system on the UK National Grid Service using federated SRB Manandhar A.S., Kleese K., Berrisford P., Brown G.D. CCLRC e-science Center Abstract As Grid enabled applications

More information

Sistemi Operativi e Reti. Cloud Computing

Sistemi Operativi e Reti. Cloud Computing 1 Sistemi Operativi e Reti Cloud Computing Facoltà di Scienze Matematiche Fisiche e Naturali Corso di Laurea Magistrale in Informatica Osvaldo Gervasi ogervasi@computer.org 2 Introduction Technologies

More information

Adding IaaS Clouds to the ATLAS Computing Grid

Adding IaaS Clouds to the ATLAS Computing Grid Adding IaaS Clouds to the ATLAS Computing Grid Ashok Agarwal, Frank Berghaus, Andre Charbonneau, Mike Chester, Asoka de Silva, Ian Gable, Joanna Huang, Colin Leavett-Brown, Michael Paterson, Randall Sobie,

More information

Build & Manage Clouds with Red Hat Cloud Infrastructure Products. TONI WILLBERG Solution Architect Red Hat toni@redhat.com

Build & Manage Clouds with Red Hat Cloud Infrastructure Products. TONI WILLBERG Solution Architect Red Hat toni@redhat.com Build & Manage Clouds with Red Hat Cloud Infrastructure Products TONI WILLBERG Solution Architect Red Hat toni@redhat.com AGENDA Cloud Concepts Market Overview Evolution to Cloud Workloads Evolution to

More information

Infrastructure Clouds for Science and Education: Platform Tools

Infrastructure Clouds for Science and Education: Platform Tools Infrastructure Clouds for Science and Education: Platform Tools Kate Keahey, Renato J. Figueiredo, John Bresnahan, Mike Wilde, David LaBissoniere Argonne National Laboratory Computation Institute, University

More information

Clodoaldo Barrera Chief Technical Strategist IBM System Storage. Making a successful transition to Software Defined Storage

Clodoaldo Barrera Chief Technical Strategist IBM System Storage. Making a successful transition to Software Defined Storage Clodoaldo Barrera Chief Technical Strategist IBM System Storage Making a successful transition to Software Defined Storage Open Server Summit Santa Clara Nov 2014 Data at the core of everything Data is

More information

Constructing your Private Cloud Strategies for IT Administrators & Decision Makers

Constructing your Private Cloud Strategies for IT Administrators & Decision Makers Constructing your Private Cloud Strategies for IT Administrators & Decision Makers Greg Shields Senior Partner and Principal Technologist, Concentrated Technology, LLC http://concentratedtech.com Virtualization

More information

Integrating a heterogeneous and shared Linux cluster into grids

Integrating a heterogeneous and shared Linux cluster into grids Integrating a heterogeneous and shared Linux cluster into grids 1,2 1 1,2 1 V. Büge, U. Felzmann, C. Jung, U. Kerzel, 1 1 1 M. Kreps, G. Quast, A. Vest 1 2 DPG Frühjahrstagung March 28 31, 2006 Dortmund

More information

PES. Batch virtualization and Cloud computing. Part 1: Batch virtualization. Batch virtualization and Cloud computing

PES. Batch virtualization and Cloud computing. Part 1: Batch virtualization. Batch virtualization and Cloud computing Batch virtualization and Cloud computing Batch virtualization and Cloud computing Part 1: Batch virtualization Tony Cass, Sebastien Goasguen, Belmiro Moreira, Ewan Roche, Ulrich Schwickerath, Romain Wartel

More information

Data Centers and Cloud Computing. Data Centers

Data Centers and Cloud Computing. Data Centers Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises

More information

Cloud and Virtualization to Support Grid Infrastructures

Cloud and Virtualization to Support Grid Infrastructures ESAC GRID Workshop '08 ESAC, Villafranca del Castillo, Spain 11-12 December 2008 Cloud and Virtualization to Support Grid Infrastructures Distributed Systems Architecture Research Group Universidad Complutense

More information

Stackato PaaS Architecture: How it works and why.

Stackato PaaS Architecture: How it works and why. Stackato PaaS Architecture: How it works and why. White Paper Published in 2012 Stackato PaaS Architecture: How it works and why. Stackato is software for creating a private Platform-as-a-Service (PaaS).

More information

Docker : devops, shared registries, HPC and emerging use cases. François Moreews & Olivier Sallou

Docker : devops, shared registries, HPC and emerging use cases. François Moreews & Olivier Sallou Docker : devops, shared registries, HPC and emerging use cases François Moreews & Olivier Sallou Presentation Docker is an open-source engine to easily create lightweight, portable, self-sufficient containers

More information

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH CONTENTS Introduction... 4 System Components... 4 OpenNebula Cloud Management Toolkit... 4 VMware

More information

Elastic Management of Cluster based Services in the Cloud

Elastic Management of Cluster based Services in the Cloud First Workshop on Automated Control for Datacenters and Clouds (ACDC09) June 19th, Barcelona, Spain Elastic Management of Cluster based Services in the Cloud Rafael Moreno Vozmediano, Ruben S. Montero,

More information

Cloud Computing. A new kind of developers? Presentation by. Nick Barcet nick.barcet@canonical.com

Cloud Computing. A new kind of developers? Presentation by. Nick Barcet nick.barcet@canonical.com Cloud Computing A new kind of developers? Presentation by Nick Barcet nick.barcet@canonical.com www.canonical.com July 2011 Cloud computing stack Salesforce.com, GoogleDocs, Office, etc... GoogleApps,

More information

Private Cloud with Fusion Middleware

<Insert Picture Here> Private Cloud with Fusion Middleware Private Cloud with Fusion Middleware Duško Vukmanović Principal Sales Consultant, Oracle dusko.vukmanovic@oracle.com The following is intended to outline our general product direction.

More information

High Performance Computing OpenStack Options. September 22, 2015

High Performance Computing OpenStack Options. September 22, 2015 High Performance Computing OpenStack PRESENTATION TITLE GOES HERE Options September 22, 2015 Today s Presenters Glyn Bowden, SNIA Cloud Storage Initiative Board HP Helion Professional Services Alex McDonald,

More information

Experience with Server Self Service Center (S3C)

Experience with Server Self Service Center (S3C) Experience with Server Self Service Center (S3C) Juraj Sucik, Sebastian Bukowiec IT Department, CERN, CH-1211 Genève 23, Switzerland E-mail: juraj.sucik@cern.ch, sebastian.bukowiec@cern.ch Abstract. CERN

More information

Manjrasoft Market Oriented Cloud Computing Platform

Manjrasoft Market Oriented Cloud Computing Platform Manjrasoft Market Oriented Cloud Computing Platform Innovative Solutions for 3D Rendering Aneka is a market oriented Cloud development and management platform with rapid application development and workload

More information

An objective comparison test of workload management systems

An objective comparison test of workload management systems An objective comparison test of workload management systems Igor Sfiligoi 1 and Burt Holzman 1 1 Fermi National Accelerator Laboratory, Batavia, IL 60510, USA E-mail: sfiligoi@fnal.gov Abstract. The Grid

More information

This presentation provides an overview of the architecture of the IBM Workload Deployer product.

This presentation provides an overview of the architecture of the IBM Workload Deployer product. This presentation provides an overview of the architecture of the IBM Workload Deployer product. Page 1 of 17 This presentation starts with an overview of the appliance components and then provides more

More information

Private Cloud Management

Private Cloud Management Private Cloud Management Speaker Systems Engineer Unified Data Center & Cloud Team Germany Juni 2016 Agenda Cisco Enterprise Cloud Suite Two Speeds of Applications DevOps Starting Point into PaaS Cloud

More information

Mobile Cloud Computing T-110.5121 Open Source IaaS

Mobile Cloud Computing T-110.5121 Open Source IaaS Mobile Cloud Computing T-110.5121 Open Source IaaS Tommi Mäkelä, Otaniemi Evolution Mainframe Centralized computation and storage, thin clients Dedicated hardware, software, experienced staff High capital

More information

www.see-grid-sci.eu Regional SEE-GRID-SCI Training for Site Administrators Institute of Physics Belgrade March 5-6, 2009

www.see-grid-sci.eu Regional SEE-GRID-SCI Training for Site Administrators Institute of Physics Belgrade March 5-6, 2009 SEE-GRID-SCI Virtualization and Grid Computing with XEN www.see-grid-sci.eu Regional SEE-GRID-SCI Training for Site Administrators Institute of Physics Belgrade March 5-6, 2009 Milan Potocnik University

More information

Computing at the HL-LHC

Computing at the HL-LHC Computing at the HL-LHC Predrag Buncic on behalf of the Trigger/DAQ/Offline/Computing Preparatory Group ALICE: Pierre Vande Vyvre, Thorsten Kollegger, Predrag Buncic; ATLAS: David Rousseau, Benedetto Gorini,

More information

An Introduction to Virtualization and Cloud Technologies to Support Grid Computing

An Introduction to Virtualization and Cloud Technologies to Support Grid Computing New Paradigms: Clouds, Virtualization and Co. EGEE08, Istanbul, September 25, 2008 An Introduction to Virtualization and Cloud Technologies to Support Grid Computing Distributed Systems Architecture Research

More information

Running Oracle Databases in a z Systems Cloud environment

Running Oracle Databases in a z Systems Cloud environment Running Oracle Databases in a z Systems Cloud environment Sam Amsavelu samvelu@us.ibm.com ISV & Channels Technical Sales - Oracle IBM Advanced Technical Skills (ATS), America Technical University/Symposia

More information

IPv6 on OpenStack. Feature Parity is a Tricky Question

IPv6 on OpenStack. Feature Parity is a Tricky Question IPv6 on OpenStack Feature Parity is a Tricky Question Today s Sequence Quick Review of OpenStack Is OpenStack IPv6 Ready? Case Study: CERN s use of OpenStack Takeaways Today s Sequence Quick Review of

More information

Computer Science. About PaaS Security. Donghoon Kim Henry E. Schaffer Mladen A. Vouk

Computer Science. About PaaS Security. Donghoon Kim Henry E. Schaffer Mladen A. Vouk About PaaS Security Donghoon Kim Henry E. Schaffer Mladen A. Vouk North Carolina State University, USA May 21, 2015 @ ICACON 2015 Outline Introduction Background Contribution PaaS Vulnerabilities and Countermeasures

More information

Global Grid User Support - GGUS - in the LCG & EGEE environment

Global Grid User Support - GGUS - in the LCG & EGEE environment Global Grid User Support - GGUS - in the LCG & EGEE environment Torsten Antoni (torsten.antoni@iwr.fzk.de) Why Support? New support groups Network layer Resource centers CIC / GOC / etc. more to come New

More information

Manjrasoft Market Oriented Cloud Computing Platform

Manjrasoft Market Oriented Cloud Computing Platform Manjrasoft Market Oriented Cloud Computing Platform Aneka Aneka is a market oriented Cloud development and management platform with rapid application development and workload distribution capabilities.

More information

Building Docker Cloud Services with Virtuozzo

Building Docker Cloud Services with Virtuozzo Building Docker Cloud Services with Virtuozzo Improving security and performance of application containers services in the cloud EXECUTIVE SUMMARY Application containers, and Docker in particular, are

More information

Shared Computing Driving Discovery: From the Large Hadron Collider to Virus Hunting. Frank Würthwein

Shared Computing Driving Discovery: From the Large Hadron Collider to Virus Hunting. Frank Würthwein Shared Computing Driving Discovery: From the Large Hadron Collider to Virus Hunting Frank Würthwein Professor of Physics University of California San Diego February 14th, 2015 The Science of the LHC The

More information

Managing a Tier-2 Computer Centre with a Private Cloud Infrastructure

Managing a Tier-2 Computer Centre with a Private Cloud Infrastructure Managing a Tier-2 Computer Centre with a Private Cloud Infrastructure Stefano Bagnasco, Riccardo Brunetti, Stefano Lusso (INFN-Torino), Dario Berzano (CERN) ACAT2013 Beijing, May 16-21, 2013 motivation

More information

Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com

Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com W H I T E P A P E R O r a c l e V i r t u a l N e t w o r k i n g D e l i v e r i n g F a b r i c

More information

Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware

Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware R. Goranova University of Sofia St. Kliment Ohridski,

More information

OpenStack Introduction. November 4, 2015

OpenStack Introduction. November 4, 2015 OpenStack Introduction November 4, 2015 Application Platforms Undergoing A Major Shift What is OpenStack Open Source Cloud Software Launched by NASA and Rackspace in 2010 Massively scalable Managed by

More information

Virtualization, SDN and NFV

Virtualization, SDN and NFV Virtualization, SDN and NFV HOW DO THEY FIT TOGETHER? Traditional networks lack the flexibility to keep pace with dynamic computing and storage needs of today s data centers. In order to implement changes,

More information

Big Data Trends and HDFS Evolution

Big Data Trends and HDFS Evolution Big Data Trends and HDFS Evolution Sanjay Radia Founder & Architect Hortonworks Inc Page 1 Hello Founder, Hortonworks Part of the Hadoop team at Yahoo! since 2007 Chief Architect of Hadoop Core at Yahoo!

More information

SUSE Cloud 2.0. Pete Chadwick. Douglas Jarvis. Senior Product Manager pchadwick@suse.com. Product Marketing Manager djarvis@suse.

SUSE Cloud 2.0. Pete Chadwick. Douglas Jarvis. Senior Product Manager pchadwick@suse.com. Product Marketing Manager djarvis@suse. SUSE Cloud 2.0 Pete Chadwick Douglas Jarvis Senior Product Manager pchadwick@suse.com Product Marketing Manager djarvis@suse.com SUSE Cloud SUSE Cloud is an open source software solution based on OpenStack

More information

Making a Smooth Transition to a Hybrid Cloud with Microsoft Cloud OS

Making a Smooth Transition to a Hybrid Cloud with Microsoft Cloud OS Making a Smooth Transition to a Hybrid Cloud with Microsoft Cloud OS Transitioning from today s highly virtualized data center environments to a true cloud environment requires solutions that let companies

More information

Enterprise Storage Solution for Hyper-V Private Cloud and VDI Deployments using Sanbolic s Melio Cloud Software Suite April 2011

Enterprise Storage Solution for Hyper-V Private Cloud and VDI Deployments using Sanbolic s Melio Cloud Software Suite April 2011 Enterprise Storage Solution for Hyper-V Private Cloud and VDI Deployments using Sanbolic s Melio Cloud Software Suite April 2011 Executive Summary Large enterprise Hyper-V deployments with a large number

More information

7 Myths about Backup & DR in Virtual Environments

7 Myths about Backup & DR in Virtual Environments NEW White Paper 7 Myths about Backup & DR in Virtual Environments by Eric Siebert, VMware vexpert Eric Siebert VMware vexpert Eric Siebert is an IT industry veteran, speaker, author and blogger with more

More information