Dynamic Resource Provisioning with HTCondor in the Cloud
|
|
- Spencer Barker
- 8 years ago
- Views:
Transcription
1 Dynamic Resource Provisioning with HTCondor in the Cloud Ryan Taylor Frank Berghaus 1
2 Overview Review of Condor + Cloud Scheduler system Condor job slot configuration Dynamic slot creation Automatic slot defragmentation Target Shares Flexible use of cloud resources 2
3 Be Dynamic Do the right thing... On any type of cloud or hypervisor Nimbus, Openstack, KVM, Xen Anywhere in the world Data Federations Squid Federator: Shoal For any kind of job SCORE MCORE HIMEM New requirement... using just one image, and minimizing static configuration Old requirement 3
4 Cloud Scheduler A python package for managing VMs on IaaS clouds Users submit HTCondor jobs Optional attributes specify VM properties Dynamically manages quantity & type of VMs in response to user demand Easily connect to many IaaS clouds, aggregate resources, present them as HTCondor batch system Used by ATLAS, Belle II, CANFAR, BaBar gc/cloud scheduler Publication in CHEP 2010 Proceedings 4
5 VMs boot, join Condor pool Contextualized with x509 credential for authentication VMs run jobs continuously CS automatically retires unneeded VMs User Job Pilot Job 5
6 VM Image ucernvm: OS loaded on demand via CVMFS 10 MB image (kernel + CVMFS client) Contextualization: Cloud init and puppet Configure Condor, CVMFS, Shoal, grid environment, etc. modules Developed by Frank Berghaus and Henric Öhman Instrumented with Ganglia monitoring agm.cern.ch 6
7 ATLAS Production in the Cloud (Jan Sep 2014) Helix Nebula Also running 10% of Belle II jobs in the cloud with Dirac Google 1.2M Completed Jobs in 2014 CERN North America Australia UK Jan 2014 Sep 2014 Frank Berghaus Over 1.2M ATLAS jobs completed Single core and multi core production 7
8 Condor Slot Configuration Static slots: 8 1 core slots (or 1 8 core slot) NUM_SLOTS = 8 SLOT1_USER = slot01 old way SLOT2_USER = slot02... Dynamic slots NUM_SLOTS_TYPE_1 = 1 SLOT_TYPE_1 = cpus=100% SLOT_TYPE_1_PARTITIONABLE = True SLOT1_1_USER = slot01 new way SLOT1_2_USER = slot
9 Dynamic Slot Creation Jobs specify resource requirements request_cpus = 2 request_memory = 8000 request_disk = # MB # KB Condor dynamically creates a slot Problem: VMs get fragmented into smaller slots Larger jobs can't run Solution: DEFRAG daemon coalesces slots together again 9
10 Automatic Slot Defragmentation Currently being tested/tuned. Thanks to Andrew Lahiff from RAL for sample configuration DAEMON_LIST = DEFRAG DEFRAG_INTERVAL = 600 DEFRAG_DRAINING_MACHINES_PER_HOUR = 10.0 DEFRAG_MAX_CONCURRENT_DRAINING = 30 DEFRAG_MAX_WHOLE_MACHINES = 50 DEFRAG_SCHEDULE = graceful DEFRAG.SETTABLE_ATTRS_ADMINISTRATOR = DEFRAG_MAX_CONCURRENT_DRAINING,DEFRAG_DRAINING_MACHINES_PER_HOUR,DEFRAG_ MAX_WHOLE_MACHINES ENABLE_RUNTIME_CONFIG = TRUE DEFRAG_RANK = ifthenelse(cpus >= 8, -10, (TotalCpus - Cpus)/(8.0 Cpus)) 10
11 Target Shares Use Condor groups to prioritize job types GROUP_NAMES = group_analysis, group_production GROUP_QUOTA_DYNAMIC_group_analysis = 0.05 GROUP_QUOTA_DYNAMIC_group_production = 0.95 GROUP_ACCEPT_SURPLUS = True In the job definition add: AccountingGroup = "group_production" or AccountingGroup = "group_analysis" Thanks to Joanna Huang and Sean Crosby from the Australian ATLAS group 11
12 Flexible Cloud Resources CERN Cloud Scheduler cloudscheduler.cern.ch Dynamically provisioned slots GridPP cloud group CERN cloud group 12
13 Flexible Cloud Resources GridPP_CLOUD GridPP_CLOUD_HIMEM request_cpus = 1 TargetClouds = GridPP request_cpus = 1 TargetClouds = CERN CERN-Prod_CLOUD request_memory = 8000 TargetClouds = GridPP request_cpus = 8 TargetClouds = CERN CERN-Prod_CLOUD _MCORE To run jobs with arbitrary resource requirements: just submit them GridPP cloud group Automatic No dedicated nodes CERN cloud group 13
14 Dynamic Resource Provisioning aka How to provision MCORE resources using IaaS 1)Connect IaaS cloud to Cloud Scheduler 2)Create Panda queue 3)Compose JDL 4)Configure Pilot Factory 5)Profit 14
15 Questions Bonus Material 15
16 Credential Transport User Securely delegates user credentials to VMs, and authenticates VMs joining the Condor pool.
17 Security Configuration Worker GSI_DAEMON_DIRECTORY = /etc/grid security GSI_DAEMON_CERT = /etc/grid security/hostcert.pem GSI_DAEMON_KEY = /etc/grid security/hostkey.pem Central Manager SEC_DEFAULT_AUTHENTICATION = REQUIRED SEC_DEFAULT_AUTHENTICATION_METHODS = GSI SEC_DEFAULT_ENCRYPTION = REQUIRED SEC_DEFAULT_ENCRYPTION_METHODS = 3DES SEC_DEFAULT_INTEGRITY = REQUIRED GRIDMAP = /etc/grid security/grid mapfile GSI_DAEMON_CERT = /etc/grid security/condor/condorcert.pem GSI_DAEMON_KEY = /etc/grid security/condor/condorkey.pem GSI_DAEMON_TRUSTED_CA_DIR = /etc/grid security/certificates/ GSI_DELEGATION_KEYBITS = 1024 GSI_DAEMON_NAME = /C=CA/O=Grid/CN=condor/condor.heprc.uvic.ca, /C=CA/O=Grid/CN=condorworker/condor.heprc.uvic.ca
18 Virtual Machine Virtual Machine Cloud Interface Cloud Interface... Virtual Machine Virtual Machine GCE Cloud Interface OpenStack OpenStack Scheduler status communication Cloud Scheduler Virtual Machine Virtual Machine Condor Central Manager User 18
19 Shoal: Squid Cache Federator Jun S&C Week Shoal tracks squids discover new ones remove missing ones Workers find best squids Build a configuration less mesh instead of statically defined site specific mappings CHEP 2013 Poster github.com/hep gc/shoal WLCG Proxy Discovery TF Ryan Taylor - ATLAS Cloud Computing R &D 19
20 Dynamic Federation Sep S&C Workshop High performance aggressive metadata caching in RAM maximal concurrency scalable ~ 10^6 hit/s per core 24 GB of RAM for metadata cache is enough for ~100 PB of data in end points Well designed stateless, no persistency standard components and protocols, not HEP specific simple, lightweight trivial to add endpoints no site action needed! Dynamic automatically detect and avoid offline endpoints federating is done on the fly 10/27/14 Ryan Taylor - ATLAS S&C Week 20
Adding IaaS Clouds to the ATLAS Computing Grid
Adding IaaS Clouds to the ATLAS Computing Grid Ashok Agarwal, Frank Berghaus, Andre Charbonneau, Mike Chester, Asoka de Silva, Ian Gable, Joanna Huang, Colin Leavett-Brown, Michael Paterson, Randall Sobie,
More informationThe Evolution of Cloud Computing in ATLAS
The Evolution of Cloud Computing in ATLAS Ryan Taylor on behalf of the ATLAS collaboration CHEP 2015 Evolution of Cloud Computing in ATLAS 1 Outline Cloud Usage and IaaS Resource Management Software Services
More informationHTCondor at the RAL Tier-1
HTCondor at the RAL Tier-1 Andrew Lahiff, Alastair Dewhurst, John Kelly, Ian Collier, James Adams STFC Rutherford Appleton Laboratory HTCondor Week 2014 Outline Overview of HTCondor at RAL Monitoring Multi-core
More informationDynamic Slot Tutorial. Condor Project Computer Sciences Department University of Wisconsin-Madison
Dynamic Slot Tutorial Condor Project Computer Sciences Department University of Wisconsin-Madison Outline Why we need partitionable slots How they ve worked since 7.2 What s new in 7.8 What s still left
More informationShoal: IaaS Cloud Cache Publisher
University of Victoria Faculty of Engineering Winter 2013 Work Term Report Shoal: IaaS Cloud Cache Publisher Department of Physics University of Victoria Victoria, BC Mike Chester V00711672 Work Term 3
More informationHEP Data-Intensive Distributed Cloud Computing System Requirements Specification Document
HEP Data-Intensive Distributed Cloud Computing System Requirements Specification Document CANARIE NEP-101 Project University of Victoria HEP Computing Group December 18, 2013 Version 1.0 1 Revision History
More informationBatch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb
Batch and Cloud overview Andrew McNab University of Manchester GridPP and LHCb Overview Assumptions Batch systems The Grid Pilot Frameworks DIRAC Virtual Machines Vac Vcycle Tier-2 Evolution Containers
More informationHEP Compu*ng in a Context- Aware Cloud Environment
HEP Compu*ng in a Context- Aware Cloud Environment Randall Sobie A.Charbonneau F.Berghaus R.Desmarais I.Gable C.LeaveC- Brown M.Paterson R.Taylor InsItute of ParIcle Physics University of Victoria and
More informationContext-aware cloud computing for HEP
Department of Physics and Astronomy, University of Victoria, Victoria, British Columbia, Canada V8W 2Y2 E-mail: rsobie@uvic.ca The use of cloud computing is increasing in the field of high-energy physics
More informationRunning real jobs in virtual machines. Andrew McNab University of Manchester
Running real jobs in virtual machines Andrew McNab University of Manchester Overview The Grid we have now Strategies for starting VMs Fabric / Cloud / Vac Hypervisors Disk images Contextualization Pilot
More informationClouds for research computing
Clouds for research computing Randall Sobie Institute of Particle Physics University of Victoria Collaboration UVIC, NRC (Ottawa), NRC-HIA (Victoria) Randall Sobie IPP/University of Victoria 1 Research
More informationPoS(EGICF12-EMITC2)005
, Or: How One HEP Experiment Is Evaluating Strategies to Incorporate The Cloud into the Existing Grid Infrastructure Daniel Colin van der Ster 1 E-mail: daniel.colin.vanderster@cern.ch Fernando Harald
More informationExploiting Virtualization and Cloud Computing in ATLAS
Home Search Collections Journals About Contact us My IOPscience Exploiting Virtualization and Cloud Computing in ATLAS This content has been downloaded from IOPscience. Please scroll down to see the full
More informationCloud services for the Fermilab scientific stakeholders
Cloud services for the Fermilab scientific stakeholders S Timm 1, G Garzoglio 1, P Mhashilkar 1*, J Boyd 1, G Bernabeu 1, N Sharma 1, N Peregonow 1, H Kim 1, S Noh 2,, S Palur 3, and I Raicu 3 1 Scientific
More informationThe Evolution of Cloud Computing in ATLAS
The Evolution of Cloud Computing in ATLAS Ryan P Taylor, 1 Frank Berghaus, 1 Franco Brasolin, 2 Cristovao Jose Domingues Cordeiro, 3 Ron Desmarais, 1 Laurence Field, 3 Ian Gable, 1 Domenico Giordano, 3
More informationOpenStack Introduction. November 4, 2015
OpenStack Introduction November 4, 2015 Application Platforms Undergoing A Major Shift What is OpenStack Open Source Cloud Software Launched by NASA and Rackspace in 2010 Massively scalable Managed by
More informationSimulation and user analysis of BaBar data in a distributed cloud
Simulation and user analysis of BaBar data in a distributed cloud A. Agarwal, University of Victoria M. Anderson, University of Victoria P. Armstrong, University of Victoria A. Charbonneau, National Research
More informationDistributed IaaS Clouds and 100G Networking for HEP applications
Distributed IaaS Clouds and 100G Networking for HEP applications HPCS 2013 June 3 Ian Gable A.Agarwal, A.Charbonneau, C.Leavett- Brown, K Lewall. R. Impey, M.Paterson, W. Podiama, R.J. Sobie, R.Taylor
More informationSolution for private cloud computing
The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details Use cases By scientist By HEP experiment System requirements and installation How to get it? 2 What
More informationEnabling multi-cloud resources at CERN within the Helix Nebula project. D. Giordano (CERN IT-SDC) HEPiX Spring 2014 Workshop 23 May 2014
Enabling multi-cloud resources at CERN within the Helix Nebula project D. Giordano (CERN IT-) HEPiX Spring 2014 Workshop This document produced by Members of the Helix Nebula consortium is licensed under
More informationIntegration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept
Integration of Virtualized Workernodes in Batch Queueing Systems, Dr. Armin Scheurer, Oliver Oberst, Prof. Günter Quast INSTITUT FÜR EXPERIMENTELLE KERNPHYSIK FAKULTÄT FÜR PHYSIK KIT University of the
More informationDynamic Resource Distribution Across Clouds
University of Victoria Faculty of Engineering Winter 2010 Work Term Report Dynamic Resource Distribution Across Clouds Department of Physics University of Victoria Victoria, BC Michael Paterson V00214440
More informationDSS. Data & Storage Services. Cloud storage performance and first experience from prototype services at CERN
Data & Storage Cloud storage performance and first experience from prototype services at CERN Maitane Zotes Resines, Seppo S. Heikkila, Dirk Duellmann, Geoffray Adde, Rainer Toebbicke, CERN James Hughes,
More informationCloud Computing PES. (and virtualization at CERN) Cloud Computing. GridKa School 2011, Karlsruhe. Disclaimer: largely personal view of things
PES Cloud Computing Cloud Computing (and virtualization at CERN) Ulrich Schwickerath et al With special thanks to the many contributors to this presentation! GridKa School 2011, Karlsruhe CERN IT Department
More informationCHEP 2013. Cloud Bursting with glideinwms Means to satisfy ever increasing computing needs for Scientific Workflows
CHEP 2013 Cloud Bursting with glideinwms Means to satisfy ever increasing computing needs for Scientific Workflows by I. Sfiligoi 1, P. Mhashilkar 2, A. Tiradani 2, B. Holzman 2, K. Larson 2 and M. Rynge
More informationStatus and Evolution of ATLAS Workload Management System PanDA
Status and Evolution of ATLAS Workload Management System PanDA Univ. of Texas at Arlington GRID 2012, Dubna Outline Overview PanDA design PanDA performance Recent Improvements Future Plans Why PanDA The
More informationWM Technical Evolution Group Report
WM Technical Evolution Group Report Davide Salomoni, INFN for the WM TEG February 7, 2012 The WM TEG, organization Mailing list, wlcg-teg-workload-mgmt@cern.ch 48 people currently subscribed; representation
More informationDeploying Business Virtual Appliances on Open Source Cloud Computing
International Journal of Computer Science and Telecommunications [Volume 3, Issue 4, April 2012] 26 ISSN 2047-3338 Deploying Business Virtual Appliances on Open Source Cloud Computing Tran Van Lang 1 and
More informationUser-friendly access to Grid and Cloud resources for 18scientific th 19 th computing January 2016 1 / 21
User-friendly access to Grid and Cloud resources for scientific computing Dr. Alexander Richards Imperial College Sci., Tech. & Med. UK (IC) 18 th 19 th January 2016 Cloud Services for Synchronisation
More informationVirtualisation Cloud Computing at the RAL Tier 1. Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013
Virtualisation Cloud Computing at the RAL Tier 1 Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013 Virtualisation @ RAL Context at RAL Hyper-V Services Platform Scientific Computing Department
More informationInfrastructure as a Service
Infrastructure as a Service Jose Castro Leon CERN IT/OIS Cloud Computing On-Demand Self-Service Scalability and Efficiency Resource Pooling Rapid elasticity 2 Infrastructure as a Service Objectives 90%
More information2) Xen Hypervisor 3) UEC
5. Implementation Implementation of the trust model requires first preparing a test bed. It is a cloud computing environment that is required as the first step towards the implementation. Various tools
More informationMirantis www.mirantis.com/training
TM Mirantis www.mirantis.com/training Goals Understand OpenStack purpose and use cases Understand OpenStack ecosystem o history o projects Understand OpenStack architecture o logical architecture o components
More informationRepoman: A Simple RESTful X.509 Virtual Machine Image Repository. Roger Impey
Repoman: A Simple RESTful X.509 Virtual Machine Image Repository Roger Impey Project Term University of Victoria R.J. Sobie, M. Anderson, P. Armstrong, A. Agarwal, Kyle Fransham, D. Harris, I. Gable, C.
More informationUsing S3 cloud storage with ROOT and CernVMFS. Maria Arsuaga-Rios Seppo Heikkila Dirk Duellmann Rene Meusel Jakob Blomer Ben Couturier
Using S3 cloud storage with ROOT and CernVMFS Maria Arsuaga-Rios Seppo Heikkila Dirk Duellmann Rene Meusel Jakob Blomer Ben Couturier INDEX Huawei cloud storages at CERN Old vs. new Huawei UDS comparative
More informationCloud Traceability Working Group Face to Face Report
Cloud Traceability Working Group Face to Face Report Ian Collier ian.collier@stfc.ac.uk GDB 10 th June 2015 Agenda & notes: https://indico.cern.ch/event/396202/ Areas of investigation from Feb F2F Externally
More informationScheduler in Cloud Computing using Open Source Technologies
Scheduler in Cloud Computing using Open Source Technologies Darshan Upadhyay Prof. Chirag Patel Student of M.E.I.T Asst. Prof. Computer Department S. S. Engineering College, Bhavnagar L. D. College of
More informationA Complete Open Cloud Storage, Virt, IaaS, PaaS. Dave Neary Open Source and Standards, Red Hat
A Complete Open Cloud Storage, Virt, IaaS, PaaS Dave Neary Open Source and Standards, Red Hat 1 Agenda 1. Traditional virtualization 2. The move to IaaS 3. Storage 4. PaaS, application encapsulation and
More informationHow To Run A Cloud Server On A Server Farm (Cloud)
StratusLab: Darn Simple Cloud Charles (Cal) Loomis (CNRS/LAL & SixSq Sàrl) FOSDEM 13: Cloud Devroom (3 February 2013) StratusLab What is it? Complete Infrastructure as a Service (IaaS) cloud distribution
More informationVirtualization. (and cloud computing at CERN)
Virtualization (and cloud computing at CERN) Ulrich Schwickerath Special thanks: Sebastien Goasguen Belmiro Moreira, Ewan Roche, Romain Wartel See also related presentations: CloudViews2010 conference,
More informationNebula Cloud Computing Project: Background, Technology, Operations, Challenges, and Status
Nebula Cloud Computing Project: Background, Technology, Operations, Challenges, and Status Raymond G. O Brien ARC CTO for IT Nebula Background Nebula cloud project evolved from a web app platform development
More informationCloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH
Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH CONTENTS Introduction... 4 System Components... 4 OpenNebula Cloud Management Toolkit... 4 VMware
More informationSUSE Cloud 2.0. Pete Chadwick. Douglas Jarvis. Senior Product Manager pchadwick@suse.com. Product Marketing Manager djarvis@suse.
SUSE Cloud 2.0 Pete Chadwick Douglas Jarvis Senior Product Manager pchadwick@suse.com Product Marketing Manager djarvis@suse.com SUSE Cloud SUSE Cloud is an open source software solution based on OpenStack
More informationDistributed Computing for CEPC. YAN Tian On Behalf of Distributed Computing Group, CC, IHEP for 4 th CEPC Collaboration Meeting, Sep.
Distributed Computing for CEPC YAN Tian On Behalf of Distributed Computing Group, CC, IHEP for 4 th CEPC Collaboration Meeting, Sep. 12-13, 2014 1 Outline Introduction Experience of BES-DIRAC Distributed
More informationSUSE Cloud Installation: Best Practices Using a SMT, Xen and Ceph Storage Environment
Best Practices Guide www.suse.com SUSE Cloud Installation: Best Practices Using a SMT, Xen and Ceph Storage Environment Written by B1 Systems GmbH Table of Contents Introduction...3 Use Case Overview...3
More informationElastic Management of Cluster based Services in the Cloud
First Workshop on Automated Control for Datacenters and Clouds (ACDC09) June 19th, Barcelona, Spain Elastic Management of Cluster based Services in the Cloud Rafael Moreno Vozmediano, Ruben S. Montero,
More informationIntroduction to OpenStack
Introduction to OpenStack Carlo Vallati PostDoc Reseracher Dpt. Information Engineering University of Pisa carlo.vallati@iet.unipi.it Cloud Computing - Definition Cloud Computing is a term coined to refer
More informationOpenStack Ecosystem and Xen Cloud Platform
OpenStack Ecosystem and Xen Cloud Platform Amit Naik Prasad Nirantar BMC Software 1 Agenda Introduction Rise of OpenStack OpenStack Details and Ecosystem OpenStack and Xen Cloud Platform - Demo Conclusion
More informationSUSE Cloud Installation: Best Practices Using an Existing SMT and KVM Environment
Best Practices Guide www.suse.com SUSE Cloud Installation: Best Practices Using an Existing SMT and KVM Environment Written by B1 Systems GmbH Table of Contents Introduction...3 Use Case Overview...3 Hardware
More informationA batch system for HEP applications on a distributed IaaS cloud
A batch system for HEP applications on a distributed IaaS cloud I. Gable, A. Agarwal, M. Anderson, P. Armstrong, K. Fransham, D. Harris C. Leavett-Brown, M. Paterson, D. Penfold-Brown, R.J. Sobie, M. Vliet
More informationData Centers and Cloud Computing
Data Centers and Cloud Computing CS377 Guest Lecture Tian Guo 1 Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing Case Study: Amazon EC2 2 Data Centers
More informationCloud Accounting. Laurence Field IT/SDC 22/05/2014
Cloud Accounting Laurence Field IT/SDC 22/05/2014 Helix Nebula Pathfinder project Development and exploitation Cloud Computing Infrastructure Divided into supply and demand Three flagship applications
More informationCMS Experience Provisioning Cloud Resources with GlideinWMS. Anthony Tiradani HTCondor Week 2015 20 May 2015
CMS Experience Provisioning Cloud Resources with GlideinWMS Anthony Tiradani Week 2015 20 May 2015 glideinwms Quick Facts glideinwms is an open- source Fermilab CompuJng Sector product driven by CMS Heavy
More informationBuilding a Volunteer Cloud
Building a Volunteer Cloud Ben Segal, Predrag Buncic, David Garcia Quintas / CERN Daniel Lombrana Gonzalez / University of Extremadura Artem Harutyunyan / Yerevan Physics Institute Jarno Rantala / Tampere
More informationVirtualization, Grid, Cloud: Integration Paths for Scientific Computing
Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Or, where and how will my efficient large-scale computing applications be executed? D. Salomoni, INFN Tier-1 Computing Manager Davide.Salomoni@cnaf.infn.it
More informationThe CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)"
The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)" J.A. Coarasa " CERN, Geneva, Switzerland" for the CMS TriDAS group." " CHEP2013, 14-18 October 2013, Amsterdam, The Netherlands
More informationIntegration of Virtualized Worker Nodes in Batch Systems
Integration of Virtualized Worker Nodes Dr. A. Scheurer, Dr. V. Büge, O. Oberst, P. Krauß Linuxtag 2010, Berlin, Session: Cloud Computing, Talk ID: #16197 KIT University of the State of Baden-Wuerttemberg
More informationPerformance Comparison Analysis of Linux Container and Virtual Machine for Building Cloud
, pp.105-111 http://dx.doi.org/10.14257/astl.2014.66.25 Performance Comparison Analysis of Linux Container and Virtual Machine for Building Cloud Kyoung-Taek Seo 1, Hyun-Seo Hwang 1, Il-Young Moon 1, Oh-Young
More informationYAN, Tian. On behalf of distributed computing group. Institute of High Energy Physics (IHEP), CAS, China. CHEP-2015, Apr. 13-17th, OIST, Okinawa
YAN, Tian On behalf of distributed computing group Institute of High Energy Physics (IHEP), CAS, China CHEP-2015, Apr. 13-17th, OIST, Okinawa Distributed computing for BESIII Other experiments wish to
More informationShareable Private Space on a Public Cloud
Shareable Private Space on a Public Cloud 1.0 Introduction: Sharable private space on public cloud (a distributed computing platform) is nontrivial task. With immerse of Free & Open Source Software (FOSS),
More informationHPC performance applications on Virtual Clusters
Panagiotis Kritikakos EPCC, School of Physics & Astronomy, University of Edinburgh, Scotland - UK pkritika@epcc.ed.ac.uk 4 th IC-SCCE, Athens 7 th July 2010 This work investigates the performance of (Java)
More informationOpenstack. Cloud computing with Openstack. Saverio Proto saverio.proto@switch.ch
Openstack Cloud computing with Openstack Saverio Proto saverio.proto@switch.ch Lugano, 23/03/2016 Agenda SWITCH role in Openstack and Cloud Computing What is Virtualization? Why is Cloud computing more
More informationCERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT
SS Data & Storage CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT HEPiX Fall 2012 Workshop October 15-19, 2012 Institute of High Energy Physics, Beijing, China SS Outline
More informationCernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment
CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment George Lestaris - Ioannis Charalampidis D. Berzano, J. Blomer, P. Buncic, G. Ganis and R. Meusel PH-SFT /
More informationUsing SUSE Cloud to Orchestrate Multiple Hypervisors and Storage at ADP
Using SUSE Cloud to Orchestrate Multiple Hypervisors and Storage at ADP Agenda ADP Cloud Vision and Requirements Introduction to SUSE Cloud Overview Whats New VMWare intergration HyperV intergration ADP
More informationEfficient Cloud Management for Parallel Data Processing In Private Cloud
2012 International Conference on Information and Network Technology (ICINT 2012) IPCSIT vol. 37 (2012) (2012) IACSIT Press, Singapore Efficient Cloud Management for Parallel Data Processing In Private
More informationAlternative models to distribute VO specific software to WLCG sites: a prototype set up at PIC
EGEE and glite are registered trademarks Enabling Grids for E-sciencE Alternative models to distribute VO specific software to WLCG sites: a prototype set up at PIC Elisa Lanciotti, Arnau Bria, Gonzalo
More informationHTCondor within the European Grid & in the Cloud
HTCondor within the European Grid & in the Cloud Andrew Lahiff STFC Rutherford Appleton Laboratory HEPiX 2015 Spring Workshop, Oxford The Grid Introduction Computing element requirements Job submission
More informationPES. Batch virtualization and Cloud computing. Part 1: Batch virtualization. Batch virtualization and Cloud computing
Batch virtualization and Cloud computing Batch virtualization and Cloud computing Part 1: Batch virtualization Tony Cass, Sebastien Goasguen, Belmiro Moreira, Ewan Roche, Ulrich Schwickerath, Romain Wartel
More informationfiles without borders
files without borders exploring Internet-connected storage for research Fabio Hernandez fabio@in2p3.fr IN2P3 / CNRS computing center, Lyon, France FJPPL compu+ng workshop, Lyon, March 11th 2015 2 Preamble
More informationComparison and Evaluation of Open-source Cloud Management Software
Comparison and Evaluation of Open-source Cloud Management Software SRIVATSAN JAGANNATHAN Masters Degree Project Stockholm, Sweden XR-EE-LCN 2012:008 Comparison and Evaluation of Open-source Cloud Management
More informationThe EGI Federated Cloud e-infrastructure
The EGI Federated Cloud e-infrastructure Enol Fernández 1,2, Diego Scardaci 1,3, Álvaro López 2 1 EGI.eu, 2 IFCA (CSIC-UC), 3 INFN-Catania www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework
More informationDynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012
Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Thomas Hauth,, Günter Quast IEKP KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz
More informationPES. High Availability Load Balancing in the Agile Infrastructure. Platform & Engineering Services. HEPiX Bologna, April 2013
PES Platform & Engineering Services High Availability Load Balancing in the Agile Infrastructure HEPiX Bologna, April 2013 Vaggelis Atlidakis, -PES/PS Ignacio Reguero, -PES/PS PES Outline Core Concepts
More informationResearch computing in a distributed cloud environment
Research computing in a distributed cloud environment K. Fransham 1, A. Agarwal 1, P. Armstrong 1, A. Bishop 1, A. Charbonneau 2, R. Desmarais 1, N. Hill 3, I. Gable 1, S. Gaudet 3, S. Goliath 3, R. Impey
More informationVirtualisation Owen Synge (DESY HH)
Virtualisation Owen Synge (DESY HH) Owen Synge Virtualisation GridKa School, 5/9/2011 Overview > Background to Virtualisation > Why use it? > Who is using it? > What is going to be done with it? > What
More informationHow To Build A Cloud Computing System With Nimbus
Nimbus: Open Source Infrastructure-as-a-Service Cloud Computing Software Workshop on adapting applications and computing services to multi-core and virtualization CERN, June 2009 Kate Keahey keahey@mcs.anl.gov
More informationEnabling Technologies for Distributed Computing
Enabling Technologies for Distributed Computing Dr. Sanjay P. Ahuja, Ph.D. Fidelity National Financial Distinguished Professor of CIS School of Computing, UNF Multi-core CPUs and Multithreading Technologies
More informationCONDOR CLUSTERS ON EC2
CONDOR CLUSTERS ON EC2 Val Hendrix, Roberto A. Vitillo Lawrence Berkeley National Lab ATLAS Cloud Computing R & D 1 INTRODUCTION This is our initial work on investigating tools for managing clusters and
More informationCS312 Solutions #6. March 13, 2015
CS312 Solutions #6 March 13, 2015 Solutions 1. (1pt) Define in detail what a load balancer is and what problem it s trying to solve. Give at least two examples of where using a load balancer might be useful,
More informationCluster, Grid, Cloud Concepts
Cluster, Grid, Cloud Concepts Kalaiselvan.K Contents Section 1: Cluster Section 2: Grid Section 3: Cloud Cluster An Overview Need for a Cluster Cluster categorizations A computer cluster is a group of
More informationOGF25/EGEE User Forum Catania, Italy 2 March 2009
OGF25/EGEE User Forum Catania, Italy 2 March 2009 Constantino Vázquez Blanco Javier Fontán Muiños Raúl Sampedro Distributed Systems Architecture Research Group Universidad Complutense de Madrid 1/31 Outline
More informationEmpowering Cloud with latest VMDIRAC improvements
Empowering Cloud with latest VMDIRAC improvements DIRAC User Workshop, Ferrara, 2015 Víctor Méndez Muñoz vmendez@uab.es Agenda A single context to drive them all Cloud flexibility: Hybrid and multi-layered
More informationA Web-based Portal to Access and Manage WNoDeS Virtualized Cloud Resources
A Web-based Portal to Access and Manage WNoDeS Virtualized Cloud Resources Davide Salomoni 1, Daniele Andreotti 1, Luca Cestari 2, Guido Potena 2, Peter Solagna 3 1 INFN-CNAF, Bologna, Italy 2 University
More informationUsing Apache VCL and OpenStack to provide a Virtual Computing Lab
Using Apache VCL and OpenStack to provide a Virtual Computing Lab Curtis Collicutt and Cameron Mann Cybera 2-59C Computing Science Centre University of Alberta Edmonton, Alberta, T6G 2E8 {curtis.collicutt,
More informationDeploying complex applications to Google Cloud. Olia Kerzhner olia@google.com
Deploying complex applications to Google Cloud Olia Kerzhner olia@google.com Cloud VMs Networks Databases Object Stores Firewalls Disks LoadBalancers Control..? Application stacks are complex Storage External
More informationCUDA in the Cloud Enabling HPC Workloads in OpenStack With special thanks to Andrew Younge (Indiana Univ.) and Massimo Bernaschi (IAC-CNR)
CUDA in the Cloud Enabling HPC Workloads in OpenStack John Paul Walters Computer Scien5st, USC Informa5on Sciences Ins5tute jwalters@isi.edu With special thanks to Andrew Younge (Indiana Univ.) and Massimo
More informationExperience in integrating enduser cloud storage for CMS Analysis
Experience in integrating enduser cloud storage for CMS Analysis Hassen Riahi CERN IT CS3 Workshop Zürich, 18 th January 2016 1/18/16 H. Riahi; CS3 workshop 2 Outline Overview Goals Integration architecture
More informationStACC: St Andrews Cloud Computing Co laboratory. A Performance Comparison of Clouds. Amazon EC2 and Ubuntu Enterprise Cloud
StACC: St Andrews Cloud Computing Co laboratory A Performance Comparison of Clouds Amazon EC2 and Ubuntu Enterprise Cloud Jonathan S Ward StACC (pronounced like 'stack') is a research collaboration launched
More informationDynamic Squid Web Cache Monitoring
University of Victoria Faculty of Software Engineering Dynamic Squid Web Cache Monitoring Department of Physics University of Victoria Victoria, BC Robert Prior V00699787 Work Term 4 Software Engineering
More informationRED HAT ENTERPRISE VIRTUALIZATION
Giuseppe Paterno' Solution Architect Jan 2010 Red Hat Milestones October 1994 Red Hat Linux June 2004 Red Hat Global File System August 2005 Red Hat Certificate System & Dir. Server April 2006 JBoss April
More informationFleSSR Project: Installing Eucalyptus Open Source Cloud Solution at Oxford e- Research Centre
FleSSR Project: Installing Eucalyptus Open Source Cloud Solution at Oxford e- Research Centre Matteo Turilli, David Wallom Eucalyptus is available in two versions: open source and enterprise. Within this
More informationKVM, OpenStack, and the Open Cloud
KVM, OpenStack, and the Open Cloud Adam Jollans, IBM Southern California Linux Expo February 2015 1 Agenda A Brief History of VirtualizaJon KVM Architecture OpenStack Architecture KVM and OpenStack Case
More informationSoftware Testing and Deployment Using Virtualization and Cloud
Software Testing and Deployment Using Virtualization and Cloud Presented by: Omer Khalid Contributors: Arsalaan Shaikh, Brice Copy 1 Outline Part I - Background Use cases, Infrastructure Part II - Concepts
More informationComparing Open Source Private Cloud (IaaS) Platforms
Comparing Open Source Private Cloud (IaaS) Platforms Lance Albertson OSU Open Source Lab Associate Director of Operations lance@osuosl.org / @ramereth About me OSU Open Source Lab Server hosting for Open
More informationCloud Computing. Lecture 5 Grid Case Studies 2014-2015
Cloud Computing Lecture 5 Grid Case Studies 2014-2015 Up until now Introduction. Definition of Cloud Computing. Grid Computing: Schedulers Globus Toolkit Summary Grid Case Studies: Monitoring: TeraGRID
More informationCloud computing: the state of the art and challenges. Jānis Kampars Riga Technical University
Cloud computing: the state of the art and challenges Jānis Kampars Riga Technical University Presentation structure Enabling technologies Cloud computing defined Dealing with load in cloud computing Service
More informationScientific Computing Data Management Visions
Scientific Computing Data Management Visions ELI-Tango Workshop Szeged, 24-25 February 2015 Péter Szász Group Leader Scientific Computing Group ELI-ALPS Scientific Computing Group Responsibilities Data
More informationIntegration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst
Integration of Virtualized Worker Nodes in Standard-Batch-Systems CHEP 2009 Prague Oliver Oberst Outline General Description of Virtualization / Virtualization Solutions Shared HPC Infrastructure Virtualization
More informationUse of Hadoop File System for Nuclear Physics Analyses in STAR
1 Use of Hadoop File System for Nuclear Physics Analyses in STAR EVAN SANGALINE UC DAVIS Motivations 2 Data storage a key component of analysis requirements Transmission and storage across diverse resources
More information