Clusters in the Cloud

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Clusters in the Cloud"

Transcription

1 Clusters in the Cloud Dr. Paul Coddington, Deputy Director Dr. Shunde Zhang, Compu:ng Specialist eresearch SA October 2014

2 Use Cases Make the cloud easier to use for compute jobs Par:cularly for users familiar with HPC clusters Personal, on- demand cluster in the cloud Cluster in the cloud A private cluster only available to a research group A shared Node- managed cluster Preferably dynamic/elas:c Cloudburs:ng for HPC Dynamically (and transparently) add extra compute nodes from cloud to an exis:ng HPC cluster

3 Cluster infrastructure Configura:on Management System Monitoring Shared File System Applica:on Distribu:on SoWware Layer Local Resource Management System / Queueing system Hardware, Network

4 Tradi:onal Sta:c Cluster Hardware/Network Dedicated hardware Long process to get new hardware Sta:c not elas:c SoWware Assumes a fairly sta:c environment (IPs etc) Not cloud- friendly Some systems need restart if cluster is changed Not adaptable to changes

5 Cluster in the cloud Hardware / Network Provisioned by the cloud (OpenStack) Get new resources in minutes Remove resources in minutes Elas:c/scalable on demand SoWware Dynamic Can easily add/remove nodes as needed

6 Possible solu:ons Condor for high- throughput compu:ng Cloud Scheduler working for a CERN LCG node Recent versions of Condor support cloud execu:on Torque/PBS sta:c cluster in cloud Works, but painful to set up and maintain Dynamic Torque/PBS cluster No exis:ng dynamic/elas:c solu:on StarCluster for personal cluster Automate setup of VMs in cloud, including cluster Can add/subtract worker nodes manually Only Amazon, SGE and Condor but not PBS/Torque

7 Our work Condor for high- throughput compu:ng Cloud Scheduler for Australian CERN LCG node Torque/PBS sta:c cluster in cloud Set up large cluster in cloud for CoEPP Scripts to automate setup and monitoring Dynamic Torque/PBS cluster Created Dynamic Torque system for OpenStack StarCluster for personal cluster Ported to OpenStack and added Torque plugin Add- ons to make it easier to use for ersa users

8 Applica:on sowware Want familiar HPC applica:ons to be available to cloud VMs And we don t want to install and maintain sowware twice, in HPC and cloud But limit on size of VM images in the cloud Want to avoid making lots of custom images We use CVMFS Read- only distributed file system, hbp based Used by CERN LHC Grid for distribu:ng sowware One VM image, with CVMFS client Downloads and caches sowware from HPC cluster

9 HTC Cluster in the Cloud NeCTAR eresearch Tools project for high- throughput compu:ng in the cloud ARC Centre of Excellence in Experimental Par:cle Physics (CoEPP) Needed a large cluster for CERN ATLAS data analysis and simula:on Tier 2 (global) and Tier 3 (local) jobs Augment exis:ng small physical clusters at mul:ple sites running Torque

10 CERN ATLAS experiment

11 CERN ATLAS experiment

12 Sta:c Cluster in the Cloud Built a large Torque cluster using cloud VMs A challenging exercise! Reliability issues, needed a lot of scripts to automate setup, monitoring, recovery, etc Some types of usage are bursty but cluster resources were sta:c Didn't take advantage of elas:city of cloud

13 Dynamic Torque Sta:c/dynamic worker nodes Sta:c: stays up all the :me Dynamic: up and down according to workload Independent of Torque/MAUI Runs as a separate process Only add/remove worker nodes Query Torque and MAUI scheduler periodically S:ll up to MAUI scheduler to decide where to run a job

14 Dynamic Torque

15 Dynamic Torque for CoEPP Ganglia Nagios CVMFS Torque/MAUI LDAP and Dynamic NFS Puppet Torque Worker nodes in SA Interac:ve nodes in Melbourne Worker nodes in Monash Worker nodes in Melbourne

16 Dynamic Torque for CoEPP

17 CoEPP Outcomes Three large clusters in use for over a year Hundreds of cores in each Condor and CloudScheduler for ATLAS Tier 2 Dynamic Torque for ATLAS Tier 3 and Belle LHC ATLAS experiment at CERN 530,000 Tier 2 jobs 325,000 CPU hours for Tier 3 jobs Belle experiment in Japan 150,000 jobs

18 Private Clusters Good for building a shared cluster for a large research group with good IT support who can set up and manage a Torque cluster What about the many individual researchers or small groups who also want a private cluster using their cloud alloca:on? But have no dedicated IT staff and very basic Unix skills? Is there a simple DIY solu:on?

19 StarCluster Generic setup Create security group for the cluster Launch VMs (master, node01, node02 ) Set up public key for password- less SSH Install NFS on master and share scratch space to all nodexx Can use EBS (Cinder) volumes as scratch space Queuing system setup (plugins) Condor, SGE, Hadoop and your own plugin!

20 StarCluster for OpenStack ersa App Repository (CVMFS server) CVMFS proxy Worker Node (Torque MOM) Worker Node (Torque MOM) OpenStack EC2 API Head Node (NFS, Torque server, MAUI) Worker Node (Torque MOM) Volume StarCluster

21 StarCluster - configura:on Availability zone Image (op:onal) Image for master Flavor (op:onal) Flavor for master Number of nodes Volume Username User ID Group ID User shell plugins

22 Start a cluster with StarCluster # fire up a new cluster (from your desktop) $ starcluster start mycluster # log in to the head node (master) to submit jobs $ starcluster sshmaster mycluster # Copy files $ starcluster put /path/to/local/file/or/dir /remote/path/ $ starcluster get /path/to/remote/file/or/dir /local/path/ # Add a compute node to the cluster $ starcluster addnode n 2 mycluster # terminate it awer use $ starcluster terminate mycluster

23 Other op:ons for Personal Cluster Elas:cluster Python code to provision VMs Ansible to configure them Ansible playbooks for Torque/SGE/, NFS/pvfs/ Heat Everything in HOT template Earlier versions had limita:ons that made it hard to implement everything May revisit in future

24 Private Cluster in the Cloud Can use your personal or project cloud alloca:on to start up your own personal cluster in the cloud No need to share! Except among your group. Can use the standard PBS/Torque queueing system to submit jobs (or not) Only your jobs in the queue But you have to set up and manage the cluster Straighqorward if you have good Unix skills (unless things go wrong ) Several groups now using this But ersa doing support when things go wrong

25 Emu Cluster in the Cloud Emu is an ersa cluster that runs in the cloud Aimed to be like an old cluster (Corvus) 8- core compute nodes But a bit different Dynamically created VMs in the cloud Can have private compute nodes Different size compute nodes if you want

26 Emu ersa- managed dynamic cluster in the cloud Shared by mul:ple cloud tenants and ersa users All nodes in SA zone ersa cloud alloca:on contributes 128 cores Users can bring in their own cloud alloca:on to launch their worker nodes in Emu Users don t need to build and look awer their own personal cluster It can also mount users Cinder volume storage to their own worker nodes via NFS Set up so researchers use their ersa accounts

27 Using your own cloud alloca:on Users add our sysadmins to their tenant So we can launch VMs on their behalf Will look at Trusts in Icehouse Add some configs to Dynamic Torque Number of sta:c/dynamic nodes, size of nodes, etc Add a group of user accounts allowed to use it Create a reserva:on for users worker nodes in MAUI A special account string needs to be put in the job To match users jobs to their group s reserved nodes A qsub filter to check if the account string is valid You can t submit a job using another group s alloca:on

28 Emu Sensu CVMFS Torque/MAUI LDAP and Dynamic NFS Salt Torque NFS Worker nodes of Tenant1 Shared worker nodes (ersa donated) Worker nodes of Tenant2

29 Sta:c cluster vs dynamic cluster Sta$c Cluster Dynamic Cluster Hardware Physical Machines Virtual Machines LRMS Torque Torque with Dynamic Torque CMS Puppet Salt Stack Monitoring Nagios, Ganglia Sensu, Graphite, Logstash App Distribu:on NFS Mount CVMFS Shared FS NFS NFS

30 Future Work Beber repor:ng and usage graphs More monitoring checks Queueing system Mul:- node jobs don t work because a new node is not trusted by exis:ng nodes Trust list is only updated when PBS server is started Could hack Torque source code Or maybe use SLURM or SGE Beber way to share user creden:als Trusts in Icehouse? Na:onal and/or other regional services

31 Future Work Spot instance queue Distributed file system NFS is the sta:c component Cannot add storage to NFS without stopping it Cannot add new nodes to the allow list dynamically needs to use iptables; update iptables instead Inves:ga:on of a dynamic and distributed FS One FS for all tenants Alterna:ves to StarCluster Heat or Elas:cluster

32 Resources Cloud Scheduler hbp://cloudscheduler.org/ Star cluster hbp://star.mit.edu.au/cluster OpenStack version hbps://github.com/shundezhang/starcluster/ Dynamic Torque hbps://github.com/shundezhang/dynamictorque

33

34 Nagios vs Sensu Nagios First designed in the last century for sta:c environment Needs to update local configura:on and restart service if a remote server is added or removed Server perform all checks and it is not scalable Sensu Modern design with AMQP as communica:on layer Local agent runs checks Weak coupling between clients and server and it is scalable

35 Imagefactory Template in XML Packages Commands Files Can backup an image in github Automa:c, no user interac:on required

36 EMU Monitoring Sensu Run health checks Logstash Collect PBS server/mom/accoun:ng logs Collectd Collect metrics of CPU, memory, disk, network etc

37 Salt Stack vs Puppet Salt Stack Puppet Architecture Server- Client Server- Client Working Modal Push Pull Communica:on Zeromq + msgpack HTTP + text Language Python Ruby Remote execu:on Yes No

CONDOR CLUSTERS ON EC2

CONDOR CLUSTERS ON EC2 CONDOR CLUSTERS ON EC2 Val Hendrix, Roberto A. Vitillo Lawrence Berkeley National Lab ATLAS Cloud Computing R & D 1 INTRODUCTION This is our initial work on investigating tools for managing clusters and

More information

Data Center Evolu.on and the Cloud. Paul A. Strassmann George Mason University November 5, 2008, 7:20 to 10:00 PM

Data Center Evolu.on and the Cloud. Paul A. Strassmann George Mason University November 5, 2008, 7:20 to 10:00 PM Data Center Evolu.on and the Cloud Paul A. Strassmann George Mason University November 5, 2008, 7:20 to 10:00 PM 1 Hardware Evolu.on 2 Where is hardware going? x86 con(nues to move upstream Massive compute

More information

Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012

Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Dynamic Extension of a Virtualized Cluster by using Cloud Resources CHEP 2012 Thomas Hauth,, Günter Quast IEKP KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz

More information

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb Batch and Cloud overview Andrew McNab University of Manchester GridPP and LHCb Overview Assumptions Batch systems The Grid Pilot Frameworks DIRAC Virtual Machines Vac Vcycle Tier-2 Evolution Containers

More information

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH CONTENTS Introduction... 4 System Components... 4 OpenNebula Cloud Management Toolkit... 4 VMware

More information

Assignment # 1 (Cloud Computing Security)

Assignment # 1 (Cloud Computing Security) Assignment # 1 (Cloud Computing Security) Group Members: Abdullah Abid Zeeshan Qaiser M. Umar Hayat Table of Contents Windows Azure Introduction... 4 Windows Azure Services... 4 1. Compute... 4 a) Virtual

More information

Deployment of Private, Hybrid & Public Clouds with OpenNebula

Deployment of Private, Hybrid & Public Clouds with OpenNebula EL / LAK (FOSS) 2010 May 14th, 2010 Deployment of Private, Hybrid & Public Clouds with OpenNebula University Complutense of Madrid The Anatomy of an IaaS Cloud Deployment of Private, Hybrid & Public Clouds

More information

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases NASA Ames NASA Advanced Supercomputing (NAS) Division California, May 24th, 2012 Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases Ignacio M. Llorente Project Director OpenNebula Project.

More information

Introduc)on of Pla/orm ISF. Weina Ma Weina.Ma@uoit.ca

Introduc)on of Pla/orm ISF. Weina Ma Weina.Ma@uoit.ca Introduc)on of Pla/orm ISF Weina Ma Weina.Ma@uoit.ca Agenda Pla/orm ISF Product Overview Pla/orm ISF Concepts & Terminologies Self- Service Applica)on Management Applica)on Example Deployment Examples

More information

Work Environment. David Tur HPC Expert. HPC Users Training September, 18th 2015

Work Environment. David Tur HPC Expert. HPC Users Training September, 18th 2015 Work Environment David Tur HPC Expert HPC Users Training September, 18th 2015 1. Atlas Cluster: Accessing and using resources 2. Software Overview 3. Job Scheduler 1. Accessing Resources DIPC technicians

More information

Hunk & Elas=c MapReduce: Big Data Analy=cs on AWS

Hunk & Elas=c MapReduce: Big Data Analy=cs on AWS Copyright 2014 Splunk Inc. Hunk & Elas=c MapReduce: Big Data Analy=cs on AWS Dritan Bi=ncka BD Solu=ons Architecture Disclaimer During the course of this presenta=on, we may make forward looking statements

More information

Chapter 3. Database Architectures and the Web Transparencies

Chapter 3. Database Architectures and the Web Transparencies Week 2: Chapter 3 Chapter 3 Database Architectures and the Web Transparencies Database Environment - Objec

More information

Return on Experience on Cloud Compu2ng Issues a stairway to clouds. Experts Workshop Nov. 21st, 2013

Return on Experience on Cloud Compu2ng Issues a stairway to clouds. Experts Workshop Nov. 21st, 2013 Return on Experience on Cloud Compu2ng Issues a stairway to clouds Experts Workshop Agenda InGeoCloudS SoCware Stack InGeoCloudS Elas2city and Scalability Elas2c File Server Elas2c Database Server Elas2c

More information

CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment

CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment CernVM Online and Cloud Gateway a uniform interface for CernVM contextualization and deployment George Lestaris - Ioannis Charalampidis D. Berzano, J. Blomer, P. Buncic, G. Ganis and R. Meusel PH-SFT /

More information

TECHNOLOGY WHITE PAPER Jun 2012

TECHNOLOGY WHITE PAPER Jun 2012 TECHNOLOGY WHITE PAPER Jun 2012 Technology Stack C# Windows Server 2008 PHP Amazon Web Services (AWS) Route 53 Elastic Load Balancing (ELB) Elastic Compute Cloud (EC2) Amazon RDS Amazon S3 Elasticache

More information

Savanna Hadoop on. OpenStack. Savanna Technical Lead

Savanna Hadoop on. OpenStack. Savanna Technical Lead Savanna Hadoop on OpenStack Sergey Lukjanov Savanna Technical Lead Mirantis, 2013 Agenda Savanna Overview Savanna Use Cases Roadmap & Current Status Architecture & Features Overview Hadoop vs. Virtualization

More information

High Performance Computing OpenStack Options. September 22, 2015

High Performance Computing OpenStack Options. September 22, 2015 High Performance Computing OpenStack PRESENTATION TITLE GOES HERE Options September 22, 2015 Today s Presenters Glyn Bowden, SNIA Cloud Storage Initiative Board HP Helion Professional Services Alex McDonald,

More information

Last time. Today. IaaS Providers. Amazon Web Services, overview

Last time. Today. IaaS Providers. Amazon Web Services, overview Last time General overview, motivation, expected outcomes, other formalities, etc. Please register for course Online (if possible), or talk to Yvonne@CS Course evaluation forgotten Please assign one volunteer

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details Use cases By scientist By HEP experiment System requirements and installation How to get it? 2 What

More information

19.10.11. Amazon Elastic Beanstalk

19.10.11. Amazon Elastic Beanstalk 19.10.11 Amazon Elastic Beanstalk A Short History of AWS Amazon started as an ECommerce startup Original architecture was restructured to be more scalable and easier to maintain Competitive pressure for

More information

Infrastructure Clouds for Science and Education: Platform Tools

Infrastructure Clouds for Science and Education: Platform Tools Infrastructure Clouds for Science and Education: Platform Tools Kate Keahey, Renato J. Figueiredo, John Bresnahan, Mike Wilde, David LaBissoniere Argonne National Laboratory Computation Institute, University

More information

Secure Hybrid Cloud Infrastructure for Scien5fic Applica5ons

Secure Hybrid Cloud Infrastructure for Scien5fic Applica5ons Secure Hybrid Cloud Infrastructure for Scien5fic Applica5ons Project Members: Paula Eerola Miika Komu MaA Kortelainen Tomas Lindén Lirim Osmani Sasu Tarkoma Salman Toor (Presenter) salman.toor@helsinki.fi

More information

Comparing Ganeti to other Private Cloud Platforms. Lance Albertson Director lance@osuosl.org @ramereth

Comparing Ganeti to other Private Cloud Platforms. Lance Albertson Director lance@osuosl.org @ramereth Comparing Ganeti to other Private Cloud Platforms Lance Albertson Director lance@osuosl.org @ramereth About me OSU Open Source Lab Server hosting for Open Source Projects Open Source development projects

More information

Cloud Based Tes,ng & Capacity Planning (CloudPerf)

Cloud Based Tes,ng & Capacity Planning (CloudPerf) Cloud Based Tes,ng & Capacity Planning (CloudPerf) Joan A. Smith Emory University Libraries joan.smith@emory.edu Frank Owen Owenworks Inc. frank@owenworks.biz Full presenta,on materials and CloudPerf screencast

More information

Experiments on cost/power and failure aware scheduling for clouds and grids

Experiments on cost/power and failure aware scheduling for clouds and grids Experiments on cost/power and failure aware scheduling for clouds and grids Jorge G. Barbosa, Al0no M. Sampaio, Hamid Harabnejad Universidade do Porto, Faculdade de Engenharia, LIACC Porto, Portugal, jbarbosa@fe.up.pt

More information

Scyld Cloud Manager User Guide

Scyld Cloud Manager User Guide Scyld Cloud Manager User Guide Preface This guide describes how to use the Scyld Cloud Manager (SCM) web portal application. Contacting Penguin Computing 45800 Northport Loop West Fremont, CA 94538 1-888-PENGUIN

More information

Monitoring the Grid at local, national, and global levels

Monitoring the Grid at local, national, and global levels Home Search Collections Journals About Contact us My IOPscience Monitoring the Grid at local, national, and global levels This content has been downloaded from IOPscience. Please scroll down to see the

More information

Putchong Uthayopas, Kasetsart University

Putchong Uthayopas, Kasetsart University Putchong Uthayopas, Kasetsart University Introduction Cloud Computing Explained Cloud Application and Services Moving to the Cloud Trends and Technology Legend: Cluster computing, Grid computing, Cloud

More information

Open Source Cloud Computing Management with OpenNebula

Open Source Cloud Computing Management with OpenNebula CloudCamp Campus Party July 2011, Valencia Open Source Cloud Computing Management with OpenNebula Javier Fontán Muiños dsa-research.org Distributed Systems Architecture Research Group Universidad Complutense

More information

Managing Traditional Workloads Together with Cloud Computing Workloads

Managing Traditional Workloads Together with Cloud Computing Workloads Managing Traditional Workloads Together with Cloud Computing Workloads Table of Contents Introduction... 3 Cloud Management Challenges... 3 Re-thinking of Cloud Management Solution... 4 Teraproc Cloud

More information

CERN Cloud Architecture

CERN Cloud Architecture CERN Cloud Architecture Belmiro Moreira belmiro.moreira@cern.ch @belmiromoreira Geneva, July 16 th, 2015 2 CERN Cloud Architecture Overview HAProxy Nova Horizon Glance Keystone Heat Cinder Ceilometer CERN

More information

Building Success on Acquia Cloud:

Building Success on Acquia Cloud: Building Success on Acquia Cloud: 10 Layers of PaaS TECHNICAL Guide Table of Contents Executive Summary.... 3 Introducing the 10 Layers of PaaS... 4 The Foundation: Five Layers of PaaS Infrastructure...

More information

Managing your Red Hat Enterprise Linux guests with RHN Satellite

Managing your Red Hat Enterprise Linux guests with RHN Satellite Managing your Red Hat Enterprise Linux guests with RHN Satellite Matthew Davis, Level 1 Production Support Manager, Red Hat Brad Hinson, Sr. Support Engineer Lead System z, Red Hat Mark Spencer, Sr. Solutions

More information

STeP-IN SUMMIT 2013. June 18 21, 2013 at Bangalore, INDIA. Performance Testing of an IAAS Cloud Software (A CloudStack Use Case)

STeP-IN SUMMIT 2013. June 18 21, 2013 at Bangalore, INDIA. Performance Testing of an IAAS Cloud Software (A CloudStack Use Case) 10 th International Conference on Software Testing June 18 21, 2013 at Bangalore, INDIA by Sowmya Krishnan, Senior Software QA Engineer, Citrix Copyright: STeP-IN Forum and Quality Solutions for Information

More information

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan Data Management in the Cloud: Limitations and Opportunities Annies Ductan Discussion Outline: Introduc)on Overview Vision of Cloud Compu8ng Managing Data in The Cloud Cloud Characteris8cs Data Management

More information

HTCondor at the RAL Tier-1

HTCondor at the RAL Tier-1 HTCondor at the RAL Tier-1 Andrew Lahiff, Alastair Dewhurst, John Kelly, Ian Collier, James Adams STFC Rutherford Appleton Laboratory HTCondor Week 2014 Outline Overview of HTCondor at RAL Monitoring Multi-core

More information

Adding IaaS Clouds to the ATLAS Computing Grid

Adding IaaS Clouds to the ATLAS Computing Grid Adding IaaS Clouds to the ATLAS Computing Grid Ashok Agarwal, Frank Berghaus, Andre Charbonneau, Mike Chester, Asoka de Silva, Ian Gable, Joanna Huang, Colin Leavett-Brown, Michael Paterson, Randall Sobie,

More information

Cloud Computing #8 - Datacenter OS. Johan Eker

Cloud Computing #8 - Datacenter OS. Johan Eker Cloud Computing #8 - Datacenter OS Johan Eker Outline What is a Datacenter OS? OpenStack Kubernetes Resource Management What is an OS? What is an OS? Manage hardware resources such as CPU, RAM, disk, I/O,

More information

Understanding ArcGIS in Virtualization and Cloud Environments

Understanding ArcGIS in Virtualization and Cloud Environments Esri Middle East and Africa User Conference December 10 12 Abu Dhabi, UAE Understanding ArcGIS in Virtualization and Cloud Environments Marwa Mabrouk Powerful GIS capabilities Delivered as Web services

More information

Comparing Open Source Private Cloud (IaaS) Platforms

Comparing Open Source Private Cloud (IaaS) Platforms Comparing Open Source Private Cloud (IaaS) Platforms Lance Albertson OSU Open Source Lab Associate Director of Operations lance@osuosl.org / @ramereth About me OSU Open Source Lab Server hosting for Open

More information

Design considera-ons and Guiding Principles for Implemen-ng Cloud Security. William Stearns Security Analyst CloudPassage

Design considera-ons and Guiding Principles for Implemen-ng Cloud Security. William Stearns Security Analyst CloudPassage Design considera-ons and Guiding Principles for Implemen-ng Cloud Security William Stearns Security Analyst CloudPassage In a nutshell How do Cloud Servers differ from Data Center Servers? How do the differences

More information

OpenStack Introduction. November 4, 2015

OpenStack Introduction. November 4, 2015 OpenStack Introduction November 4, 2015 Application Platforms Undergoing A Major Shift What is OpenStack Open Source Cloud Software Launched by NASA and Rackspace in 2010 Massively scalable Managed by

More information

Scalable Architecture on Amazon AWS Cloud

Scalable Architecture on Amazon AWS Cloud Scalable Architecture on Amazon AWS Cloud Kalpak Shah Founder & CEO, Clogeny Technologies kalpak@clogeny.com 1 * http://www.rightscale.com/products/cloud-computing-uses/scalable-website.php 2 Architect

More information

Multi Provider Cloud. Srinivasa Acharya, Engineering Manager, Hewlett-Packard rsacharya@hp.com

Multi Provider Cloud. Srinivasa Acharya, Engineering Manager, Hewlett-Packard rsacharya@hp.com Multi Provider Cloud Srinivasa Acharya, Engineering Manager, Hewlett-Packard rsacharya@hp.com Agenda Introduction to OpenStack Multi Hypervisor Architecture Use cases for Multi Hypervisor cloud Ironic

More information

Lustre Monitoring with OpenTSDB

Lustre Monitoring with OpenTSDB Lustre Monitoring with OpenTSDB 2015/9/22 DataDirect Networks Japan, Inc. Shuichi Ihara 2 Lustre Monitoring Background Lustre is a black box Users and Administrators want to know what s going on? Find

More information

How the ersa Problem became the ersa Solu3on. Why a network and network security is impera3ve for ersa s NeCTAR cloud. Paul Bartczak Infrastructure

How the ersa Problem became the ersa Solu3on. Why a network and network security is impera3ve for ersa s NeCTAR cloud. Paul Bartczak Infrastructure How the ersa Problem became the ersa Solu3on. Why a network and network security is impera3ve for ersa s NeCTAR cloud. Paul Bartczak Infrastructure Manager About ersa eresearch SA is a collabora3ve joint

More information

Develop a process for applying updates to systems, including verifying properties of the update. Create File Systems

Develop a process for applying updates to systems, including verifying properties of the update. Create File Systems RH413 Manage Software Updates Develop a process for applying updates to systems, including verifying properties of the update. Create File Systems Allocate an advanced file system layout, and use file

More information

HPCC Monitoring and Reporting (Technical Preview) Boca Raton Documentation Team

HPCC Monitoring and Reporting (Technical Preview) Boca Raton Documentation Team HPCC Monitoring and Reporting (Technical Preview) Boca Raton Documentation Team HPCC Monitoring and Reporting (Technical Preview) Boca Raton Documentation Team Copyright 2015 HPCC Systems. All rights reserved

More information

Shoal: IaaS Cloud Cache Publisher

Shoal: IaaS Cloud Cache Publisher University of Victoria Faculty of Engineering Winter 2013 Work Term Report Shoal: IaaS Cloud Cache Publisher Department of Physics University of Victoria Victoria, BC Mike Chester V00711672 Work Term 3

More information

Kaseya Fundamentals Workshop DAY THREE. Developed by Kaseya University. Powered by IT Scholars

Kaseya Fundamentals Workshop DAY THREE. Developed by Kaseya University. Powered by IT Scholars Kaseya Fundamentals Workshop DAY THREE Developed by Kaseya University Powered by IT Scholars Kaseya Version 6.5 Last updated March, 2014 Day Two Overview Day Two Lab Review Patch Management Configura;on

More information

Chapter 2: Getting Started

Chapter 2: Getting Started Chapter 2: Getting Started Once Partek Flow is installed, Chapter 2 will take the user to the next stage and describes the user interface and, of note, defines a number of terms required to understand

More information

U-LITE: a proposal for scientific computing at LNGS. S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011

U-LITE: a proposal for scientific computing at LNGS. S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011 U-LITE: a proposal for scientific computing at LNGS S. Parlati, P. Spinnato, S. Stalio LNGS 13 Sep. 2011 20 years of Scientific Computing at LNGS Early 90s: highly centralized structure based on VMS cluster

More information

Making System z Sexy Again with Social and Collaboration Software

Making System z Sexy Again with Social and Collaboration Software Making System z Sexy Again with Social and Collaboration Software Tuesday, August 9, 2011: 4:30 PM-5:30 PM Paul Houde Client Technical Professional for Collaboration Solutions on System z Agenda Importance

More information

CloudCenter Full Lifecycle Management. An application-defined approach to deploying and managing applications in any datacenter or cloud environment

CloudCenter Full Lifecycle Management. An application-defined approach to deploying and managing applications in any datacenter or cloud environment CloudCenter Full Lifecycle Management An application-defined approach to deploying and managing applications in any datacenter or cloud environment CloudCenter Full Lifecycle Management Page 2 Table of

More information

Automating Big Data Benchmarking for Different Architectures with ALOJA

Automating Big Data Benchmarking for Different Architectures with ALOJA www.bsc.es Jan 2016 Automating Big Data Benchmarking for Different Architectures with ALOJA Nicolas Poggi, Postdoc Researcher Agenda 1. Intro on Hadoop performance 1. Current scenario and problematic 2.

More information

Integration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept

Integration of Virtualized Workernodes in Batch Queueing Systems The ViBatch Concept Integration of Virtualized Workernodes in Batch Queueing Systems, Dr. Armin Scheurer, Oliver Oberst, Prof. Günter Quast INSTITUT FÜR EXPERIMENTELLE KERNPHYSIK FAKULTÄT FÜR PHYSIK KIT University of the

More information

Mark Bennett. Search and the Virtual Machine

Mark Bennett. Search and the Virtual Machine Mark Bennett Search and the Virtual Machine Agenda Intro / Business Drivers What to do with Search + Virtual What Makes Search Fast (or Slow!) Virtual Platforms Test Results Trends / Wrap Up / Q & A Business

More information

Cloudera Enterprise Reference Architecture for Google Cloud Platform Deployments

Cloudera Enterprise Reference Architecture for Google Cloud Platform Deployments Cloudera Enterprise Reference Architecture for Google Cloud Platform Deployments Important Notice 2010-2015 Cloudera, Inc. All rights reserved. Cloudera, the Cloudera logo, Cloudera Impala, Impala, and

More information

OpenNebula Open Souce Solution for DC Virtualization. C12G Labs. Online Webinar

OpenNebula Open Souce Solution for DC Virtualization. C12G Labs. Online Webinar OpenNebula Open Souce Solution for DC Virtualization C12G Labs Online Webinar What is OpenNebula? Multi-tenancy, Elasticity and Automatic Provision on Virtualized Environments I m using virtualization/cloud,

More information

w w w. u l t i m u m t e c h n o l o g i e s. c o m Infrastructure-as-a-Service on the OpenStack platform

w w w. u l t i m u m t e c h n o l o g i e s. c o m Infrastructure-as-a-Service on the OpenStack platform w w w. u l t i m u m t e c h n o l o g i e s. c o m Infrastructure-as-a-Service on the OpenStack platform http://www.ulticloud.com http://www.openstack.org Introduction to OpenStack 1. What OpenStack is

More information

Deploying Your Application On Public Cloud

Deploying Your Application On Public Cloud #GHC14 Deploying Your Application On Public Cloud Egle Sigler @eglute Iccha Sethi @IcchaSethi October 9, Egle Sigler Principal Architect at Rackspace Works with OpenStack POWER: Professional Organization

More information

Automated Configuration of Open Stack Instances at Boot Time

Automated Configuration of Open Stack Instances at Boot Time Automated Configuration of Open Stack Instances at Boot Time N Praveen 1, Dr. M.N.Jayaram 2 Post Graduate Student 1, Associate Professor 2, EC Department, SJCE, Mysuru, India Abstract: Cloud Computing

More information

Amazon Web Services Primer. William Strickland COP 6938 Fall 2012 University of Central Florida

Amazon Web Services Primer. William Strickland COP 6938 Fall 2012 University of Central Florida Amazon Web Services Primer William Strickland COP 6938 Fall 2012 University of Central Florida AWS Overview Amazon Web Services (AWS) is a collection of varying remote computing provided by Amazon.com.

More information

The Evolution of Cloud Computing in ATLAS

The Evolution of Cloud Computing in ATLAS The Evolution of Cloud Computing in ATLAS Ryan Taylor on behalf of the ATLAS collaboration CHEP 2015 Evolution of Cloud Computing in ATLAS 1 Outline Cloud Usage and IaaS Resource Management Software Services

More information

Survey of the Operating Landscape Investigating Incidents in the Cloud

Survey of the Operating Landscape Investigating Incidents in the Cloud Survey of the Operating Landscape Investigating Incidents in the Cloud SESSION ID: CSV-T09 Paul A. Henry Security & Forensics Analyst vnet Security, LLC @phenrycissp Jacob Williams Chief Scientist CSRgroup

More information

Cloud services for the Fermilab scientific stakeholders

Cloud services for the Fermilab scientific stakeholders Cloud services for the Fermilab scientific stakeholders S Timm 1, G Garzoglio 1, P Mhashilkar 1*, J Boyd 1, G Bernabeu 1, N Sharma 1, N Peregonow 1, H Kim 1, S Noh 2,, S Palur 3, and I Raicu 3 1 Scientific

More information

Copyright 2014, Oracle and/or its affiliates. All rights reserved. 2

Copyright 2014, Oracle and/or its affiliates. All rights reserved. 2 2 CON8358 - Building a Private Cloud with OpenStack Oracle OpenStack for Oracle Linux Ronen Kofman Director of Product Development Oracle OpenStack September, 2014 Safe Harbor Statement The following is

More information

Of Pets and Cattle and Hearts

Of Pets and Cattle and Hearts Of Pets and Cattle and Hearts The SUSE Systems Management Story Joachim Werner Senior Product Manager, SUSE joe@suse.com Who am I? 2 SUSE and Linux Workloads in the Enterprise SUSE Studio Datacenter Private

More information

2) Xen Hypervisor 3) UEC

2) Xen Hypervisor 3) UEC 5. Implementation Implementation of the trust model requires first preparing a test bed. It is a cloud computing environment that is required as the first step towards the implementation. Various tools

More information

Elastic Management of Cluster based Services in the Cloud

Elastic Management of Cluster based Services in the Cloud First Workshop on Automated Control for Datacenters and Clouds (ACDC09) June 19th, Barcelona, Spain Elastic Management of Cluster based Services in the Cloud Rafael Moreno Vozmediano, Ruben S. Montero,

More information

SURFsara HPC Cloud Workshop

SURFsara HPC Cloud Workshop SURFsara HPC Cloud Workshop www.cloud.sara.nl Tutorial 2014-06-11 UvA HPC and Big Data Course June 2014 Anatoli Danezi, Markus van Dijk cloud-support@surfsara.nl Agenda Introduction and Overview (current

More information

Accelerate OpenStack* Together. * OpenStack is a registered trademark of the OpenStack Foundation

Accelerate OpenStack* Together. * OpenStack is a registered trademark of the OpenStack Foundation Accelerate OpenStack* Together * OpenStack is a registered trademark of the OpenStack Foundation Where are your workloads running Ensuring Boundary Control in OpenStack Cloud. Raghu Yeluri Principal Engineer,

More information

ArcGIS for Server in the Amazon Cloud. Michele Lundeen Esri

ArcGIS for Server in the Amazon Cloud. Michele Lundeen Esri ArcGIS for Server in the Amazon Cloud Michele Lundeen Esri What we will cover ArcGIS for Server in the Amazon Cloud Why How Extras Why do you need ArcGIS Server? Some examples Publish - Dynamic Map Services

More information

HDFS Cluster Installation Automation for TupleWare

HDFS Cluster Installation Automation for TupleWare HDFS Cluster Installation Automation for TupleWare Xinyi Lu Department of Computer Science Brown University Providence, RI 02912 xinyi_lu@brown.edu March 26, 2014 Abstract TupleWare[1] is a C++ Framework

More information

Manjrasoft Market Oriented Cloud Computing Platform

Manjrasoft Market Oriented Cloud Computing Platform Manjrasoft Market Oriented Cloud Computing Platform Aneka Aneka is a market oriented Cloud development and management platform with rapid application development and workload distribution capabilities.

More information

Behind the scene III Cloud computing

Behind the scene III Cloud computing Behind the scene III Cloud computing Athens, 15.11.2014 M. Dolenc / R. Klinc Why we do it? Engineering in the cloud is a combina3on of cloud based services and rich interac3ve applica3ons allowing engineers

More information

VMware vcloud Automation Center 6.0

VMware vcloud Automation Center 6.0 VMware 6.0 Reference Architecture TECHNICAL WHITE PAPER Table of Contents Overview... 4 Initial Deployment Recommendations... 4 General Recommendations... 4... 4 Load Balancer Considerations... 4 Database

More information

Neptune. A Domain Specific Language for Deploying HPC Software on Cloud Platforms. Chris Bunch Navraj Chohan Chandra Krintz Khawaja Shams

Neptune. A Domain Specific Language for Deploying HPC Software on Cloud Platforms. Chris Bunch Navraj Chohan Chandra Krintz Khawaja Shams Neptune A Domain Specific Language for Deploying HPC Software on Cloud Platforms Chris Bunch Navraj Chohan Chandra Krintz Khawaja Shams ScienceCloud 2011 @ San Jose, CA June 8, 2011 Cloud Computing Three

More information

Cloud computing - Architecting in the cloud

Cloud computing - Architecting in the cloud Cloud computing - Architecting in the cloud anna.ruokonen@tut.fi 1 Outline Cloud computing What is? Levels of cloud computing: IaaS, PaaS, SaaS Moving to the cloud? Architecting in the cloud Best practices

More information

TECHNOLOGY WHITE PAPER Jan 2016

TECHNOLOGY WHITE PAPER Jan 2016 TECHNOLOGY WHITE PAPER Jan 2016 Technology Stack C# PHP Amazon Web Services (AWS) Route 53 Elastic Load Balancing (ELB) Elastic Compute Cloud (EC2) Amazon RDS Amazon S3 Elasticache CloudWatch Paypal Overview

More information

The Pitfalls of Encrypted Networks in Banking Operations Compliance Success in two industry cases

The Pitfalls of Encrypted Networks in Banking Operations Compliance Success in two industry cases The Pitfalls of Encrypted Networks in Banking Operations Compliance Success in two industry cases Elba Horta Regional Sales Manager, Southern Europe SSH Communica1ons Security elba.horta@ssh.com ENABLE,

More information

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida Plug-and-play Virtual Appliance Clusters Running Hadoop Dr. Renato Figueiredo ACIS Lab - University of Florida Advanced Computing and Information Systems laboratory Introduction You have so far learned

More information

Chapter 1 - Web Server Management and Cluster Topology

Chapter 1 - Web Server Management and Cluster Topology Objectives At the end of this chapter, participants will be able to understand: Web server management options provided by Network Deployment Clustered Application Servers Cluster creation and management

More information

WHITE PAPER. Software Defined Storage Hydrates the Cloud

WHITE PAPER. Software Defined Storage Hydrates the Cloud WHITE PAPER Software Defined Storage Hydrates the Cloud Table of Contents Overview... 2 NexentaStor (Block & File Storage)... 4 Software Defined Data Centers (SDDC)... 5 OpenStack... 5 CloudStack... 6

More information

SGE & Amazon EC2. Chris Dagdigian 2008 OSGC Conference

SGE & Amazon EC2. Chris Dagdigian 2008 OSGC Conference SGE & Amazon EC2 Chris Dagdigian 2008 OSGC Conference Putting my $$ where my mouth is I m about to pay $.30 for the privilege of showing you this demo by firing up a 3-node cluster within EC2 Using cheap

More information

Load and Performance Load Testing. RadView Software October 2015 www.radview.com

Load and Performance Load Testing. RadView Software October 2015 www.radview.com Load and Performance Load Testing RadView Software October 2015 www.radview.com Contents Introduction... 3 Key Components and Architecture... 4 Creating Load Tests... 5 Mobile Load Testing... 9 Test Execution...

More information

Database Services for Physics @ CERN

Database Services for Physics @ CERN Database Services for Physics @ CERN Deployment and Monitoring Radovan Chytracek CERN IT Department Outline Database services for physics Status today How we do the services tomorrow? Performance tuning

More information

LSKA 2010 Survey Report Job Scheduler

LSKA 2010 Survey Report Job Scheduler LSKA 2010 Survey Report Job Scheduler Graduate Institute of Communication Engineering {r98942067, r98942112}@ntu.edu.tw March 31, 2010 1. Motivation Recently, the computing becomes much more complex. However,

More information

depl Documentation Release 0.0.1 depl contributors

depl Documentation Release 0.0.1 depl contributors depl Documentation Release 0.0.1 depl contributors December 19, 2013 Contents 1 Why depl and not ansible, puppet, chef, docker or vagrant? 3 2 Blog Posts talking about depl 5 3 Docs 7 3.1 Installation

More information

CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT

CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT SS Data & Storage CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT HEPiX Fall 2012 Workshop October 15-19, 2012 Institute of High Energy Physics, Beijing, China SS Outline

More information

ABRAHAM MARTIN @ABRAHAM_MARTINC ARCHITECTURE OF A CLOUD SERVICE USING PYTHON TECHNOLOGIES

ABRAHAM MARTIN @ABRAHAM_MARTINC ARCHITECTURE OF A CLOUD SERVICE USING PYTHON TECHNOLOGIES ABRAHAM MARTIN @ABRAHAM_MARTINC ARCHITECTURE OF A CLOUD SERVICE USING PYTHON TECHNOLOGIES MANAGED WEB SERVICE Born to solve a problem around university Servers under desks Security problems MANAGED WEB

More information

UZH Experiences with OpenStack

UZH Experiences with OpenStack GC3: Grid Computing Competence Center UZH Experiences with OpenStack What we did, what went well, what went wrong. Antonio Messina 29 April 2013 Setting up Hardware configuration

More information

ArcGIS for Server: In the Cloud

ArcGIS for Server: In the Cloud DevSummit DC February 11, 2015 Washington, DC ArcGIS for Server: In the Cloud Bonnie Stayer, Esri Session Outline Cloud Overview - Benefits - Types of clouds ArcGIS in AWS - Cloud Builder - Maintenance

More information

VM Management for Green Data Centres with the OpenNebula Virtual Infrastructure Engine

VM Management for Green Data Centres with the OpenNebula Virtual Infrastructure Engine OGF-EU: Using IT to reduce Carbon Emissions and Delivering the Potential of Energy Efficient Computing OGF25, Catania, Italy 5 March 2009 VM Management for Green Data Centres with the OpenNebula Virtual

More information

Pilot-Streaming: Design Considerations for a Stream Processing Framework for High- Performance Computing

Pilot-Streaming: Design Considerations for a Stream Processing Framework for High- Performance Computing Pilot-Streaming: Design Considerations for a Stream Processing Framework for High- Performance Computing Andre Luckow, Peter M. Kasson, Shantenu Jha STREAMING 2016, 03/23/2016 RADICAL, Rutgers, http://radical.rutgers.edu

More information

Development in Azure. Dan Gartner Developer Technology Specialist Microsoft

Development in Azure. Dan Gartner Developer Technology Specialist Microsoft Development in Azure Dan Gartner Developer Technology Specialist Microsoft MSDN Azure Benefits Visual Studio / Azure Integration Azure SDK 2.5 Visual Studio Online Build and Load Test Application Insights

More information

Design and Implementation of IaaS platform based on tool migration Wei Ding

Design and Implementation of IaaS platform based on tool migration Wei Ding 4th International Conference on Mechatronics, Materials, Chemistry and Computer Engineering (ICMMCCE 2015) Design and Implementation of IaaS platform based on tool migration Wei Ding State Key Laboratory

More information

Ansible in Depth WHITEPAPER. ansible.com +1 800-825-0212

Ansible in Depth WHITEPAPER. ansible.com +1 800-825-0212 +1 800-825-0212 WHITEPAPER Ansible in Depth Get started with ANSIBLE now: /get-started-with-ansible or contact us for more information: info@ INTRODUCTION Ansible is an open source IT configuration management,

More information

VMware vrealize Automation

VMware vrealize Automation VMware vrealize Automation Reference Architecture Version 6.0 and Higher T E C H N I C A L W H I T E P A P E R Table of Contents Overview... 4 What s New... 4 Initial Deployment Recommendations... 4 General

More information

Data Centers and Cloud Computing

Data Centers and Cloud Computing Data Centers and Cloud Computing CS377 Guest Lecture Tian Guo 1 Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing Case Study: Amazon EC2 2 Data Centers

More information