Infrastructure Clouds for Science and Education: Platform Tools

Size: px
Start display at page:

Download "Infrastructure Clouds for Science and Education: Platform Tools"

Transcription

1 Infrastructure Clouds for Science and Education: Platform Tools Kate Keahey, Renato J. Figueiredo, John Bresnahan, Mike Wilde, David LaBissoniere Argonne National Laboratory Computation Institute, University of Chicago University of Florida 7/16/2012 1

2 The Power of Infrastructure Clouds Virtualization opens the flood gates Outsourcing Virtual appliances Freeze your stack in time Run it anywhere Multi-cloud applications Run many copies all over the world! Elasticity 7/16/2012 2

3 Harnessing The Power Organization tools and techniques 7/16/2012 3

4 Towards a Power Adapter 7/16/2012 4

5 What Needs To Be Harnessed VM (appliance) creation and development configuration management tools (chef, puppet) VM hypervisors Infrastructure-as-a-Service (IaaS) Cloud applications virtual clusters, cloudinit.d, CloudFormation, Elasticity Auto-scaling tools, phantom Workflow Swift, etc 7/16/2012 5

6 What Needs To Be Organized? VM (appliance) creation and development configuration management tools (chef, puppet) VM hypervisors Infrastructure-as-a-Service (IaaS) Cloud applications virtual clusters, cloudinit.d, CloudFormation, Elasticity Auto-scaling tools, phantom Workflow Swift, etc 7/16/2012 6

7 VM Applications An entire system frozen in time Full software stacks (versions) Configuration files Important for science! A dedicated modular service Web service, database, AMQP node, etc Demos A binary single file (or set of files) Easy to freeze 7/16/2012 7

8 Developing Appliances A single binary image? Many developers? Version control? Merging conflicts? Base image with a description Ex: Ubuntu base images plus a set of scripts Configuration Management Software Chef, Puppet, FG Rain, etc 7/16/2012 8

9 Chef Software stack description ruby and json A library of cookbooks Cookbooks contain recipes Ex: apache2 server with php4 Attributes to customize each recipe Ex: on what port will apache listen Templates for configuration files Appliance developers make recipes Version control can be done with git/svn/cvs 7/16/2012 9

10 Example Recipe app_dir = node[:appdir] ve_dir = node[:virtualenv][:path] git app_dir do repository node[:autoscale][:git_repo] reference node[:autoscale][:git_branch] action :sync user node[:username] group node[:groupname] end execute "run install" do cwd app_dir user node[:username] group node[:groupname] command "python setup.py install" end 7/16/

11 Example Template phantom: system: type: epu rabbit: <%= node[:autoscale][:rabbit_host] %> rabbit_port: <%= node[:autoscale][:rabbit_port] %> rabbit_ssl: False rabbit_user: <%= node[:autoscale][:rabbit_username] %> rabbit_pw: <%= node[:autoscale][:rabbit_password] %> rabbit_exchange: <%= node[:autoscale][:rabbit_exchange] %> authz: type: sqldb dburl: <%= node[:autoscale][:dburl] %> phantom: system: type: epu rabbit: vm-102.uc.futuregrid.org rabbit_port: 5672 rabbit_ssl: False rabbit_user: XXX rabbit_pw: PPPPPP rabbit_exchange: default_dashi_exchange authz: type: sqldb dburl: 7/16/

12 What Needs To Be Organized? VM (appliance) creation and development configuration management tools (chef, puppet) VM hypervisors Infrastructure-as-a-Service (IaaS) Cloud applications virtual clusters, cloudinit.d, CloudFormation, Elasticity Auto-scaling tools, phantom Workflow Swift, etc 7/16/

13 Cloud Applications More than 1 VM needed for the job Information exchange is needed Manual information exchange Multi-cloud Cloud independence required Web Server database Web Web Web Servers nginx Web Server 7/16/

14 Cloud Management Tools Architecture description VM type, location, count Volumes Networks Other services Contextualization Exchange dynamically determined information IP addrs, security information. Bootstrap component connections Ex: mount NFS, connect to DB, etc 7/16/

15 A Simplified Deployment Scenario 7/16/

16 A Grid in Your Pocket Pierre EC2 7/16/

17 A Grid in Your Pocket Jamie Pierre OOI private cloud EC2 7/16/

18 A Grid in Your Pocket David Jamie Pierre FutureGrid OOI private cloud EC2 7/16/

19 CloudFormation Assemble AWS services Run AMIs. Connect EBS volumes to AMIs Associate and SQS queue, etc JSON descriptions AWS only No configuration management software integration Manual integration with Chef 7/16/

20 cloudinit.d Multicloud VM dependency management Uses the libcloud abstraction library Integrated with chef solo ini file format descriptions Coupled with any executable script Launch plan end-users/operators Lightweight Copy launch plan and one click action Easily reconfigured for various clouds Launch plan/application developers: Minimal software assumptions (ssh) Stem cell deployment approach Incremental launch plan development [svc-alamohttp] iaas_key: XXXXXX iaas_secret: XXXX iaas_host: alamo.futuregrid.org iaas_port: 8443 iaas: Nimbus image: ubunut10.10 ssh_username: ubuntu localsshkeypath: ~/.ssh/fg.pem readypgm: http-test.py bootpgm: http-boot.sh 7/16/

21 cloudinit.d Overview Services Run Levels Collections of services without dependencies on each other Launch Plan An ordered set of run levels 7/16/

22 Cloudinit.d Features Launch plan database Web Server Web Server Web Server Repeatability: write a launch plan once, deploy many times 7/16/

23 Cloudinit.d Features Launch plan database Web Server Web Server Web Server Deploy on cloud and non-cloud resources from many providers 7/16/

24 Cloudinit.d Features Launch plan database Web Server Web Server Web Server Run-level 1 Run-level 2 Coordination of interdependent launches 7/16/

25 Cloudinit.d Features Launch plan database Web Server Web Server Web Server Run-level 1 Run-level 2 User-defined launch tests 7/16/

26 Cloudinit.d Features Launch plan database Web Server Web Server Web Server Run-level 1 Run-level 2 Test-based monitoring and repair 7/16/

27 Cloudinit.d Features Launch plan database Web Server Web Server Web Server Run-level 1 Run-level 2 Test-based monitoring and repair 7/16/

28 A Single Service Application Boot Here we show how cloudinit.d automatically creates a HTTP server from a simple distribution base image Poll the IaaS service to determine when the VM is running Infrastructure Cloud Cloudinit.d Request The a VM new is VM running bootpgm readypgm Check Status Verify ssh works scp over scp over the boot the ready program contextualization program If the has a successful exit code (0), then the new simple cloud application is set to go! Iaas Interface sshd VM HTTP New VM Server sshd needs to startup and be Run the boot Run the accessible on program. the ready new VM program Now the VM has been contextualized to be a web server

29 What Needs To Be Organized? VM (appliance) creation and development configuration management tools (chef, puppet) VM hypervisors Infrastructure-as-a-Service (IaaS) Cloud applications virtual clusters, cloudinit.d, CloudFormation, Elasticity Auto-scaling tools, phantom Workflow Swift, etc 7/16/

30 Escalation Pattern Challenge: leverage on-demand, large but unreliable provider pool Applications that absorb resources Applications that tolerate failures User Domain (configuration and security) Operational Units Domain Management: Monitor and regulate domain properties based on system-specific and application-specific metrics 7/16/

31 Scaling Considerations Reasons to scale Business vs science Cost vs quota Lossy environment VMs fail more often than bare metal N preserving Spot instances If the price is right Backfill If resources are idle 7/16/

32 Amazon Auto Scaling and CloudWatch Auto Scaling in EC2 Policies to scale up and down servers Min, Max, and desired size Integrated with AWS CloudWatch Sensors Triggers CPU load, disk capacity, load balancer loads,etc Custom sensors No contextualization REST API AWS only 7/16/

33 Phantom Scaling Services Multi-cloud Fail-over and even distribution policies Monitor scaling factors and failures Generic/system qualities: deployment status, load, bank account, etc. Application-specific qualities, e.g., a workload queue for ALiEn, PBS, AMQP, and others Evaluate against policies Scale and/or recover For user components For system components Across different cloud providers Release as a Service 0.1 running on FutureGrid now Initially available as a service on FutureGrid resources Provides high availability Sensor information Apply Policy Reliably provision, manage and contextualize resources 7/16/

34 Infrastructure Platform Goals Multi-cloud Work across private, community and commercial clouds Any Scale Scale in response to a diverse set of sensors/triggers Both system and application sensors High Availability Any VM can die : system or user VMs Minimizing time to recovery (TTR) Your Polices, Our Enactment User-defined sensors/triggers and policies Engineered from the ground up to work with infrastructure clouds Easy on the user 7/16/

35 How Can Science Plug Into This Power Example Embarrassingly Parallel Scientific Application Demonstration 7/16/

36 Using Nimbus Domains Task Queue M subtask messages Application Start the workers

37 Using Nimbus Domains Message Queue M subtask messages Get task Infrastructure Compute Cloud Application Start the workers N preserving policy Preserve N worker VMs Cumulus/S3 Results/Checkpoints

38 Phantom Architecture REST HTTPS nginx Web Application FutureGrid Clouds Web Application HTTPS MySQL REST Service REST Service REST Service Provisioner IaaS Clouds DTRS RabbitMQ EPUM Zookeeper Cluster 7/16/

39 Adventures in Availability Time to scale (TTS) PENDING (request) STARTED (deployment) RUNNING (contextualization) TTS: preliminary results for 2,000 VMs provisioned on AWS EC2 7/16/

40 Applications Elastic MapReduce Science Gateways Custom Applications (OOI) Schedulers Data Transfer Systems Workflow Systems (Swift) Application adaptation: Library of generic sensors Application-specific sensors Policies Decision Engine Infrastructure Platform Contextualization, multi-cloud bridge, repeatable launches, scaling, elasticity and High Availability 7/16/

Infrastructure Outsourcing in Mul2- Cloud Environment

Infrastructure Outsourcing in Mul2- Cloud Environment Infrastructure Outsourcing in Mul2- Cloud Environment Kate Keahey, Patrick Armstrong, David LaBissoniere, Pierre Riteau Argonne National Laboratory Computation Institute, University of Chicago 11/20/12

More information

EEDC. Scalability Study of web apps in AWS. Execution Environments for Distributed Computing

EEDC. Scalability Study of web apps in AWS. Execution Environments for Distributed Computing EEDC Execution Environments for Distributed Computing 34330 Master in Computer Architecture, Networks and Systems - CANS Scalability Study of web apps in AWS Sergio Mendoza sergio.mendoza@est.fib.upc.edu

More information

19.10.11. Amazon Elastic Beanstalk

19.10.11. Amazon Elastic Beanstalk 19.10.11 Amazon Elastic Beanstalk A Short History of AWS Amazon started as an ECommerce startup Original architecture was restructured to be more scalable and easier to maintain Competitive pressure for

More information

salsadpi: a dynamic provisioning interface for IaaS cloud

salsadpi: a dynamic provisioning interface for IaaS cloud salsadpi: a dynamic provisioning interface for IaaS cloud Tak-Lon (Stephen) Wu Computer Science, School of Informatics and Computing Indiana University, Bloomington, IN taklwu@indiana.edu Abstract On-demand

More information

Infrastructure Outsourcing in Multi-Cloud Environment

Infrastructure Outsourcing in Multi-Cloud Environment Infrastructure Outsourcing in Multi-Cloud Environment Kate Keahey Argonne National Laboratory keahey@mcs.anl.gov Patrick Armstrong oldpatricka@uchicago.edu John Bresnahan Argonne National Laboratory bresnaha@mcs.anl.gov

More information

Alfresco Enterprise on AWS: Reference Architecture

Alfresco Enterprise on AWS: Reference Architecture Alfresco Enterprise on AWS: Reference Architecture October 2013 (Please consult http://aws.amazon.com/whitepapers/ for the latest version of this paper) Page 1 of 13 Abstract Amazon Web Services (AWS)

More information

Scalable Architecture on Amazon AWS Cloud

Scalable Architecture on Amazon AWS Cloud Scalable Architecture on Amazon AWS Cloud Kalpak Shah Founder & CEO, Clogeny Technologies kalpak@clogeny.com 1 * http://www.rightscale.com/products/cloud-computing-uses/scalable-website.php 2 Architect

More information

Introduction to DevOps on AWS

Introduction to DevOps on AWS Introduction to DevOps on AWS David Chapman December 2014 Contents Contents Abstract Introduction Agile Evolution to DevOps Infrastructure as Code AWS CloudFormation AWS AMI Continuous Deployment AWS CodeDeploy

More information

Cloud computing - Architecting in the cloud

Cloud computing - Architecting in the cloud Cloud computing - Architecting in the cloud anna.ruokonen@tut.fi 1 Outline Cloud computing What is? Levels of cloud computing: IaaS, PaaS, SaaS Moving to the cloud? Architecting in the cloud Best practices

More information

SC12 Cloud Compu,ng for Science Tutorial: Introduc,on to Infrastructure Clouds

SC12 Cloud Compu,ng for Science Tutorial: Introduc,on to Infrastructure Clouds SC12 Cloud Compu,ng for Science Tutorial: Introduc,on to Infrastructure Clouds John Breshnahan, Patrick Armstrong, Kate Keahey, Pierre Riteau Argonne National Laboratory Computation Institute, University

More information

TECHNOLOGY WHITE PAPER Jun 2012

TECHNOLOGY WHITE PAPER Jun 2012 TECHNOLOGY WHITE PAPER Jun 2012 Technology Stack C# Windows Server 2008 PHP Amazon Web Services (AWS) Route 53 Elastic Load Balancing (ELB) Elastic Compute Cloud (EC2) Amazon RDS Amazon S3 Elasticache

More information

Last time. Today. IaaS Providers. Amazon Web Services, overview

Last time. Today. IaaS Providers. Amazon Web Services, overview Last time General overview, motivation, expected outcomes, other formalities, etc. Please register for course Online (if possible), or talk to Yvonne@CS Course evaluation forgotten Please assign one volunteer

More information

Nimbus: Cloud Computing with Science

Nimbus: Cloud Computing with Science Nimbus: Cloud Computing with Science March 2010 globusworld, Chicago Kate Keahey keahey@mcs.anl.gov Nimbus Project University of Chicago Argonne National Laboratory Cloud Computing for Science Environment

More information

TECHNOLOGY WHITE PAPER Jan 2016

TECHNOLOGY WHITE PAPER Jan 2016 TECHNOLOGY WHITE PAPER Jan 2016 Technology Stack C# PHP Amazon Web Services (AWS) Route 53 Elastic Load Balancing (ELB) Elastic Compute Cloud (EC2) Amazon RDS Amazon S3 Elasticache CloudWatch Paypal Overview

More information

Web Application Hosting in the AWS Cloud Best Practices

Web Application Hosting in the AWS Cloud Best Practices Web Application Hosting in the AWS Cloud Best Practices May 2010 Matt Tavis Page 1 of 12 Abstract Highly-available and scalable web hosting can be a complex and expensive proposition. Traditional scalable

More information

www.boost ur skills.com

www.boost ur skills.com www.boost ur skills.com AWS CLOUD COMPUTING WORKSHOP Write us at training@boosturskills.com BOOSTURSKILLS No 1736 1st Amrutha College Road Kasavanhalli,Off Sarjapur Road,Bangalore-35 1) Introduction &

More information

Deploying Business Virtual Appliances on Open Source Cloud Computing

Deploying Business Virtual Appliances on Open Source Cloud Computing International Journal of Computer Science and Telecommunications [Volume 3, Issue 4, April 2012] 26 ISSN 2047-3338 Deploying Business Virtual Appliances on Open Source Cloud Computing Tran Van Lang 1 and

More information

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida Plug-and-play Virtual Appliance Clusters Running Hadoop Dr. Renato Figueiredo ACIS Lab - University of Florida Advanced Computing and Information Systems laboratory Introduction You have so far learned

More information

Scalable Application. Mikalai Alimenkou http://xpinjection.com 11.05.2012

Scalable Application. Mikalai Alimenkou http://xpinjection.com 11.05.2012 Scalable Application Development on AWS Mikalai Alimenkou http://xpinjection.com 11.05.2012 Background Java Technical Lead/Scrum Master at Zoral Labs 7+ years in software development 5+ years of working

More information

Auto-Scaling Model for Cloud Computing System

Auto-Scaling Model for Cloud Computing System Auto-Scaling Model for Cloud Computing System Che-Lun Hung 1*, Yu-Chen Hu 2 and Kuan-Ching Li 3 1 Dept. of Computer Science & Communication Engineering, Providence University 2 Dept. of Computer Science

More information

Cloud Computing with Nimbus

Cloud Computing with Nimbus Cloud Computing with Nimbus April 2010 Condor Week Kate Keahey keahey@mcs.anl.gov Nimbus Project University of Chicago Argonne National Laboratory Cloud Computing for Science Environment Complexity Consistency

More information

Amazon EC2 Product Details Page 1 of 5

Amazon EC2 Product Details Page 1 of 5 Amazon EC2 Product Details Page 1 of 5 Amazon EC2 Functionality Amazon EC2 presents a true virtual computing environment, allowing you to use web service interfaces to launch instances with a variety of

More information

OGF25/EGEE User Forum Catania, Italy 2 March 2009

OGF25/EGEE User Forum Catania, Italy 2 March 2009 OGF25/EGEE User Forum Catania, Italy 2 March 2009 Constantino Vázquez Blanco Javier Fontán Muiños Raúl Sampedro Distributed Systems Architecture Research Group Universidad Complutense de Madrid 1/31 Outline

More information

Développement logiciel pour le Cloud (TLC)

Développement logiciel pour le Cloud (TLC) Développement logiciel pour le Cloud (TLC) 7. Infrastructure-as-a-Service Guillaume Pierre Université de Rennes 1 Fall 2012 http://www.globule.org/~gpierre/ Développement logiciel pour le Cloud (TLC) 1

More information

Savanna Hadoop on. OpenStack. Savanna Technical Lead

Savanna Hadoop on. OpenStack. Savanna Technical Lead Savanna Hadoop on OpenStack Sergey Lukjanov Savanna Technical Lead Mirantis, 2013 Agenda Savanna Overview Savanna Use Cases Roadmap & Current Status Architecture & Features Overview Hadoop vs. Virtualization

More information

ArcGIS for Server: In the Cloud

ArcGIS for Server: In the Cloud DevSummit DC February 11, 2015 Washington, DC ArcGIS for Server: In the Cloud Bonnie Stayer, Esri Session Outline Cloud Overview - Benefits - Types of clouds ArcGIS in AWS - Cloud Builder - Maintenance

More information

Razvoj Java aplikacija u Amazon AWS Cloud: Praktična demonstracija

Razvoj Java aplikacija u Amazon AWS Cloud: Praktična demonstracija Razvoj Java aplikacija u Amazon AWS Cloud: Praktična demonstracija Robert Dukarić University of Ljubljana Faculty of Computer and Information Science Laboratory for information systems integration Competence

More information

Running your Java EE 6 Applications in the Cloud

Running your Java EE 6 Applications in the Cloud Running your Java EE 6 Applications in the Cloud Arun Gupta Oracle blogs.sun.com/arungupta @arungupta The following/preceding is intended to outline our general product direction. It is intended for information

More information

Amazon Web Services Primer. William Strickland COP 6938 Fall 2012 University of Central Florida

Amazon Web Services Primer. William Strickland COP 6938 Fall 2012 University of Central Florida Amazon Web Services Primer William Strickland COP 6938 Fall 2012 University of Central Florida AWS Overview Amazon Web Services (AWS) is a collection of varying remote computing provided by Amazon.com.

More information

SUSE Cloud 2.0. Pete Chadwick. Douglas Jarvis. Senior Product Manager pchadwick@suse.com. Product Marketing Manager djarvis@suse.

SUSE Cloud 2.0. Pete Chadwick. Douglas Jarvis. Senior Product Manager pchadwick@suse.com. Product Marketing Manager djarvis@suse. SUSE Cloud 2.0 Pete Chadwick Douglas Jarvis Senior Product Manager pchadwick@suse.com Product Marketing Manager djarvis@suse.com SUSE Cloud SUSE Cloud is an open source software solution based on OpenStack

More information

OpenNebula Open Souce Solution for DC Virtualization. C12G Labs. Online Webinar

OpenNebula Open Souce Solution for DC Virtualization. C12G Labs. Online Webinar OpenNebula Open Souce Solution for DC Virtualization C12G Labs Online Webinar What is OpenNebula? Multi-tenancy, Elasticity and Automatic Provision on Virtualized Environments I m using virtualization/cloud,

More information

OSG All Hands Meeting

OSG All Hands Meeting Nimbus Update March 2010 OSG All Hands Meeting Kate Keahey keahey@mcs.anl.gov Nimbus Project University of Chicago Argonne National Laboratory Nimbus: Cloud Computing for Science Allow providers to build

More information

EXECUTIVE SUMMARY CONTENTS. 1. Summary 2. Objectives 3. Methodology and Approach 4. Results 5. Next Steps 6. Glossary 7. Appendix. 1.

EXECUTIVE SUMMARY CONTENTS. 1. Summary 2. Objectives 3. Methodology and Approach 4. Results 5. Next Steps 6. Glossary 7. Appendix. 1. CONTENTS 1. Summary 2. Objectives 3. Methodology and Approach 4. Results 5. Next Steps 6. Glossary 7. Appendix EXECUTIVE SUMMARY Tenzing Managed IT services has recently partnered with Amazon Web Services

More information

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases NASA Ames NASA Advanced Supercomputing (NAS) Division California, May 24th, 2012 Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases Ignacio M. Llorente Project Director OpenNebula Project.

More information

w w w. u l t i m u m t e c h n o l o g i e s. c o m Infrastructure-as-a-Service on the OpenStack platform

w w w. u l t i m u m t e c h n o l o g i e s. c o m Infrastructure-as-a-Service on the OpenStack platform w w w. u l t i m u m t e c h n o l o g i e s. c o m Infrastructure-as-a-Service on the OpenStack platform http://www.ulticloud.com http://www.openstack.org Introduction to OpenStack 1. What OpenStack is

More information

A Complete Open Cloud Storage, Virt, IaaS, PaaS. Dave Neary Open Source and Standards, Red Hat

A Complete Open Cloud Storage, Virt, IaaS, PaaS. Dave Neary Open Source and Standards, Red Hat A Complete Open Cloud Storage, Virt, IaaS, PaaS Dave Neary Open Source and Standards, Red Hat 1 Agenda 1. Traditional virtualization 2. The move to IaaS 3. Storage 4. PaaS, application encapsulation and

More information

rackspace.com/cloud/private

rackspace.com/cloud/private rackspace.com/cloud/private Rackspace Private Cloud Installation (2014-11-21) Copyright 2014 Rackspace All rights reserved. This documentation is intended for users who want to install Rackspace Private

More information

Deployment of Private, Hybrid & Public Clouds with OpenNebula

Deployment of Private, Hybrid & Public Clouds with OpenNebula EL / LAK (FOSS) 2010 May 14th, 2010 Deployment of Private, Hybrid & Public Clouds with OpenNebula University Complutense of Madrid The Anatomy of an IaaS Cloud Deployment of Private, Hybrid & Public Clouds

More information

Best Practices for Python in the Cloud: Lessons Learned @ActiveState

Best Practices for Python in the Cloud: Lessons Learned @ActiveState Best Practices for Python in the Cloud: Lessons Learned @ActiveState Best Practices for Python in the Cloud Presented by: Gisle Aas, Senior Developer, ActiveState whoami? Gisle Aas! gisle@activestate.com!

More information

Getting Started Hacking on OpenNebula

Getting Started Hacking on OpenNebula LinuxTag 2013 Berlin, Germany, May 22nd Getting Started Hacking on OpenNebula Carlos Martín Project Engineer Acknowledgments The research leading to these results has received funding from Comunidad de

More information

Web Application Hosting in the AWS Cloud Best Practices

Web Application Hosting in the AWS Cloud Best Practices Web Application Hosting in the AWS Cloud Best Practices September 2012 Matt Tavis, Philip Fitzsimons Page 1 of 14 Abstract Highly available and scalable web hosting can be a complex and expensive proposition.

More information

CONDOR CLUSTERS ON EC2

CONDOR CLUSTERS ON EC2 CONDOR CLUSTERS ON EC2 Val Hendrix, Roberto A. Vitillo Lawrence Berkeley National Lab ATLAS Cloud Computing R & D 1 INTRODUCTION This is our initial work on investigating tools for managing clusters and

More information

Rebalancing in a Multi-Cloud Environment

Rebalancing in a Multi-Cloud Environment Rebalancing in a Multi-Cloud Environment Dmitry Duplyakin University of Colorado dmitry.duplyakin @colorado.edu Paul Marshall University of Colorado paul.marshall @colorado.edu Kate Keahey Argonne National

More information

OpenNebula Open Souce Solution for DC Virtualization

OpenNebula Open Souce Solution for DC Virtualization 13 th LSM 2012 7 th -12 th July, Geneva OpenNebula Open Souce Solution for DC Virtualization Constantino Vázquez Blanco OpenNebula.org What is OpenNebula? Multi-tenancy, Elasticity and Automatic Provision

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details Use cases By scientist By HEP experiment System requirements and installation How to get it? 2 What

More information

JAVA IN THE CLOUD PAAS PLATFORM IN COMPARISON

JAVA IN THE CLOUD PAAS PLATFORM IN COMPARISON JAVA IN THE CLOUD PAAS PLATFORM IN COMPARISON Eberhard Wolff Architecture and Technology Manager adesso AG, Germany 12.10. Agenda A Few Words About Cloud Java and IaaS PaaS Platform as a Service Google

More information

Cloud Computing with Amazon Web Services and the DevOps Methodology. www.cloudreach.com

Cloud Computing with Amazon Web Services and the DevOps Methodology. www.cloudreach.com Cloud Computing with Amazon Web Services and the DevOps Methodology Who am I? Max Manders @maxmanders Systems Developer at Cloudreach @cloudreach Director / Co-Founder of Whisky Web @whiskyweb Who are

More information

Cloud Models and Platforms

Cloud Models and Platforms Cloud Models and Platforms Dr. Sanjay P. Ahuja, Ph.D. 2010-14 FIS Distinguished Professor of Computer Science School of Computing, UNF A Working Definition of Cloud Computing Cloud computing is a model

More information

A Comparison of Clouds: Amazon Web Services, Windows Azure, Google Cloud Platform, VMWare and Others (Fall 2012)

A Comparison of Clouds: Amazon Web Services, Windows Azure, Google Cloud Platform, VMWare and Others (Fall 2012) 1. Computation Amazon Web Services Amazon Elastic Compute Cloud (Amazon EC2) provides basic computation service in AWS. It presents a virtual computing environment and enables resizable compute capacity.

More information

SC12 Cloud Compu,ng for Science Tutorial: Cloud Challenges

SC12 Cloud Compu,ng for Science Tutorial: Cloud Challenges SC12 Cloud Compu,ng for Science Tutorial: Cloud Challenges Kate Keahey, John Breshnahan, Patrick Armstrong, Pierre Riteau Argonne National Laboratory Computation Institute, University of Chicago 11/13/12

More information

Leveraging BlobSeer to boost up the deployment and execution of Hadoop applications in Nimbus cloud environments on Grid 5000

Leveraging BlobSeer to boost up the deployment and execution of Hadoop applications in Nimbus cloud environments on Grid 5000 Leveraging BlobSeer to boost up the deployment and execution of Hadoop applications in Nimbus cloud environments on Grid 5000 Alexandra Carpen-Amarie Diana Moise Bogdan Nicolae KerData Team, INRIA Outline

More information

Best Practices for Sharing Imagery using Amazon Web Services. Peter Becker

Best Practices for Sharing Imagery using Amazon Web Services. Peter Becker Best Practices for Sharing Imagery using Amazon Web Services Peter Becker Objectives Making Imagery Accessible Store massive volumes of imagery on inexpensive cloud storage Use elastic compute for image

More information

OpenNebula Open Souce Solution for DC Virtualization

OpenNebula Open Souce Solution for DC Virtualization OSDC 2012 25 th April, Nürnberg OpenNebula Open Souce Solution for DC Virtualization Constantino Vázquez Blanco OpenNebula.org What is OpenNebula? Multi-tenancy, Elasticity and Automatic Provision on Virtualized

More information

Planning, Provisioning and Deploying Enterprise Clouds with Oracle Enterprise Manager 12c Kevin Patterson, Principal Sales Consultant, Enterprise

Planning, Provisioning and Deploying Enterprise Clouds with Oracle Enterprise Manager 12c Kevin Patterson, Principal Sales Consultant, Enterprise Planning, Provisioning and Deploying Enterprise Clouds with Oracle Enterprise Manager 12c Kevin Patterson, Principal Sales Consultant, Enterprise Manager Oracle NIST Definition of Cloud Computing Cloud

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details System requirements and installation How to get it? 2 What is CC1? The CC1 system is a complete solution

More information

Last time. Today. IaaS Providers. Amazon Web Services, overview

Last time. Today. IaaS Providers. Amazon Web Services, overview Last time General overview, motivation, expected outcomes, other formalities, etc. Please register for course Online (if possible), or talk to CS secretaries Course evaluation forgotten Please assign one

More information

DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2

DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2 DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing Slide 1 Slide 3 A style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet.

More information

Stackato PaaS Architecture: How it works and why.

Stackato PaaS Architecture: How it works and why. Stackato PaaS Architecture: How it works and why. White Paper Published in 2012 Stackato PaaS Architecture: How it works and why. Stackato is software for creating a private Platform-as-a-Service (PaaS).

More information

The OpenNebula Cloud Platform for Data Center Virtualization

The OpenNebula Cloud Platform for Data Center Virtualization CloudOpen 2012 San Diego, USA, August 29th, 2012 The OpenNebula Cloud Platform for Data Center Virtualization Carlos Martín Project Engineer Acknowledgments The research leading to these results has received

More information

The Virtualization Practice

The Virtualization Practice The Virtualization Practice White Paper: Managing Applications in Docker Containers Bernd Harzog Analyst Virtualization and Cloud Performance Management October 2014 Abstract Docker has captured the attention

More information

NCTA Cloud Operations

NCTA Cloud Operations NCTA Cloud Operations 093018 Lesson 1: Cloud Operations Topic A: Overview of Cloud Computing Solutions Identify the core concepts of cloud computing. Operations Terminology Identify the terminology used

More information

!"#$%&'()*'+),-./)0' 9##+':,%-.;),0'

!#$%&'()*'+),-./)0' 9##+':,%-.;),0' Università degli studi di Roma Tor Vergata Facoltà di Ingegneria!"#$%&'()*'+),-./)0' 12/#'+.3-)04,.' 0.3-)04,.5.&672&.,%"#87.4' 9##+':,%-.;),0' 2 9##+':,%-.;),0'

More information

Cloud Computing. A new kind of developers? Presentation by. Nick Barcet nick.barcet@canonical.com

Cloud Computing. A new kind of developers? Presentation by. Nick Barcet nick.barcet@canonical.com Cloud Computing A new kind of developers? Presentation by Nick Barcet nick.barcet@canonical.com www.canonical.com July 2011 Cloud computing stack Salesforce.com, GoogleDocs, Office, etc... GoogleApps,

More information

Drupal in the Cloud. Scaling with Drupal and Amazon Web Services. Northern Virginia Drupal Meetup

Drupal in the Cloud. Scaling with Drupal and Amazon Web Services. Northern Virginia Drupal Meetup Drupal in the Cloud Scaling with Drupal and Amazon Web Services Northern Virginia Drupal Meetup 3 Dec 2008 Cast of Characters Eric at The Case Foundation: The Client With typical client challenges Cost:

More information

Migrating a running service to AWS

Migrating a running service to AWS Migrating a running service to AWS Nick Veenhof Ricardo Amaro DevOps Track https://events.drupal.org/barcelona2015/sessions/migrating-runningservice-mollom-aws-without-service-interruptions-and-reduce

More information

Introduction to Cloud Computing on Amazon Web Services (AWS) with focus on EC2 and S3. Horst Lueck

Introduction to Cloud Computing on Amazon Web Services (AWS) with focus on EC2 and S3. Horst Lueck Introduction to Cloud Computing on Amazon Web Services (AWS) with focus on EC2 and S3 Horst Lueck 2011-05-17 IT Pro Forum http://itproforum.org Thanks to Open Office Impress The Cloud the Name The 90s

More information

Managing Multi-Tiered Applications with AWS OpsWorks

Managing Multi-Tiered Applications with AWS OpsWorks Managing Multi-Tiered Applications with AWS OpsWorks Daniele Stroppa January 2015 Contents Contents Abstract Introduction Key Concepts Design Micro-services Architecture Provisioning and Deployment Managing

More information

Cloud Computing #8 - Datacenter OS. Johan Eker

Cloud Computing #8 - Datacenter OS. Johan Eker Cloud Computing #8 - Datacenter OS Johan Eker Outline What is a Datacenter OS? OpenStack Kubernetes Resource Management What is an OS? What is an OS? Manage hardware resources such as CPU, RAM, disk, I/O,

More information

OnApp Cloud. The complete platform for cloud service providers. 114 Cores. 286 Cores / 400 Cores

OnApp Cloud. The complete platform for cloud service providers. 114 Cores. 286 Cores / 400 Cores OnApp Cloud The complete platform for cloud service providers 286 Cores / 400 Cores 114 Cores 218 10 86 20 The complete platform for cloud service providers OnApp software turns your datacenter into an

More information

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH CONTENTS Introduction... 4 System Components... 4 OpenNebula Cloud Management Toolkit... 4 VMware

More information

Amazon Web Services. Elastic Compute Cloud (EC2) and more...

Amazon Web Services. Elastic Compute Cloud (EC2) and more... Amazon Web Services Elastic Compute Cloud (EC2) and more... I don t work for Amazon I do however, have a small research grant from Amazon (in AWS$) Portions of this presentation are reproduced from slides

More information

Open Source Cloud Computing Management with OpenNebula

Open Source Cloud Computing Management with OpenNebula CloudCamp Campus Party July 2011, Valencia Open Source Cloud Computing Management with OpenNebula Javier Fontán Muiños dsa-research.org Distributed Systems Architecture Research Group Universidad Complutense

More information

Chapter 3 Cloud Infrastructure. Cloud Computing: Theory and Practice. 1

Chapter 3 Cloud Infrastructure. Cloud Computing: Theory and Practice. 1 Chapter 3 Cloud Infrastructure Chapter 3 1 Contents IaaS services from Amazon. Regions and availability zones for Amazon Web Services. Instances attributes and cost. A repertoire of Amazon Web Services.

More information

Shoal: IaaS Cloud Cache Publisher

Shoal: IaaS Cloud Cache Publisher University of Victoria Faculty of Engineering Winter 2013 Work Term Report Shoal: IaaS Cloud Cache Publisher Department of Physics University of Victoria Victoria, BC Mike Chester V00711672 Work Term 3

More information

OVERVIEW. The complete IaaS platform for service providers

OVERVIEW. The complete IaaS platform for service providers OVERVIEW The complete IaaS platform for service providers The complete IaaS platform for service providers With OnApp you can sell more services, automate more of your infrastructure and extend your reach

More information

TestOps: Continuous Integration when infrastructure is the product. Barry Jaspan Senior Architect, Acquia Inc.

TestOps: Continuous Integration when infrastructure is the product. Barry Jaspan Senior Architect, Acquia Inc. TestOps: Continuous Integration when infrastructure is the product Barry Jaspan Senior Architect, Acquia Inc. This talk is about the hard parts. Rainbows and ponies have left the building. Intro to Continuous

More information

Designing Apps for Amazon Web Services

Designing Apps for Amazon Web Services Designing Apps for Amazon Web Services Mathias Meyer, GOTO Aarhus 2011 Montag, 10. Oktober 11 Montag, 10. Oktober 11 Me infrastructure code databases @roidrage www.paperplanes.de Montag, 10. Oktober 11

More information

Cloud services for the Fermilab scientific stakeholders

Cloud services for the Fermilab scientific stakeholders Cloud services for the Fermilab scientific stakeholders S Timm 1, G Garzoglio 1, P Mhashilkar 1*, J Boyd 1, G Bernabeu 1, N Sharma 1, N Peregonow 1, H Kim 1, S Noh 2,, S Palur 3, and I Raicu 3 1 Scientific

More information

15-319 / 15-619 Cloud Computing. Recitation 5 September 29 th & October 1 st 2015

15-319 / 15-619 Cloud Computing. Recitation 5 September 29 th & October 1 st 2015 15-319 / 15-619 Cloud Computing Recitation 5 September 29 th & October 1 st 2015 1 Overview Administrative Stuff Last Week s Reflection Project 2.1, OLI Modules 5 & 6, Quiz 3 New concepts This week s schedule

More information

Efficient Cloud Management for Parallel Data Processing In Private Cloud

Efficient Cloud Management for Parallel Data Processing In Private Cloud 2012 International Conference on Information and Network Technology (ICINT 2012) IPCSIT vol. 37 (2012) (2012) IACSIT Press, Singapore Efficient Cloud Management for Parallel Data Processing In Private

More information

Apache Stratos (incubating) 4.0.0-M5 Installation Guide

Apache Stratos (incubating) 4.0.0-M5 Installation Guide Apache Stratos (incubating) 4.0.0-M5 Installation Guide 1. Prerequisites 2. Product Configuration 2.1 Message Broker Configuration 2.2 Load Balancer Configuration 2.3 Cloud Controller Configuration 2.4

More information

How To Scale A Server Farm

How To Scale A Server Farm Basics of Cloud Computing Lecture 3 Scaling Applications on the Cloud Satish Srirama Outline Scaling Information Systems Scaling Enterprise Applications in the Cloud Auto Scaling 25/02/2014 Satish Srirama

More information

Last time. Today. IaaS Providers. Amazon Web Services, overview

Last time. Today. IaaS Providers. Amazon Web Services, overview Last time General overview, motivation, expected outcomes, other formalities, etc. Please register for course Online (if possible), or talk to CS secretaries Cloud computing introduction General concepts

More information

STeP-IN SUMMIT 2013. June 18 21, 2013 at Bangalore, INDIA. Performance Testing of an IAAS Cloud Software (A CloudStack Use Case)

STeP-IN SUMMIT 2013. June 18 21, 2013 at Bangalore, INDIA. Performance Testing of an IAAS Cloud Software (A CloudStack Use Case) 10 th International Conference on Software Testing June 18 21, 2013 at Bangalore, INDIA by Sowmya Krishnan, Senior Software QA Engineer, Citrix Copyright: STeP-IN Forum and Quality Solutions for Information

More information

Cloudify and OpenStack Heat

Cloudify and OpenStack Heat Cloudify and OpenStack Heat General Cloudify is an application orchestration platform that provides a complete solution for automating and managing application deployment and DevOps processes on top of

More information

Migration Scenario: Migrating Batch Processes to the AWS Cloud

Migration Scenario: Migrating Batch Processes to the AWS Cloud Migration Scenario: Migrating Batch Processes to the AWS Cloud Produce Ingest Process Store Manage Distribute Asset Creation Data Ingestor Metadata Ingestor (Manual) Transcoder Encoder Asset Store Catalog

More information

Continuous Integration in the Cloud with Hudson

Continuous Integration in the Cloud with Hudson Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers Rise of Continuous Integration Offload from people, push to computers $ computers

More information

CloudCenter Full Lifecycle Management. An application-defined approach to deploying and managing applications in any datacenter or cloud environment

CloudCenter Full Lifecycle Management. An application-defined approach to deploying and managing applications in any datacenter or cloud environment CloudCenter Full Lifecycle Management An application-defined approach to deploying and managing applications in any datacenter or cloud environment CloudCenter Full Lifecycle Management Page 2 Table of

More information

Virtual Machine Management with OpenNebula in the RESERVOIR project

Virtual Machine Management with OpenNebula in the RESERVOIR project CISCO Cloud Computing Research Symposium (C 3 RS) November 5 & 6, 2008 San Jose, CA Virtual Machine Management with OpenNebula in the RESERVOIR project Ruben Santiago Montero Distributed Systems Architecture

More information

Migration Scenario: Migrating Backend Processing Pipeline to the AWS Cloud

Migration Scenario: Migrating Backend Processing Pipeline to the AWS Cloud Migration Scenario: Migrating Backend Processing Pipeline to the AWS Cloud Use case Figure 1: Company C Architecture (Before Migration) Company C is an automobile insurance claim processing company with

More information

Data Centers and Cloud Computing

Data Centers and Cloud Computing Data Centers and Cloud Computing CS377 Guest Lecture Tian Guo 1 Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing Case Study: Amazon EC2 2 Data Centers

More information

Apache Libcloud on Openstack

Apache Libcloud on Openstack GC3: Grid Computing Competence Center Large Scale Computing Infrastructures Lab 5: libcloud Sergio Maffioletti GC3: Grid Computing Competence Center, University of Zurich

More information

International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan

International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud Ruben Santiago Montero Distributed

More information

Administrative Issues

Administrative Issues Administrative Issues Make use of office hours We will have to make sure that you have tried yourself before you ask Monitor AWS expenses regularly Always do the cost calculation before launching services

More information

Financial Services Grid Computing on Amazon Web Services January 2013 Ian Meyers

Financial Services Grid Computing on Amazon Web Services January 2013 Ian Meyers Financial Services Grid Computing on Amazon Web Services January 2013 Ian Meyers (Please consult http://aws.amazon.com/whitepapers for the latest version of this paper) Page 1 of 15 Contents Abstract...

More information

An Introduction to Cloud Computing Concepts

An Introduction to Cloud Computing Concepts Software Engineering Competence Center TUTORIAL An Introduction to Cloud Computing Concepts Practical Steps for Using Amazon EC2 IaaS Technology Ahmed Mohamed Gamaleldin Senior R&D Engineer-SECC ahmed.gamal.eldin@itida.gov.eg

More information

CloudStack and Big Data. Sebastien Goasguen @sebgoa May 22nd 2013 LinuxTag, Berlin

CloudStack and Big Data. Sebastien Goasguen @sebgoa May 22nd 2013 LinuxTag, Berlin CloudStack and Big Data Sebastien Goasguen @sebgoa May 22nd 2013 LinuxTag, Berlin Google trends Start of Clouds Cloud computing trending down, while Big Data is booming. Virtualization BigData on the Trigger

More information

C2C: An Automated Deployment Framework for Distributed Applications on Multi-Clouds

C2C: An Automated Deployment Framework for Distributed Applications on Multi-Clouds C2C: An Automated Deployment Framework for Distributed Applications on Multi-Clouds Flora Karniavoura, Antonis Papaioannou, and Kostas Magoutis Institute of Computer Science (ICS) Foundation for Research

More information

SGE & Amazon EC2. Chris Dagdigian 2008 OSGC Conference

SGE & Amazon EC2. Chris Dagdigian 2008 OSGC Conference SGE & Amazon EC2 Chris Dagdigian 2008 OSGC Conference Putting my $$ where my mouth is I m about to pay $.30 for the privilege of showing you this demo by firing up a 3-node cluster within EC2 Using cheap

More information

OpenStack Introduction. November 4, 2015

OpenStack Introduction. November 4, 2015 OpenStack Introduction November 4, 2015 Application Platforms Undergoing A Major Shift What is OpenStack Open Source Cloud Software Launched by NASA and Rackspace in 2010 Massively scalable Managed by

More information