1 Setting up a private cloud for academic environment with open source software Cloud Computing Course ITU of Copenhagen February 27 th, 2012
2 Who am I? Zoran Pantić Infrastructure Architect & Systems Specialist Corporate University of Copenhagen &
3 Agenda Non-technical part: A bit about the project Why OSS? Focusing on academic environments Technical part: UEC/Eucalyptus Reflections on hardware, software, network & redundancy Scaling out UEC Conclusion Questions? Video demo
4 Project: Implementing private cloud solution in academic environments Based on Open Source Software (OSS) Focus on the logistical and technical challenges, and strategies of setting up a private cloud for academic environment Goal - providing guidelines and tutorials for implementing private cloud solution in academic environments: Design of the server- and network infrastructure
5 Why OSS? In general: Lowering the costs (no licensing headaches!) Interchangeability & portability (general) Socio-organizational reasons UEC/Eucalyptus: Amazon AWS-like on-premise private cloud Using Amazons API Big community supporting it
6 Academic environments why private cloud? Usually, the budget is low, and the project should start as soon as possible Growing strongly: The need for processing large data volumes The need to conserve power by optimizing server utilization
7 Academic environments why private cloud? (continued) Private clouds: Higher ROI than traditional infrastructure More customizable Quick responses to changes in demands Rapid deployment Increased security Focus on an organization s core business Effort required for running a private cloud is having a downward tendency
9 Academic environments private cloud sociological challenges Sociological challenges, mostly political and economic: Existing structures oppose implementation of private cloud, Weak transparency of who is in charge of systems and economy, Researches cannot be market cost-effective, Administrators de facto in charge, instead of scientific groups, Tendency of IT department implementing things because they are interesting and fun, while maybe there is no need for those systems.
10 Academic environments private cloud technological challenges Technological challenges: Private cloud maturity, Problems porting of programming code, IT departments should be big enough, with enough expertize OSS: community cannot fix all your problems
11 Suggestions for implementing cloud solutions in academic environments To determine the needs and their nature, consult the professors that are in charge of the project (and its funding), Once started, implementation should be top-down steered, A test case should be designed and implemented, Researchers should be allowed to thoroughly test the solution - free of charge, Make sure that implementation succeeds first time! In general - get a very clear picture of what services are to be offered, who will use them, what they will use them for, and how!
12 Focus on academic environments Difference in implementing for infantry and supply troops Infantry - to support research, scientific computing and High Performance Computing (HPC) Supply - to support daily operational systems and tasks i.e. joint administration Bookkeeping, administration, Communications (telephony, e- mail, messaging) Infantry stateless instances vs. Supply stateful instances
13 Academic environments Infantry Uses non-standard & advanced research instruments Applicable in research, scientific computing and HPC, i.e.: Generally if users need VMs that they administer themselves (root access) - more appropriate to supply them with machines from private cloud, then giving access to virtual hosts behind firewall Organizations like ITU: for numerous different projects Organizations like DCSC: 1/3 of the jobs would be runnable on private cloud in HPC: Only in low end, for low memory and low core number jobs
14 Academic environments Infantry (continued) Summarized suggestions Have social psychology in mind as important factor Consult the professor in charge of money for the project Implement an open source solution UEC based on Eucalyptus, OpenStack, Joyent SmartOS (with both HW-level and OS-level virtualization!), OpenNebula, )
15 Academic environments Infantry UEC WebGUI
16 Academic environments Infantry HybridFox
17 Academic environments Supply Needs a stable and supported solution Summarized suggestions Have social psychology in mind as important factor Consult the system owner in charge of money for the project Implement a proprietary solution from reputable provider Microsoft Hyper-V, VMware Virtual Infrastructure, Sign a support agreement & agree a good SLA
18 Academic environments Supply VMware vsphere
19 UEC/Eucalyptus components UEC/Eucalyptus is an on-premise private cloud platform, designed as a distributed system - a modular set of 5 simple elements: Cloud Controller (CLC) Walrus Storage Controller (WS3) Cluster Controller (CC) Storage Controller (SC) Node Controller (NC)
21 Cloud Controller (CLC) Entry point to Eucalyptus cloud web interfaces for administering the infrastructure web services interface (EC2/S3 compliant) for end users /client tools Frontend for managing the entire UEC infrastructure Gathers info on usage and availability of the resources in the cloud Arbitrates the available resources, dispatching the load to the clusters Only one per cloud (no redundancy)
22 Walrus Storage Controller (WS3) Equivalent to Amazon s S3 Bucket based storage system with put/get storage model WS3 is storing the machine images and snapshots Persistent simple storage service, storing and serving files
23 Cluster Controller (CC) Entry point to a cluster Manages NCs and instances running on them Controls the virtual network available to the instances Collects information on NCs, reporting it to CLC One or several per cloud Only one per cluster (no redundancy)
24 Storage Controller (SC) Allows creation of block storage similar to Amazon s Elastic Block Storage (EBS) Provides the persistent storage for instances on the cluster level, in form of block level storage volumes Supports creation of storage volumes, attaching, detaching and creation of snapshots
25 Node Controller (NC) Compute node ( work horse ) Controls the instances supported hypervisors: KVM (preferred) and Xen in open source version, and VMware (ESX/ESXi) in Enterprise Edition Communicating with both OS and the hypervisor running on the node, and Cluster Controller Gathers the data about physical resource availability on the node and their utilization, and data about instances running on that node, reporting it to CC One or several per cluster
26 Reflections on hardware Processor architecture: Definitely 64-bit for performance reasons Multiprocessor, multicore, hyper threading VT-x enabled Node Controllers is a must Intel VT or AMD-V virtualization extensions Disk configuration: Local disks: RAID 10 (storage limits soon reached) Preferably SAN (iscsi) open source, see Nexenta /Napp-it
27 Reflections on software Ubuntu versions: Newest new features, but less stability (more bugs) LTS (Long Time Support) for more stability or larger deployments
28 Reflections on network 2 or 3 networks: WAN, Cloud public & Cloud private Firewall: open source based pfsense - to make the whole environment independent of the network infrastructure / environment where it will be plugged in
29 Reflections on redundancy No redundancy available in UEC by design In case of software or hardware error on a component: no failover solution is available; Solution: adding a new server, and then restoring the data
30 Scaling out the environment CLOUD CLUSTER 1 CLUSTER 2 CLUSTER 3 NC NC NC NC NC NC
31 Suggested scaling out possibilities 2 physical servers Server 1: CLC/WS3/CC/SC Server 2: NC 3 physical servers: Server 1: CLC/WS3 Server 2: CC/SC Server 3: NC
32 Suggested scaling out possibilities 4 physical servers Server 1: CLC Server 2: WS3 Server 3: CC/SC Server 4: NC 5 physical servers Server 1: CLC/WS3 Server 2: CC1/SC1 Server 3: NC1 Server 4: CC2/SC2 Server 5: NC2
33 Conclusion & recommendations for private clouds based on open source Although still at an early stage, being hard to install, manage and maintain for a regular admin and have steep learning curve (admins & users), implementation is suggested, at affordable, smaller scale Implement on a current/modern hardware Keep the knowledge updated Keep software platform and hardware updated if possible Monitor & analyze costs, available features and complexity, compared to budget, needs and internal resources available Asses the implementation possibilities based on the analyses
34 Alternative public clouds More mature Well documented Rich with features Easy to use Examples: Amazon s initiatives for academic use: Amazon Education program with grants for research applications; Having a project, academic organization applies for a recurring grant, gets the approval within two weeks time, and starts using it immediately after. Locally in Denmark, CABO was willing to supply the project with resources.
36 Demo Demonstration of UEC environment and WebGUI
37 Thank you! Thank you for your attention! Still having questions?
Guidelines for Building a Private Cloud Infrastructure Zoran Pantić and Muhammad Ali Babar Tech Report TR-2012-153 ISBN: 978-87-7949-254-7 IT University of Copenhagen, Denmark, 2012 ITU Technical Report
Installing Ubuntu Enterprise Cloud in a Physical Environment Supplement (2) to Guidelines for Building a Private Cloud Infrastructure Zoran Pantid and Muhammad Ali Babar Tech Report TR-155 ISBN: 978-7949-256-1
Installing and Scaling out Ubuntu Enterprise Cloud in Virtual Environment Supplement to Guidelines for Building a Private Cloud Infrastructure Zoran Pantić and Muhammad Ali Babar Tech Report TR-2012-154
The Definitive Guide tm To Cloud Computing Ch apter 10: Key Steps in Establishing Enterprise Cloud Computing Services... 185 Ali gning Business Drivers with Cloud Services... 187 Un derstanding Business
Private Cloud Computing Essentials The 2X Private Cloud Computing Essentials This white paper contains a brief guide to Private Cloud Computing. Contents Introduction.... 3 About Private Cloud Computing....
Computer Science 13 (2) 2012 http://dx.doi.org/10.7494/csci.2012.13.2.103 J. Chwastowski R. Grzymkowski M. Kruk M. Nabożny Z. Natkaniec A. Olszewski H. Pa lka Z. Sobocińska T. Sośnicki M. Szostak P. Syktus
Green-Cloud: Economics-inspired Scheduling, Energy and Resource Management in Cloud Infrastructures Rodrigo Tavares Fernandes email@example.com Instituto Superior Técnico Avenida Rovisco
Relational Database Management Systems in the Cloud: Microsoft SQL Server 2008 R2 Miles Ward July 2011 Page 1 of 22 Table of Contents Introduction... 3 Relational Databases on Amazon EC2... 3 AWS vs. Your
Software and services are moving off corporate and organizational networks and into the cloud. The trends are clear and the pace is accelerating. But what exactly is the cloud? Can client-server software
Windows Server 2008 R2 Hyper-V Live Migration Table of Contents Overview of Windows Server 2008 R2 Hyper-V Features... 3 Dynamic VM storage... 3 Enhanced Processor Support... 3 Enhanced Networking Support...
www.jitae.org Journal of Information Technology and Application in Education Vol. 3 Iss. 3, September 2014 doi: 10.14355/jitae.2014.0303.03 Design of Cloud Services for Cloud Based IT Education Li Chao
An Oracle Technical White Paper May 2011 Oracle Optimized Solution for Enterprise Cloud Infrastructure Introduction... 1 Overview of the Oracle Optimized Solution for Enterprise Cloud Infrastructure...
Masaryk University Faculty of Informatics Master Thesis Database management as a cloud based service for small and medium organizations Dime Dimovski Brno, 2013 2 Statement I declare that I have worked
Front cover IBM SmartCloud: Building a Cloud Enabled Data Center Redguides for Business Leaders Pietro Iannucci Manav Gupta Learn how to choose the infrastructure as a service (IaaS) solution that best
DEDICATED vs. CLOUD: Comparing dedicated and cloud infrastructure for high availability (HA) and non-high availability applications Avi Freedman / Technical Advisor A white paper by TABLE OF CONTENTS Introduction
Systems Engineering at MITRE CLOUD COMPUTING SERIES Products to Build a Private Cloud Lawrence Pizette Geoffrey Raines July 2010 Executive Summary Federal information technology (IT) leaders who are seeking
A Comparison and Critique of Eucalyptus, OpenNebula and Nimbus Peter Sempolinski and Douglas Thain University of Notre Dame Abstract Eucalyptus, OpenNebula and Nimbus are three major open-source cloud-computing
Contents Introduction...1 Overview of x86 Virtualization...2 CPU Virtualization...3 The Challenges of x86 Hardware Virtualization...3 Technique 1 - Full Virtualization using Binary Translation...4 Technique
Cloud Computing Tutorial CLOUD COMPUTINGTUTORIAL by tutorialspoint.com tutorialspoint.com i ABOUT THE TUTORIAL Cloud Computing Tutorial Cloud Computing provides us a means by which we can access the applications
The Case for Cloud Computing Robert L. Grossman University of Illinois at Chicago and Open Data Group To understand clouds and cloud computing, we must first understand the two different types of clouds.
Estimating the Cost of a GIS in the Amazon Cloud An Esri White Paper August 2012 Copyright 2012 Esri All rights reserved. Printed in the United States of America. The information contained in this document
MASTER THESIS TITLE: Infrastructure as a Service (IaaS): Application case for TrustedX MASTER DEGREE: Master in Science in Telecommunication Engineering & Management AUTHOR: Alberto Guirao Villalonga DIRECTOR:
The Massachusetts Open Cloud (MOC) October 11, 2012 Abstract The Massachusetts open cloud is a new non-profit open public cloud that will be hosted (primarily) at the MGHPCC data center. Its mission is
Microsoft System Center 2012 R2 Why Microsoft? For Virtualizing & Managing SharePoint July 2014 v1.0 2014 Microsoft Corporation. All rights reserved. This document is provided as-is. Information and views