Isabell Sippli Cloud Architect, Lab Based Services IBM Software Group 2013 IBM Corporation

Transcription

1 1 Isabell Sippli Cloud Architect, Lab Based Services IBM Software Group Disclaimer This document represents the author's views and opinions. It does not necessarily represent IBM's position or strategy. Acknowledgements Thanks to Michael Fork (IBM) and Thomas Spatzier (IBM) for their contributions. 2 1

2 Agenda Introduction & Overview Components IBM and OpenStack Demo Orchestration and Other Current Topics 3 Agenda Introduction & Overview Components IBM and OpenStack Demo Orchestration and Other Current Topics 4 2

3 Goals for the next 90 mins get a basic understanding of OpenStack what is it and what not? how can I use it? why is it a cool thing?... see OpenStack in action... get an understanding for it s architecture... Hot topics recent OpenStack developments... OpenStack contribution processs... IBM s involvement... questions and discussion 5 OpenStack in a nutshell Working software that is constantly developed and enhanced A vivid community of developers and cloud computing experts, driven by different companies An increasing number of enterprises either base their cloud implementations on OpenStack or build on top of it! screenshot from openstack website 6 3

4 Overview working software OpenStack delivers software to address the heart of every Infrastructure-as-a-Service (IaaS) offering Compute Network Storage Each area is delivered by one ore more core projects(with fancy names like nova, glance, cinder... ) All core projects written in pythonand are licensed under Apache 2.0 Linux of the data center eliminate vendor lock-in, maintain workload portability Build a great engine, packagers make a great car (think Linux to RHEL/SUSE) Designed for commodity hardware 7 Openstack projects conceptual architecture (core) IaaS horizon Dashboard working software Provides sample UI Reference implementation of API usage swift Object Store use API to store image files glance Image Library use API to manage images nova Compute use API for volumes for instances cinder Block Storage use API for network connectivity for instances (was: quantum) Network keystone Identity use authentication service via API 8 4

5 OpenStack community vivid community More than people and 200 companies Active online community through mailing lists, IRC, wiki Semi-yearly design summits this week in Hong Kong Companies need to donate money AND people that ACTIVELY contribute 9 and many more History and releases vivid community Founded in 2010 as an opensource project by Rackspace Hosting and NASA 8 releases so far, almost bi-yearly Latest release Havana (10/2013) Next release Icehouse (Spring 2013) Each release new version of the existing core projects new core projects are released overall architectural picture might change 10 5

6 User stories some examples user stories 11 IBM Cisco WebEx ( people centric collaboration applications ) WebEx is hosted on an OpenStack based cloud environment in a private cloud environment High resilience requirements Uses OpenStack Nova, Neutron, Cinder, Horizon, Keystone, Glance Paypal: enabling a private cloud that helps the company s developers quickly respond to its customers increasing demands and constantly changing needs, while developing a stable platform for customers to pay for their purchases. to support key areas of mobility, payments, credit, merchants and the web operations for more than 117 million active registered accounts Rolled out in December 2011, Uses OpenStack Nova, Neutron, Cinder, Horizon, Keystone, Glance, Swift, Ceilometer, Heat Virtualization Manager vs Cloud Infrastructure Services Virtualization Manager Purpose optimized for longer-lived virtual machines managed by Server Administrator Centralizes enterprise server virtualization administration tasks High degree of flexibility designed to accommodate virtualization all workloads Significant focus on managing availability and QoS for long-lived workloads with level of isolation Characteristics derived from exploiting enterprise class hardware Cloud Infrastructure Services Purpose optimized for shorter-term virtual machines managed via end-user or automated process Decentralized control, embraces eventual consistency, focus on making good enough decisions High degree of standardization Significant focus on ensuring availability of control plane Characteristics driven by software 12 Examples: VMControl or VMware vcenter Example: OpenStack 6

7 Design tenets... delivering essential infrastructure on an available, scalable, elastic control plane 1. Scalability and elasticity are our main goals 2. Any feature that limits our main goals must be optional 3. Everything should be asynchronous 1. If you can't do something asynchronously, see #2 4. All required components must be horizontally scalable 5. Always use shared nothing architecture (SN) or sharding 1. If you can't share nothing/shard, see #2 6. Distribute everything 1. Especially logic. Move logic to where state naturally exists. 7. Accept eventual consistency and use it where it is appropriate. 8. Test everything. 1. We require tests with submitted code. (We will help you if you need it) Source: 13 Agenda Introduction & Overview Components IBM and OpenStack Demo Orchestration and Other Current Topics 14 7

8 OpenStack logical architecture horizon Dashboard horizon Horizon Database swift Object Store swift-proxy glance Image Library glance-api nova-compute nova Compute nova-api cinder Block Storage cinder-api Network server account container object account container object glance-registry glance hypervisor nova-scheduler nova- nova queue... cinder-volume cinder cinder-scheduler agents queue plugins keystone Identity keystone token backend (kvs etc) catalog backend policy backend identity backend (kvs etc) (custom,rules)(ldap,etc) 15 Structure from: Identity management: Keystone What it is for Authentication credential validation and data about users, tenants and roles keystone Identity keystone keystonetakes API requests and processes them and is the central entry point for token, catalog, policy and identity services Identity: authentication of users Token: generate token for further use in OpenStack APIs Policy: role and privileges Catalog: list of services and how to reach them Each backend service allows for plugging-in different implementations of a service (like LDAP for identity services) token backend (kvs etc) catalog backend (kvs etc) policy backend (custom,rules) identity backend (ldap,etc) 16 8

9 Compute - Nova What is it for: Manage virtualized server resources: CPU/Memory/Disk/Network Interfaces the heart of OpenStack, fairly complex and distributed nova-api: accepts and responds to end user compute API calls nova-compute:controls virtual machine instances via hypervisor's APIs XenAPI for XenServer/XCP, libvirt for KVM or QEMU, VMwareAPI for VMware, etc. nova-scheduler:determines host to place workload on Console services for graphical/text console Policies for quota-type restrictions nova-compute hypervisor nova Compute nova-scheduler nova- nova-api nova queue Nova control plane is built on queue and database RabbitMQ is default queue, MySQL the default DB Other qeueing systems and SQL databases supported Queueing system to connect various componets API requests are validated and placed on queue Responses flow back through queue SQL database stores most of the build-time and runtime state for a cloud infrastructure e.g. instance types that are available for use, instances in use, networks available and projects Single cell (1 Queue, 1 Database) typically scales from physical machines Cells can be rolled up to support larger deployments nova-compute hypervisor nova Compute nova-scheduler nova- nova-api nova queue

10 Network management: (was: quantum) What it is for: Managing network resources (virtual networks, ports/interfaces, addresses) Network -serversatisfies requests, does orchestration of components server pluginscontrol logic of network devices (hardware or software) agentsprovide drill down to hardware/software implementation can be deployed on a network hardware appliance agents queue plugins Many plugins and agents exist for various hardware and software network devices 19 Block Storage: Cinder What it is for: Provide volumes for Nova instances -manages the creation, attaching and detaching of the block devices to servers API allows for manipulation of volumes, volume types (similar to compute flavors) and volume snapshots cinder Block Storage cinder-api cinder-apitakes API requests and forwards them to cinder-volume instance for action cinder-volumeacts on requests and can drive storage provider cinder-schedulerselects blocks storage provider for creating the volume cinder-volume cinder cinder-scheduler 20 10

11 Object Store: Swift What it is for: Serve objects (files) and metadata Goal is scalability (scale-out), availability and reliability More so than performance Periodic replication of data to ensure reliability and availability swift Object Store Swift-proxy Swift-proxy handles API calls Takes incoming requests from Object API or raw HTTPs Utilizes components below, e.g. Upload files, serve files Account Container Object account container object Account maps users to swift accounts Container maps accounts to containers (folders) Object stores object data and metadata File systems used (extended attributes for metadata) Periodic replication services ensures consistency and availability through the cluster 21 Image Store: Glance What it is for: Image registry supports image discovery, image retrieval and image storage glance-api takes API requests and processes them glance Image Library glance-api glance-registry glance-registryfor image meta data management (store, process, retrieve, e.g. size, type) database to store the actual image metadata storage repository for the actual image files Various backends available, often swift is used Periodic replication services ensures consistency and availability through the cluster glance 22 11

12 Dashboard: Horizon What it is for: provides an end user and administrator interface to OpenStack services. sample implementation for using the OpenStack APIs horizon Dashboard horizon Horzion is a modular Django web application Logic and interaction with OpenStack APIs... separated from... presentation logic customizable Horizon Database Horizon database for application state most of relevant data consumed from other APIs 23 1st Annual OpenStack User Survey (April 2013)

13 1st Annual OpenStack User Survey Details (April 2013) 25 Agenda Introduction & Overview Components IBM and OpenStack Demo Orchestration and Other Current Topics 26 13

14 One example: SmartCloud Orchestrator: an open and scalable platform Self-Service Catalogue Orchestration Monitor Backup & Restore Security Compliance Image Management Multi tier Application Infrastructure-as-a-Service (IaaS) TOSCA Dev Tools SC Enterprise EC2 VMWare Power VM Xen Z VM Hyper-V 27 KVM Agenda Introduction & Overview Components IBM and OpenStack Demo Orchestration and Other Current Topics 28 14

15 Demo of OpenStack in action 29 Agenda Introduction & Overview Components IBM and OpenStack Demo Orchestration and Other Current Topics 30 15

16 AppSrv compute instance1 network DB compute instance2 storage Hot topics: recent developments in OpenStack Orchestrationhas been one of the most discussed topics at Havana summit in Portland and is receiving high interest from contributing companies Heat is evolving as a layer on-top of base OpenStack services that performs patternbased orchestration across base APIs Has become core project in current Havana release Various task scheduling and workflow as a service incubation is ongoing Convention - TaskSystem-as-a-Service proposal Marconi OpenStack Queuing and Notification Service Celery distributed task management Metering/Monitoringis getting centralized in Ceilometerproject to provide a common, pluggable monitoring and metering infrastructure, also as input for billing systems adds billable as-a-service capailities 31 Heat OpenStack Orchestration Service Template (currently AWS CloudFormation) Heat API Layer Heat API Layer Heat API Layer AppSrv DB Stack Heat DB Heat Engine Heat Engine Heat Engine compute instance1 compute instance2 storage network OpenStack base services compute network block storage object storage 32 16

17 OpenStack contribution getting started Legal pre-reqs: Each individual has to sign the Individual Contributor License Agreement (ICLA) (done via Launchpad account see below) Create a Launchpad account used for authentication / single-sign-on for all OpenStack process related tools Get engaged with the community Subscribe to mailing lists (openstack, openstack-dev) Follow IRC channels: each project has its own IRC channel where you can meet all key participants in a project (details in wikis) Start doing reviews to get familiar with code and build reputation in the community 33 OpenStack contribution process typical flow You have a great idea? Create a blueprint: Socialize your blueprint on the mailing list or IRC channels. Start writing a first patch to show how the idea can work. or You want to help improve OpenStack? Browse bugs: Become assignee for a bug and implement a fix. Good to get familiar with code and build reputation. 34 Figure taken from: 17

18 How to OpenStack Get a first experience with OpenStack at TryStack.org ( Public hosted offering to try OpenStack in a sanoxenvironment Allows you to try what it means to run workloads OpenStack based cloud Allows for getting a first impression on the dashboard/rest APIs Try a first installation based on one of the OpenStack-ready Linux distributions E.g. Ubuntu Server (already contains OpenStack Grizzly release) E.g. Red Hat based installation as described in Red Hat s RDO community Get more hands on experience and do some first development using DevStack ( Easy to set up all-in-one OpenStack environment based on current github repository content Can run locally in a VM on your laptop 35 Summary 1) OpenStack is working software, based on a great community, and increasingly used by major enterprises 2) OpenStack is based on a modular, flexible, scalable architecture worth taking a look at when creating distributed architectures on your own 3) OpenStack is a great example to learn what makes up an IaaS offering. 4) OpenStack is evolving to be more than just an IaaS platform orchestration, built-in metering etc

19 Essay topic OpenStack is the emerging cloud operating system to provide Infrastructure, Network and Storage-As-A-Service via prescriptive, well-defined APIs. Describe the overall OpenStack architecture, while highlighting concrete examples of OpenStack s design tenets. Discuss characteristics of a good API in general and compare the OpenStack APIs against these characteristics. OpenStack is the emerging cloud operating system to provide Infrastructure, Network and Storage-As-A-Service via prescriptive, well-defined APIs. Describe the overall OpenStack architecture, while highlighting concrete examples of OpenStack s design tenets. Discuss the main reasons of OpenStack s success. Highlight the projects that will grow the most in the next 2 years according to your opinion, and share your reasoning. 37 References Fork, Michael J: What it is, who contributes, where it is used, how we are participating, IBM internal, PPT Presentation, April 2012 Fork, Michael J: OpenStack Deeper Dive, IBM internal, PPT Presentation, April 2012 Pepples, Ken: OpenStack Folsom Architecture, (last accessed 10/27/2012) Various authors: OpenStack Cloud Software, Oct 2013 (last accessed 11/03/2013), especially Vogels, W.: Eventually consistent, ACM Queue vol. 6, no. 6, December 2008 Wilder, Bill: Cloud Architecture Patterns, O Reilly Media,

20 39 20