OPENSTACK 4 APPLICATION DEVELOPMENT CONVERTING A LEGACY APP TO AN OPENSTACK APP BUILDING APPS FROM SCRATCH OPENSTACK APP DESCRIPTION AND DEPLOYMENT

Transcription

1

2

3 CONTENTS INTRODUCTION WHO THIS BOOK IS FOR WHAT THIS BOOK COVERS HOW THIS BOOK IS STRUCTURED WHAT YOU NEED TO USE THIS BOOK CONVENTIONS SOURCE CODE ERRATA P2P.WROX.COM PART I OPENSTACK OVERVIEW 1 INTRODUCING OPENSTACK WHAT IS CLOUD COMPUTING? WHY SHOULD I CARE? UNDERSTANDING THE ARCHITECTURE SUMMARY 2 UNDERSTANDING THE OPENSTACK ECOSYSTEM: CORE PROJECTS IDENTITY COMPUTE STORAGE IMAGING DASHBOARD NETWORKING BRINGING IT ALL TOGETHER SUMMARY 3 UNDERSTANDING THE OPENSTACK ECOSYSTEM: ADDITIONAL PROJECTS OPENSTACK HEAT OPENSTACK DATABASE AS A SERVICE: TROVE DESIGNATE: DNS AS A SERVICE MAGNUM MURANO: APPLICATION AS A SERVICE CEILOMETER: TELEMETRY AS A SERVICE SUMMARY PART II DEVELOPING AND DEPLOYING APPLICATIONS WITH

4 OPENSTACK 4 APPLICATION DEVELOPMENT CONVERTING A LEGACY APP TO AN OPENSTACK APP BUILDING APPS FROM SCRATCH OPENSTACK APP DESCRIPTION AND DEPLOYMENT STRATEGIES SUMMARY 5 IMPROVING ON THE APPLICATION FAILURE SCENARIOS HOSTNAME AND IP ADDRESSING SCALING IMPROVING OUR APPLICATION SUMMARY 6 DEPLOYING THE APPLICATION BARE METAL, VIRTUAL MACHINES, AND CONTAINERS ORCHESTRATION AND CONFIGURATION MANAGEMENT MONITORING AND METERING ELASTICITY UPDATING AND PATCHING SUMMARY BOOK WRAP UP TITLE PAGE COPYRIGHT ABOUT THE AUTHOR ABOUT THE TECHNICAL EDITORS CREDITS ACKNOWLEDGMENTS EULA

5 List of Illustrations Chapter 1 Figure 1.1 Figure 1.2 Figure 1.3 Figure 1.4 Figure 1.5 Figure 1.6 Chapter 2 Figure 2.1 Figure 2.2 Figure 2.3 Figure 2.4 Figure 2.5 Figure 2.6 Figure 2.7 Figure 2.8 Figure 2.9 Figure 2.10 Figure 2.11 Figure 2.12 Figure 2.13 Chapter 3 Figure 3.1 Figure 3.2 Figure 3.3 Figure 3.4 Figure 3.5 Figure 3.6 Figure 3.7 Figure 3.8

6 Figure 3.9 Figure 3.10 Figure 3.11 Figure 3.12 Figure 3.13 Chapter 4 Figure 4.1 Figure 4.2 Figure 4.3 Figure 4.4 Figure 4.5 Figure 4.6 Chapter 5 Figure 5.1 Figure 5.2 Figure 5.3 Figure 5.4 Figure 5.5 Figure 5.6 Figure 5.7 Figure 5.8 Figure 5.9 Figure 5.10 Chapter 6 Figure 6.1 Figure 6.2 Figure 6.3 Figure 6.4 Figure 6.5

7

8 INTRODUCTION OpenStack is a set of software packages that manage virtualized resources, including computing, networking, and storage. It enables you to create and destroy virtual machines, connect them together with private networks, provide network-based storage, and make them available to the rest of your network and the world. OpenStack provides consistent, uniform API services for all of this, hiding hypervisor and vendor specific details from the applications that are using the APIs. It also provides a user interface, built on top of the same APIs, that allows users to see and manage their virtual resources.

9 WHO THIS BOOK IS FOR This book is for application developers that are interested in learning more about OpenStack and how it will transform the application design and development process. It is for someone who is new to the cloud environment, who wants a broad understanding of that environment, as well as a deep enough knowledge to make practical use of OpenStack.

10 WHAT THIS BOOK COVERS This book will provide a broad understanding of cloud concepts and how they fit into the life of an application developer. It will drill in deeply to the OpenStack services that are most important to an application developer, and show you how these services will change not only how you deploy applications, but also how you design them. It will provide detailed information on each service, and provide examples of how each service may be used by an application developer.

11 HOW THIS BOOK IS STRUCTURED This book was written in two parts. Part 1 provides an overview of OpenStack. The purpose of this part is to lay the groundwork, covering all of the OpenStack technologies and what is most important. Part 2 takes the reader through developing and deploying applications with OpenStack. In this part you will build an example on top of OpenStack that drills down much deeper on the technologies, provides tips, and helps you learn about OpenStack through the lens of these same technologies. Here is a list of the chapters: Part I: OpenStack Overview Chapter 1: Introduction to OpenStack Chapter 2: Understanding the OpenStack Ecosystem: Core Projects Chapter 3: Understanding the OpenStack Ecosystem: Additional Projects Part II: Developing and Deploying Applications with OpenStack Chapter 4: Application Development Chapter 5: Improving on the Application Chapter 6: Deploying the Application

12 WHAT YOU NEED TO USE THIS BOOK You should understand the basics of application development - how applications are composed of multiple servers like web servers, application servers, and database servers. You do not need any cloud-specific knowledge, though you should be aware of what virtualization and virtual machines are, and have a basic understanding of networks.

13 CONVENTIONS To help you get the most from the text and keep track of what s happening, we ve used a number of conventions throughout the book. Examples that you can download and try out for yourself generally appear in a box like this: EXAMPLE TITLE This section gives a brief overview of the example.

14 Source This section includes the source code. Source code Source code Source code

15 Output This section lists the output: Example output Example output Example output NOTE Notes indicates notes, tips, hints, tricks, or and asides to the current discussion. As for styles in the text: We highlight new terms and important words when we introduce them. We show code within the text like so: persistence.properties.

16 SOURCE CODE As you work through the examples in this book, you may choose either to type in all the code manually, or to use the source code files that accompany the book. All the source code used in this book is available for download at Specifically for this book, the code download is on the Download Code tab at: and at: You can also search for the book at by ISBN (the ISBN for this book is ) to find the code. And a complete list of code downloads for all current Wrox books is available at Note Because many books have similar titles, you may find it easiest to search by ISBN; this book s ISBN is Once you download the code, just decompress it with your favorite compression tool. Alternately, you can go to the main Wrox code download page at to see the code available for this book and all other Wrox books.

17 ERRATA We make every effort to ensure that there are no errors in the text or in the code. However, no one is perfect, and mistakes do occur. If you find an error in one of our books, like a spelling mistake or faulty piece of code, we would be very grateful for your feedback. By sending in errata, you may save another reader hours of frustration, and at the same time, you will be helping us provide even higher quality information. To find the errata page for this book, go to And click the Errata link. On this page you can view all errata that has been submitted for this book and posted by Wrox editors. If you don t spot your error on the Book Errata page, go to and complete the form there to send us the error you have found. We ll check the information and, if appropriate, post a message to the book s errata page and fix the problem in subsequent editions of the book.

18 P2P.WROX.COM For author and peer discussion, join the P2P forums at The forums are a Web-based system for you to post messages relating to Wrox books and related technologies and interact with other readers and technology users. The forums offer a subscription feature to you topics of interest of your choosing when new posts are made to the forums. Wrox authors, editors, other industry experts, and your fellow readers are present on these forums. At you will find a number of different forums that will help you, not only as you read this book, but also as you develop your own applications. To join the forums, just follow these steps: 1. Go to and click the Register link. 2. Read the terms of use and click Agree. 3. Complete the required information to join, as well as any optional information you wish to provide, and click Submit. 4. You will receive an with information describing how to verify your account and complete the joining process. NOTE You can read messages in the forums without joining P2P, but in order to post your own messages, you must join. Once you join, you can post new messages and respond to messages other users post. You can read messages at any time on the Web. If you would like to have new messages from a particular forum ed to you, click the Subscribe to this Forum icon by the forum name in the forum listing. For more information about how to use the Wrox P2P, be sure to read the P2P FAQs for answers to questions about how the forum software works, as well as many common questions specific to P2P and Wrox books. To read the FAQs, click the FAQ link on any P2P page.

19

20 PART I OpenStack Overview CHAPTER 1: INTRODUCING OPENSTACK CHAPTER 2: UNDERSTANDING THE OPENSTACK ECOSYSTEM: CORE PROJECTS CHAPTER 3: UNDERSTANDING THE OPENSTACK ECOSYSTEM: ADDITIONAL PROJECTS

21

22 1 Introducing OpenStack WHAT S IN THIS CHAPTER? Models of cloud computing Relevance of cloud computing to application developers Why OpenStack is a good cloud platform choice How OpenStack is put together

23 WHAT IS CLOUD COMPUTING? There is so much hype around cloud computing that it is often difficult to get a clear sense of what anyone means by those words. Is it just virtualization? Is it Software-as-a-Service (SaaS), such as Microsoft s Office 365 and Salesforce.com? Or is it the ability to get a virtual machine instantly from Amazon Web Services (AWS) or Azure? And what about online storage such as Dropbox? Types of Cloud Computing The reality is that cloud computing refers to all of these things just described and more. The National Institute of Standards and Technology (NIST) has come up with an official definition based upon five key components: on-demand selfservice, broad network access, pooled resources, elasticity, and metered service. In general, these characteristics may be provided in several different models. These models help sort out the confusion and hype. In fact, these can be thought of as layers in a stack, with each layer being built on top of the previous one (see Figure 1.1). Figure 1.1 In Figure 1.1, Manually Provisioned Infrastructure represents the traditional method of building your information technology infrastructure this is not cloud computing. In this environment, physical machines are racked, connected, and configured on a one-by-one basis. This provides complete control, but requires substantial time and effort to build out, or to change when necessary. Of course, all clouds need to run on physical gear at some point, so this provides the basic foundation for everything else. One of the keys to making cloud computing successful, however, is to move the complexity out of this layer and up higher in the stack. Infrastructure-as-a-Service (IaaS) is the most basic layer in the cloud computing

24 stack. This is OpenStack s primary focus, as well as the primary focus for AWS. It enables automated or self-service provisioning of compute, networking, and storage. Typically, these resources are provided as Virtual Machines (VMs), but you could also use it to spin up bare metal servers (i.e. physical hosts). This is known as Metal-as-a-Service, and OpenStack provides a project for managing this service as well. Alternatively, you can also spin up containers rather than VMs or bare metal servers. The essential point is that it enables the provisioning of compute instances, with (optionally) attached networking and storage. Platform-as-a-Service (PaaS) builds on top of IaaS to enable the provisioning of applications, rather than simply the infrastructure that might be used to run the application. So, a PaaS provides core common services needed by applications, along with the machinery to configure and deploy applications to use those services. A PaaS typically will provide a complete application stack (web server, application server, database server, etc.) into which you can easily deploy your application. Heroku ( is an example of a popular PaaS for applications built with a variety of standard frameworks, such as Ruby-on-Rails. With Heroku you can deploy your application to the Internet with a simple git push. As the application author and deployer, you don t need to worry about configuring and deploying the different tiers, or even worry about how to scale them. If you follow the Heroku conventions, everything is handled by the PaaS. Software-as-a-Service (SaaS) is the layer farthest from the underlying physical infrastructure. It may be built on IaaS or a PaaS, but need not be the point is the user never really knows. This is the simplest form of cloud computing from the point of view of the user because they have no insight into the actual mechanics or systems behind the service. It s just a service they use. Often this is provided in the form of a website, such as Salesforce.com. But you can also get lower-level services such as Database-as-a-Service, where you simply request via an API (or website) for a database with certain parameters, and are given an IP and port to connect to. As a user of the service, you don t need to worry about how to scale that service though you will need to pay more as your use of the service increases. Put succinctly, IaaS provides the tools to build your systems from the ground up. PaaS allows you to deploy your applications, without needing to worry about the underlying infrastructure. SaaS allows you to buy your applications you do not even need to deploy or manage them at all. This is a steady progression of decreasing control and complexity, while increasing direct business value. While these are general models for cloud computing, in reality the distinctions between them are not always crystal clear. The relationship of SaaS to PaaS in particular can be complicated. A specific, complex Software-as-a-Service may use PaaS or even other more granular Software-as-a-Service. Even a PaaS may assemble lower-level pieces as a collection of software services. For example, most services will require an identity management (authentication, authorization, and accounting) service. This identity service is one of the key features a PaaS provides

25 to applications. However, there is no reason that service cannot be, in turn, provided by some external SaaS! In this case, a key function of the PaaS is provided via a low-level SaaS. Cloud Infrastructure Deployment Models In addition to the functionality provided by a cloud, there are several different deployment models for clouds. Public clouds are the ones familiar to most developers. These cloud services are made available to the general public for a fee. The fee is generally on a usage basis, enabling organizations to utilize their operating budgets rather than their capital budgets. The customers have no need to maintain or operate the hardware or cloud infrastructure, leaving that responsibility completely to the cloud operator. Amazon Web Services (AWS) is currently the largest public cloud and dominates the industry. Microsoft and VMware also operate public clouds, and a number of service providers do as well. Rackspace, in particular, provides an OpenStackbased public cloud, and is one of the primary contributors to the OpenStack project. Private clouds, on the other hand, are internal to an organization. They represent the evolution of the traditional corporate data center. Only internal customers within the enterprise, and perhaps close partners, use private clouds. The corporate IT department or a contractor will purchase, setup, and maintain the hardware and software for the cloud. The cloud infrastructure may use chargeback to distribute costs among the business units, but the cloud itself is still dedicated to the single enterprise. Organizations may operate private clouds for a number of reasons. The cost of a private cloud, if well run, may be less than utilizing the public clouds. Additionally, many industries have security or regulatory reasons that disallow the use of a public cloud for many workloads. These organizations are required to run those workloads in a private cloud. See Figure 1.2 for a look at the structure of public, private, and hybrid clouds.

26 Figure 1.2 Hybrid clouds combine both private and public clouds. The goal with hybrid clouds is to keep general operating costs low by using the private cloud for most of the workloads, but to enable spillover into the public cloud when necessary. The spillover could happen due to capacity reasons perhaps during the holiday season your private cloud doesn t have enough capacity or for disaster recovery. This model avoids the capacity constraints of a private cloud while still keeping costs under control.

27 WHY SHOULD I CARE? As an application developer or architect, you may wonder why does all of this matter to me? All of this discussion covered so far focuses on the reason a business may want to move to the cloud. But why should that affect the application developer? The answer lies in a couple of different areas: the effect on the development process, and the effect on your application architecture. Cloud services enable much more efficient processes for managing development, test, and production environments. These updated processes and methods represent the DevOps mentality applying standard software development practices, such as source code version control, to the operational aspects of the application. This means capturing all of the configuration and deployment information in scripts and templates, and controlling their changes just as you would application code. Scripts and templates can be built that produce a complete application environment. These can be used to automatically deploy not only the application, but also infrastructure required for the application, including virtual machines, networking, firewalls, load balancers, domain name services you name it, and someone is working on making it available as-a-service. By automating the creation and destruction of these environments, you can ensure consistency between development, test, and production environments. For complex applications with many different services running on different machines, this can be a dramatic time saver. OpenStack, and as-a-service thinking in particular, will also end up changing the software and deployment architectures of your application. By relegating the common and routine functions to the cloud infrastructure, you free your time and thought to focus on the most important thing your application s functionality. For example, a traditional application that allows large file uploads will need to designate temporary and permanent storage locations for those files, and manage the storage resources to ensure that the disk doesn t fill. The system administrator or deployer will need to devise a strategy to backup that data or replicate it to other data centers. But with the right cloud platform, you can simply delegate that function to the infrastructure, and get all of the benefits without devoting special effort. Designing your application to work with the cloud services also dramatically simplifies scaling the application. The scalability of the individual services becomes the responsibility of the cloud operator, not the application developer or administrator. As long as the application makes effective use of those services, it will scale as needed with little to no work from the developers themselves. Being able to utilize as-a-service functions is one way your design will shift. Another is to plan for horizontal scaling rather than vertical scaling. That is, scaling by adding more machines (horizontally) rather than creating bigger

28 machines (vertically). With most applications today, it is easiest to scale by getting a bigger, faster machine. This locks you into planning for peak capacity of each application individually. For each application you need to provision the largest machine you may need at peak load. But with applications built for the cloud, you instead scale by adding more machines. These machines can be smaller, and with cloud automation, can be added, removed, or resized as needed. This ability to scale up and down as needed is called elastic scaling, and is one of the key features of cloud computing. A frequently used analogy is that traditional servers are like pets, while cloudbased servers are cattle. This describes a necessary shift in mentality for a traditional application architect. The idea is that a pet is unique and special, with its own unique name. A lot of resources are spent to raise and nurture one, and if it is sick, it will be nursed back to health. Cattle, on the other hand, are not treated specially or carefully raised. They are treated en masse they are given numbers, not names and a sick one is culled to prevent any spread of disease through the herd. The implication here is that cloud-based servers should be disposable and easily re-deployed, and not require careful hand configuration. That way, if there is a problem with one, you do not spend time trying to figure it out and fix it you simply replace it with a new one. This is the logical extension of the ability to scale elastically. Why take the time to figure out what s wrong with a machine when it s behaving badly? Just pull it out of the application and replace it with a new one while you debug the problem (not to fix that machine, but to prevent the issue in the future). What Is OpenStack? OpenStack bills itself as a cloud operating system. Fundamentally, it solves the IaaS problem. It provides the ability to abstract the physical compute, storage, and networking resources into pools. Those resources can then be divvied up among users in a secure way. Users only need to pay for what they are using, rather than having to provision their applications for peak load. OpenStack is a collection of open source software projects, backed by a non-profit organization, the OpenStack Foundation. These projects work together to provide a consistent API layer, while enabling the actual services to be provided by a variety of different vendor or open source implementations. At the core, these services include the functionality you need to run a cloud, that is, the ability to spin up virtual machines, the ability to allocate, manage, and share storage among those machines, and the ability enable these machines to communicate with one another securely over the network.

29 KEEPING TRACK OF RELEASES OpenStack has official releases every six months. In order to make it easier to keep track of all these releases, they are given names in alphabetical order. Below is the name of each release, and its release date, through the Liberty release. Austin: October 2010 Bexar: February 2011 Cactus: April 2011 Diablo: September 2011 Essex: April 2012 Folsom: September 2012 Grizzly: April 2013 Havana: October 2013 Icehouse: April 2014 Juno: October 2014 Kilo: April 2015 Liberty: October 2015 In addition to the release name, each release is identified by the year and release during that year <year>.<release>.<patch>. For example, Kilo is also known as , as the first release in Patch releases for Kilo are , , etc. The second major release of 2015 is Liberty, which is also known as All of these services are accessible via RESTful APIs, as well as command-line interfaces and a web-based user interface called Horizon. Horizon is convenient for setting up things on an ad-hoc basis, but doesn t offer the full capabilities of the APIs and of course the APIs and CLI tools can be easily scripted (see Figure 1.3).

30 Figure 1.3 The next table shows the major services provided by OpenStack, along with their names. OpenStack community members will usually refer to each service by its name, so it s helpful to see them all in one place and get a handle on what each one does. In fact, there are many more services, but these are the most common ones you will find.

31 Name Service Description Horizon Dashboard A graphical user interface for managing your cloud Keystone Identity Authentication, authorization, and OpenStack service information Nova Compute Spin up, manage, and terminate virtual machines Cinder Swift Block Storage Disk volumes (that outlive an instance) and snapshots of instances Object Storage Shared, replicated, redundant storage for images, files, and other media accessible via Hypertext Transfer Protocol (HTTP) Neutron Network Provide secure tenant networking Glance Image Provide storage and access to VM images and snapshots Heat Orchestration Spin up groups of machines, networks, and other resources via templates Designate DNS Create domains and records in the DNS infrastructure Ceilometer Telemetry Monitor resources usage across the cloud Trove Database Provide access to private tenant databases Ironic Bare Metal Spin up instances on physical hardware Magnum Containers Manage containers within instances Murano Application Deploy packaged applications across multiple instances Sahara Data Processing Cluster Provides a Hadoop or Spark cluster as a service A default installation of OpenStack will include reference versions of each service. For example, by default an OpenStack cloud will use the Kernel-based Virtual Machine (KVM) hypervisor to manage virtual machines. One of the most important aspects of the OpenStack architecture, however, is the driver or pluginbased nature of each service. With this design, you can use an implementation other than the reference one. In your cloud, you can swap out KVM with ESXi, Xen, or other hypervisors. The APIs used to launch and manage VMs remain the same, regardless of the underlying hypervisor. This same concept extends across OpenStack services, enabling the same APIs with different service implementations. This level of flexibility behind the scenes, while providing a consistent API, is one of the keys to the success of OpenStack. Users can build their applications and automation on top of OpenStack, without having to worry that they are locking

32 themselves into a single backend provider of computer, networking, or storage. The APIs won t change even if they swap out the backend. OpenStack is frequently used in enterprises for private clouds, though there are some public cloud services that are based on it. There are also companies that will create and operate a private OpenStack cloud for you within their data centers. In this case, the hardware is not shared with other customers, so you have the predictability and security of the private cloud but do not have to find and hire the experts to maintain it. Even in private cloud environments, OpenStack is a multi-tenant cloud platform. This means that multiple users or groups of users tenants can utilize the physical resources of the cloud, while keeping all of their virtualized resources private. For a tenant, the OpenStack environment appears, for the most part, to be theirs and theirs alone. But for the operator, the underlying physical resources and software systems are shared. In OpenStack, tenants are also sometimes referred to as projects. In a multi-tenant OpenStack cloud, each tenant is allocated a quota for the various types of resources that may be used. The quota provides a maximum limit for that tenant for that particular resource. You will have a quota for CPUs, memory, storage, networks, subnets, and floating IPs, among other resources. This prevents any single tenant from consuming all of the resources. Why OpenStack? There are a number of cloud management platform options out there. The most obvious and dominant player is VMware with their vrealize suite of software. So, why should you take your time to learn about OpenStack rather than vrealize, AWS, Azure, CloudStack, or any of the other solutions? About 15 years ago, IT professionals faced a very similar set of questions about Linux and proprietary UNIX systems. Solaris, HP-UX, AIX and their competitors were solid, well known, and widely deployed products, whereas Linux was a graduate student s project that was difficult to install and operate and was fairly immature, with driver and other compatibility issues. It was not clear at all at the time that spending effort learning and understanding Linux was worth it. History though, has proven that such a choice would have been the right one. All of those expensive, proprietary UNIX implementations have lost their value proposition they really don t have much that is unique to offer anymore. Linux has continued to grow and has taken over most of the environments where those systems once thrived. This isn t just a simple analogy. There is a relentless pressure in this industry to reduce costs, and to increase the velocity of feature delivery deliver more, faster, and cheaper. The way to achieve more, faster is standardization. This is the same basic principle as building libraries and frameworks in programming. A standard architecture behaves in a predicable manner, providing core services on which you

33 can rely and build. There is no need to repeat the process of developing that architecture over and over, allowing you to focus on the new functionality. The way you achieve cheaper is to make those standards open and free. This combination of open and standard leads to commoditization essentially the development of interchangeable components that are the same regardless of the manufacturer or vendor. Commodities imply a lot of competition, and there is little or no product differentiation for which to charge extra. This drives down the costs dramatically. Linux has both of these characteristics open and standard in UNIX-like operating systems, and that is why it won. Not because it was better, but because it was cheaper and faster to use as a base for building new functionality. Linux is just one example, of course. This story has repeated over and over in the technology industry. With machine architectures we have the x86 platform, and standard architectures for memory, disks, and serial bus-based peripherals. In fact, if you take the broader view, you can see that the commoditization has continuously moved up the value chain. It started with hardware, moved to operating systems, and these days even sophisticated databases and distributed system components are being commoditized. In databases, we used to have Informix, DB2, Oracle, Sybase, and others. But MySQL and PostgresSQL are open and standard, and they have completely dominated the low-end of the database market. Oracle still leads in the high-end, and is able to provide value in those more specialized environments, but as the open source products improve, the space for the proprietary vendors constricts. In some way, cloud computing is the culmination of this commoditization process in the industry. Broadly, you can think of the revolution happening in the computing industry as a refocusing of the industry on the core functions of computing. The abstraction of the computing infrastructure into simply compute, storage, and networking components, and breaking of these out from being vertically integrated, to horizontally integrated, is truly transformative. It brings full commoditization to these elements, which are the basic foundation of the industry. Cloud platform management will follow the same pattern. The proprietary platforms like vrealize will thrive for a time, but in the long run the open and standard systems will win. While there may always be a place for the proprietary solutions in more specialized environments, the most common platforms will be open source. You can see this already happening: the Zenoss 2014 State of the Open Source Cloud Survey ( found that 30 percent of respondents were already using an open source cloud, up 72 percent from 17.2 percent in Another 34 percent of the respondents planned to implement an open source cloud in the future. Understanding this gives you an advantage to focus on the eventual winner, instead of chasing what will ultimately be a setting star.

34 There are several open, standard cloud management platforms. So even if you believe that the bet on open and standard is the way to go, why should you bet on OpenStack? The answer here is simple momentum. OpenStack is by far the most widely used and supported open source cloud management platform, and it has the largest community of developers and vendors push it forward. The same survey mentioned above found that 69 percent of respondents with an open source cloud were using OpenStack in 2014, up from 51 percent in An amazing 86 percent of those considering an open source cloud deployment are looking at OpenStack. The OpenStack developer and user communities have grown dramatically as well. The OpenStack Foundation 2014 Annual Report ( provides detailed insight into this growth. In 2013, the best quarter for mean monthly active developers had 391 developers in 2014 this measure was up 45 percent to 569 developers. Large investments from HP, Cisco, RedHat, IBM, Dell, Mirantis, Rackspace, and many other vendors have driven this surge. The incredible growth in the number of users, developers, and other interested parties can be seen from the attendance at the twice annual OpenStack Summits, seen in Figure 1.4 (source: openstack.org). Figure 1.4 Clearly OpenStack has the momentum to succeed.

35 UNDERSTANDING THE ARCHITECTURE OpenStack is built on a loosely coupled architecture. Each component is built independently and runs its own services. These services may be distributed among a number of different machines with different responsibilities. This enables scaling of particular functions, by adding machines with particular roles. It also enables redundancy; a highly available deployment will contain several of each type of machine. Software Architecture Individual components interact with one another via well-defined application programming interfaces (APIs) typically based on representational state transfer (REST) conventions, though in some cases using remote procedure calls (RPC) or notifications over a message bus. Typically, these services will maintain data in a relational database usually MySQL or PostgreSQL. The message bus and database are shared across services, but the interactions between those services remain clearly delineated. This enables different services to grow and change independently from the others, so long as they provide backward-compatibility in the APIs. Each of the major services compute (Nova), networking (Neutron), block storage (Cinder), etc. have several internal processes and components. Generally, they will each have an API service that provides an HTTP-based RESTful API. This API service will communicate with the other components via the message bus. The Horizon service is a web-based UI that interacts with the various services. Similarly, there are command-line tools to interact with each service. These tools are optional; you can build your own interface directly to the service APIs if you wish. Horizon and the official CLI clients do not have any special access; everyone uses the same APIs. Each client really only needs to be informed of the location of Keystone, the identity service. This service contains a catalog of all services and API endpoints available in the OpenStack platform (see Figure 1.5).

36 Figure 1.5 In Figure 1.5, what you see is a simplified depiction of how the services interact. Each service has an API component, which communicates with Keystone s API via HTTPS to provide authentication and authorization information. Each API service uses the message bus to communicate with several other processes for that service (just called Services in the diagram). As needed, these downstream service processes will call the APIs of other services. For example, Nova will call the Neutron API to acquire a port on a particular network. Deployment Architecture How are all of these different pieces of software deployed on the hardware? This is actually pretty flexible. For development or just experimentation, you can even run everything on a single machine. However, a more typical deployment will have several controller nodes (for high availability purposes), along with additional network, compute, and controller nodes. Each high-level service (compute, networking, storage, and others) consists of multiple daemons (background processes). These daemons are spread out across the various types of nodes. That is, you do not run individual services on individual nodes, but rather spread each service out across different types of nodes. For example, all of the services share the database and messaging components (typically MySQL and RabbitMQ, respectively). You may run these each on

37 separate clusters, with each cluster spread over different failure domains. Additionally, you may have several physical nodes that provide the API endpoints, behind a physical load balancer. Different daemons for Nova and Neutron will be spread across the network and compute nodes. Figure 1.6 shows a simplified diagram of this layout. Figure 1.6 Notice the different types of nodes in Figure 1.6. Compute nodes run the hypervisor and therefore the actual VM instances, as well as provide the ephemeral storage for instances. They will also run Neutron networking agents to manage the connectivity between VMs (called east-west traffic). The network nodes usually provide the connectivity between VMs and outside the cloud (called north-south traffic), as well as the advanced network services like load balancing and VPN access. Depending on the choices made by the administrators and users, there may be agents providing network routing services on the network nodes, directly on the compute nodes, or both. The block storage nodes provide volume services to the instances that is, they provide access to persistent storage for disk volumes that can be attached and

38 detached to instances. Clouds that offer object storage will also have separate clusters for that. Object storage provides shared, replicated, redundant storage for images, files, and other media accessible via HTTP. Various segregated networks connect all of these nodes. Every node is accessible via the management network, which is used for different parts of OpenStack to communicate with one another. All of the message bus, database, and crossproject API traffic go over the management network. The data network connects all of the compute nodes, network nodes, and block storage nodes. The internal cloud tenant traffic is carried on this network, whereas the external network provides access to the outside world. Since the compute nodes do not communicate with the outside world, but only with other nodes in the cloud infrastructure, they need not have connectivity to the external network, but only need access to the data network. Only the network nodes need to connect to the external network. Finally, some installations will use an API network, which provides access between the outside world and the OpenStack end points (API and Horizon), separate from the external network used by tenants. Pros and Cons This architecture provides a great deal of flexibility. This enables the scalability by letting the cloud operator deploy additional nodes to scale the infrastructure. It also allows the ability to create highly available services, since you can split each service out and make them redundant across failure domains. However, it is very complex, and can be quite difficult to set up and maintain. As a user of the cloud, this will be transparent to you. But a properly run cloud will have enough redundancy built in that you can expect a high-level of reliability from the OpenStack infrastructure. Another substantial benefit to this architecture is avoiding vendor lock-in. Each service provides a plugin or driver-based architecture. This enables each service to work with any number of vendor platforms to provide the actual service. For compute, you can use the default KVM hypervisor, ESXi, Xen, or one of many other hypervisor choices. The networking service defaults to using Open vswitch to provide Layer 2 (the data link or MAC address layer) connectivity, and the Linux networking stack (iptables, routing, and namespaces) to implement Layer 3 (IP layer) functionality. However, there are more than 20 different vendor plugins to swap all or part of that default implementation. In fact, these vendor implementations can be used at the same time, in the same cloud. By avoiding vendor lock-in, OpenStack enables more competition between the vendors, pushing down prices in the market. The ability to use multiple vendors at once makes transitioning from one vendor to another more feasible, and also allows the choice of vendor for solving specific use cases. An interesting feature introduced with the Kilo release of OpenStack is federated identity. This takes the distributed nature of OpenStack and allows it to span

39 across multiple clouds, even from different providers. Two cloud providers can set up a trust relationship, enabling users of one provider to use the same credentials with another, trusted provider. Thus the same workload management tools you use for a single cloud can theoretically be used to manage workloads across multiple clouds. For capacity burst use cases, this is a powerful feature. OpenStack Distributions With the complexity of the architecture, a number of companies have stepped in to help with installation and management of an OpenStack platform in a private cloud. These include names familiar from the Linux distribution world, such as RedHat, SUSE, and Canonical (Ubuntu), as well as new players that are focused only on OpenStack, such as Mirantis. INDUSTRY CONSOLIDATION In fact the OpenStack industry has seen a great deal of consolidation in 2014 and Several pure-play OpenStack companies have been gobbled up by the bigger players. Many large integrators and enterprise software vendors are also jumping into the OpenStack distribution game, with the likes of IBM, HP, and Oracle joining the fray. If you don t have an OpenStack cloud available already, or you want to learn more about the architecture and how all of the pieces fit together, you can setup your own OpenStack playground. You can use one of these distributions to setup your own small cloud. Each of the distribution vendors provides their own tools for setup of OpenStack. They are primarily targeted at production environments, and as such can be pretty hard to get started with on your own. For example, Canonical s offering requires a minimum of seven physical nodes just to bring up the environment. If you are setting something up small, your best options are probably RedHat s open source distribution (as opposed to their supported version running on RedHat Enterprise Linux), called RDO ( The nice thing about this distribution is that it offers a simple all in one option to deploy the entire environment on a single node. If you would like to tinker with the actual code of the various OpenStack services, you could also setup a devstack environment. Devstack ( is a powerful set of scripts to create and configure an OpenStack development environment. While the detailed instructions online are quite good, here are a few hints to make your devstack setup go smoothly. You ll want a fresh Ubuntu ( or Fedora ( installation. Don t

40 try to run devstack on your regular machine you ll want a dedicated machine (virtual or physical). If you have a virtualization product like VMware Workstation or Fusion, or the free VirtualBox for your laptop or desktop, the best thing to do is create a base server installation of your OS of choice (enabling all of the extra repositories), and then snapshot it. This will make it easy to start over if you trash your environment. The instructions will have you create a local.conf file, which the devstack scripts use to capture all of the specifics of your installation. There are only a few items you need to set in your local.conf. [[local localrc]] ADMIN_PASSWORD=stack DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD SERVICE_TOKEN=some-random-string FIXED_RANGE= /24 FLOATING_RANGE= /25 PUBLIC_NETWORK_GATEWAY= LOGFILE=/opt/stack/logs/stack.log disable_service n-net enable_service neutron q-svc q-agt q-dhcp q-l3 q-meta The first section here sets up the networking. You should pick a FIXED_RANGE that does not overlap your existing network. Your FLOATING_RANGE can correspond to an existing unused subnet on your network, with the PUBLIC_NETWORK_GATEWAY being the local default gateway on your subnet. The LOGFILE setting simply helps you debug if your devstack does not come up properly, whereas the remainder of the file disables Nova networking and enables Neutron networking. You will need access to either devstack or another OpenStack instance to follow the examples throughout this book.

41 GETTING THE OPENSTACK CLI CLIENTS To follow along with the examples, you ll need access to a machine with the OpenStack clients installed. You can learn how to install the clients at which will include instructions for a variety of operating systems. The examples in this book will use Linux. The easiest way to use these clients is to set the necessary authentication information in environment variables: $ export OS_USERNAME=username OS_PASSWORD=password OS_TENANT_NAME=tenant-name $ export OS_AUTH_URL= This allows you to call the clients without passing those parameters: $ openstack flavor list ID Name RAM Disk Ephemeral VCPUs Is Public m1.tiny True 2 m1.small True 3 m1.medium True 4 m1.large True 42 m1.nano True 5 m1.xlarge True 84 m1.micro True If your services endpoints are using HTTPS, you ll need to change the OS_AUTH_URL to reflect that. If you are using self-signed certificates, you also need to pass in the insecure option.

42 SUMMARY In this chapter, you have learned about the various types of cloud computing IaaS, PaaS, and SaaS and how they related to one another. OpenStack fills the IaaS, and perhaps in the future the PaaS functions, in the clouds. More importantly you learned that driving costs lower while delivering more features, more quickly is the driving force behind the cloud computing revolution. Finally, you learned about the major components of OpenStack Nova, Neutron, Glance, and Keystone, and how to set up a playground for experimenting with OpenStack.

43

44 2 Understanding the OpenStack Ecosystem: Core Projects WHAT S IN THIS CHAPTER? How the different OpenStack components work together and how authentication works within the infrastructure A look at how a compute instance is composed and the different hypervisors supported in OpenStack How data is stored in the infrastructure and understanding the differences between Block Storage and Object Storage How instance templates and snapshots are created and where they are stored The different ways to manage your OpenStack resources: GUI versus CLI versus APIs How the network is designed in OpenStack and the different network components available and exposed through the APIs At this point, you have an understanding of why cloud computing is important to application developers, and a general overview of OpenStack. In this chapter, you will learn the core services in more detail. These are the services most critical to running an application compute, network, and storage. You will also learn about the management services to make those possible, such as the identity service, which allows you to authenticate in order to create your applications. Sometimes, it may seem that the descriptions in this chapter go into more detail than you need to run an application. However, you can think of these features as tools and building blocks. You need to have a solid understanding of what is possible, so you can see new ways to build flexible, scalable, and robust applications (see Figure 2.1).

45 Figure 2.1

46 IDENTITY The identity service within OpenStack, named Keystone, is responsible for authentication, authorization and accounting (AAA) and currently implements and provides the OpenStack Identity API. The main goal of this identity service is to process and validate authentication and authorization requests, then return an authentication token, which is used to authenticate the user against the APIs and can be used to contact the other services of an OpenStack infrastructure. These services can be discovered using the catalog returned in the authentication response (detailed later in this chapter). Keystone currently implements two versions of the Identity API (v2, v3). The second version has been used for years and is still mainly used today in the different libraries and clients supporting OpenStack. The third version is quite recent and provides a more pluggable and flexible design, allowing using multiple authentication mechanisms (the original password method, but moreover wellknown and used mechanisms, such as OAuth or SAML2), and the ability to combine these methods in a single request. This last Identity API has a multi-tenant design and has simple resources: Region: an OpenStack infrastructure that optionally may have sub-regions Service with Endpoints: an OpenStack registered service in Keystone that can have zero, one, or multiple endpoints to reach this one (e.g. public, internal, admin) Domain: a container for the users, groups, and projects Project (known as Tenant in the second version of the API): owning a set of OpenStack resources User: a single API consumer, which should have really restricted authorizations in your application Group: a collection of different users of the same domain Role: an authorization that a user or a group of users can obtain on a project or a domain All of these resources can be managed using the Identity Admin API, which is available as a create, read, update, and delete (CRUD) RESTful API. Using Tokens and Re-Authentication The authentication against the different OpenStack services is based on tokens provided by the identity service (Keystone) or configured in the service itself (e.g. admin tokens). A token provided by an identity service is an arbitrary string that contains the User identity and optionally an authorization called scope. The authorization