Vision & High Level Design Overview OpenDI Release 1 October 2008 v1.6 J. Carolan, J. Kirby, L. Springer, J. Stanford http://opendi.kenai.com Abstract This document provides a high level overview of the motivation, problem space, and OpenDI software. The intended audience is architects, customers, and others interested in becoming familiar with the major functional areas of the OpenDI framework and how we got here. This document is not a detailed design document, however a detailed design will extend from this overview.
Table of Contents Section 1: Introduction...5 The Enterprise Cloud Imperative...5 Towards a Data Center Operating System...5 Customer Needs and Issues...6 Motivations Architecture at the Right Level of Abstraction...9 Leading up to Clouds (Or Utility Part Deux)...9 Section 2: Introduction to OpenDI...11 Five Principles of Cloud Optimization...11 Three Boundaries...11 Applications can be Optimized for Clouds...12 Resources are Controlled Via Policy and are Transient (vs. Static)...13 There is a Dynamic Feedback Loop that Affects the Clouds Services...13 OpenDI Releases and Realization...13 Cloud Architecture Concepts...14 Executive Platforms Multi-layer Orchestration Framework...18 Legacy Integration...20 Model-driven and Actionable...21 Changes...21 Cloud Modeling...21 Cloud Control and Provisioning -- A multi-dimensional problem...22 Granularity...22 Feedback Clouds are a Two-way Street...24 A Deployment Example...25 Section 3: Design Overview...27 Major Design Considerations...27 Be the Glue...27 Model Driven...27 Distributed...27 Enterprise Context...28 Separation of Concerns...28 Policy-Centric...28 Controlled Transparency...28 Basic End To End Flows...28 Solution Developer...28 Operator...30 Release Manager...32 Administrator...32 Structural Design...32 Major Systems...35 Bootstrap...35 Authentication/Authorization...35 Task Management...36
Coordination...36 Controllers...36 Event Messaging...38 Adapters...39 Registration...39 Factory...42 Types...42 Client Proxy...43 Workflow...43 DCRM (Data Center Reference Model)...44 Provisioning...45 Persistence...46 Policy...46 Monitoring...46 Reporting...47 Communication (Internal)...47 Appendix 1...48 OpenDI/JBI Integration Requirements and ESB Analysis...48 Appendix 2...58 Useful Websites...58 Other Resources...58
Illustration Index Illustration 1: Lots of Clouds -- One Deployment Model?...6 Illustration 2: Granularity, Scale, and Granularity of Changes...7 Illustration 3: Three Boundaries of Cloud Infrastructure...12 Illustration 4: Releases Modeled after Cybernetic Feedback...13 Illustration 5: Service Platform and Executive Platform...15 Illustration 6: The Stack...16 Illustration 7: Tools and Platform Example...16 Illustration 8: Layers and Functional Areas...17 Illustration 9: Orchestration...18 Illustration 10: Multi-Platform Orchestration...19 Illustration 11: Models, Tools, and Use Cases...22 Illustration 12: Standardized Operating Environment Granularity...23 Illustration 13: Deployment Example...25 Illustration 14: Solution Developer Workflow...30 Illustration 15: Operator Workflow...31 Illustration 16: OpenDI High-Level Structural View...34 Illustration 17: Message Flow...38 Illustration 18: Adapter Registration...42
p. 5 Section 1 Section 1: Introduction The Enterprise Cloud Imperative Increasingly developers are looking elsewhere to deploy their applications. Increasingly their businesses' cost pressures are wining out and see value in cloud platforms to help manage cost. These businesses also ask the question if Amazon or Network.COM can do this why can't I run my own infrastructure this way? Secondly, other systems companies are increasingly investing in the cloud infrastructure space, including building large data centers, partnering with cloud providers, etc. IBM Plans $360M Cloud Data Center in NC Sun's On-Demand Network.com Grows Again HP Moving Defense Department Into The Cloud Sun's On-Demand Network.com Grows Again HP, Intel, Yahoo Team on Cloud Testbed Microsoft: New Design for $500M Iowa Site Most of these companies own management platforms that customers use to manage their own data centers. Increasingly they will use these tools to deploy to the internal cloud or the external cloud only differentiated by the SLA and other criteria. The types of resources they can run one will basically be the same. For example: IBM Advances Cloud Computing w/ Tivoli Provisioning Manager Sun Announces Availability of Sun xvm Ops Center, Delivers on Sun's Virtualization Strategy HP : The Next Wave: Everything as a Service HP-Opsware Deal Shows IT Automation Kicking Into High Gear -- IT Towards a Data Center Operating System OpenDI in a nutshell is a project based on three major releases to further the concept of the data center-wide (or even multiple data centers!) operating system. It provides a data bus, specification for events (that change or trigger changes), and an adapter framework that assist in the integration of components of the system. These components include: Tools (software packages) that help manage infrastructure (e.g. xvm Ops Center or CiscoWorks) Databases and other state management systems that help maintain the models Prescriptive models that help describe the policies of the systems Devices that provide resources and thus the services that utilize those services.
p. 6 The world of utility, N1, and clouds is upon us again, this time with some major differences: Virtualization everywhere from the server to the OS to the network Powerful servers multi-purpose, general purpose systems that use software to provide their key differentiation rather than custom hardware, ASICS, etc. And the proliferation and adoption of utility models such as Amazon and Salesforce.COM. Illustration 1: Lots of Clouds -- One Deployment Model? Customer Needs and Issues Our customers are not only looking to develop a strategy to utilize off-site resources but also internal resources. They are looking for a unified process for dealing with requests to deploy securely applications across these types of infrastructures. They are looking for a way to rationalize and consolidate their tools used in this space, and are finding that some of their basic IT tools are unsuitable for an increasingly flexible, dynamic world.
p. 7 Illustration 2: Granularity, Scale, and Granularity of Changes 1) CMDB and Configuration Management Many companies do not have detailed visibility into what applications are running where, system utilization, the relationship between network, storage, and compute nodes, etc. -- even for environments that don't change very often. Even companies that are providing as a service offerings are managing by spreadsheet instead of by model. This is a critical issue for the cloudscape. 2) Process maturity and Time To Market Existing operational processes and 'next gen application deployment must merge successfully to ensure systemic qualities of the services meet target service levels. Today, many cloud prototypes allow developers a great deal of control. Conversely, traditional operations organizations move relatively slow (usually with well-defined processes) and cannot meet the temporal needs of dynamic applications. This highlights another area... 3) Increasing Integration Required across Data Center Management Silos virtualization requires the network to be more closely integrated than in previous technology deployments. Applications are now defining the functions of the virtualized operating system, riding on top of a common VM domain. To master operational control of the cloud there must be a closer working relationship between these silos, and in turn, the automation technologies that are used. These views must all be taken into account by the cloud management system. 4) Business and SLAs The business must be able to define the quality of service and IT must be able to choose both internal and external resource providers. If IT doesn't make these decisions, business units
p. 8 themselves will. The process (see above) must incorporate both to ensure systemic qualities of the services are maintained at expected levels. 5) Governance and Integrity Constant changes must match back to a governance policy and meet the governmental requirements of the business. This may require both internal and external resources and their hosted applications to undergo additional testing, verification, and audit. 6) Big changes or small changes Customers need to be able to facilitate changes whether large or small. They already use similar change processes for these actions, but the system must be able to perform the change (based on the model) and ensure execution. Customers may choose to manage by a configuration file or a VM image, or a developer may simply want to update a WAR file. These should all be made possible by the system.
p. 9 Motivations Architecture at the Right Level of Abstraction OpenDI was initiated as a field-based project within Sun's Global Systems Engineering. Since systems engineering for our customers usually requires a solution, we start with architecture and architecture principles and work our way to implementation. Increasingly we have found that it is difficult to operate at a sufficient level of abstraction to provide a general solution atractive to more than one customer, thus allowing customer differences to be factored in and design trade offs or decisions as part of that solution. Technologies were changing so rapidly that we would integrate one such product by the time it was no longer being used and something else would take its place, sometimes doing better or worse. At the same time, virtualization had not yet become a buzz word but the dynamics of IT infrastructure was certainly going to change and the industry had yet to develop standards and best practices related to this. Additionally, members of the original team looked at Sun's stack and how it would work in a virtualized and dynamic world or Project JxSON. From JxSON we learned about and documented patters around application development, deployment, relationships between consumer and providers, how important the network was to make virtualization work, etc. These learnings fed into Dynamic Infrastructure, a set of consulting services supported by the DI reference architectures and implementations. So why OpenDI? Products changing, with little value in point to point integration except for early prototypes Lack of overall service orchestration and understanding of complex use cases and dependencies Lack of abstracted models, where most models are strictly embedded into a set of tools and data could be useful elsewhere, is duplicated, not the right level of granularity, etc. Treat dynamic provisioning and control as a multi-dimensional problem with varying depths Leading up to Clouds (Or Utility Part Deux) Over the course of the last two years, there has been serious changes to the IT landscape, punctuated by the economic climate that is now pervasive across the world. Where growth was the number one priority
p. 10 everywhere, it is now measured and limited by efficiency and the need to provide highly optimized platforms at very low cost. Another key change is the growth of cloud infrastructures, mostly outside the space of one's own data center. This includes force.com, AWS (Amazon web services) and others. These typically are categorized as 3 different types of services: Platform broadly defined, but includes Project Caroline (Glassfish platform) or Force.com's application platform (also Google Apps), and Ning in some ways is a good example of both a Platform and Software as a Service. Software specific end user functionality provided as software of the network (e.g. Salesforce.com) or a composite of the above then offered as SaaS. Infrastructure broad infrastructure, including storage, typically is delimited by the OS, but including networking interconnects and routing, to run any services usually a VM or container. See Joyent for example. May include storage and some basic advanced networking (though very limited.) Utility 2.0 For the enterprise, this is the second major attempt (and press coverage) at moving towards a common shared infrastructure. May include any of the above models. Shared Platforms Shared Infrastructure Software Utility as a Shared Service The difference this time around is serious consideration and cost pressures (demands) to find creative solutions... External Hosting (someone-elses cloud accessible via the network) Internal Hosting (Utility 2.0) this is the enterprise view of cloud computing when hosted internally. It looks like the N1 vision to some degree. The industry is providing robust virtualization toolsets though focused at the infrastructure platform itself, and not the surrounding issues of complexity, deployment issues, application provisioning, network management, and more that come with it.
p. 11 Section 2: Introduction to OpenDI OpenDI provides a framework for higher level integration and automation using open standards. Versions of OpenDI (and its predecessors) have been deployed at customers for several years providing them $100s of millions in cost savings and 80-90% better efficiency in staff costs and time to market. OpenDI is under process to open source in early FY09. We believe this framework approach is key and particularly appropriate for Network.COM. As customers, and Network.COM continues to build new next generation platforms while continuing to integrate with legacy components. One size fits all and a single product approach will not work. A great example of this is a large scale cloud project Sun has been asked to participate in that has over 45 different legacy management tools that need some level of analysis and eventually higher level orchestration. Some may go away but not soon enough to replaced by a single non-existent product; and then the holy grail will come under fire as some new technology is introduced. Five Principles of Cloud Optimization As background, OpenDI proposes five key principles in address cloud infrastructure optimization Three Boundaries There are three major boundaries to cloud infrastructure. The resulting layers are linked together via a common orchestration communication bus and may use common models (OpenDI in this paper).
p. 12 Illustration 3: Three Boundaries of Cloud Infrastructure 1) The Service Access Layer defines and regulates access to the overall cloud platform services. 2) Application Platform Orchestration Layer defines and regulates the application stacks and associated systems (in some ways this is A2V application to virtualization ) 3) Infrastructure Platform Orchestration Layer defines and regulates the infrastructure platform and associated systems (most call this P2V moving from a physical instance to a virtual one) Applications can be Optimized for Clouds The forces behind this are mostly around dynamics -- how often do changes occur, at what granularity, etc. Complex cloud optimized applications appear more like application frameworks, SOAs, and clusters. Clouds also work well with simple applications, basic binaries, and integrated resource managers (e.g. GridEngine). Clouds are Instantiated, Modified and Controlled Via Models
p. 13 Models are distinct entities within the cloud. Models are independent of their tools. These models help describe the rules, configuration, changes, and policies that are allowed or not within the dynamic cloudscape. Resources are Controlled Via Policy and are Transient (vs. Static) Tight resource coupling while possible (and allowed by the OpenDI framework) should be avoided. This allows for fail in place and other continuous architecture techniques as well as overall maintainability. These policies must help maintain the views of the system. For example, network admins must be able to describe what functions are allowed and not allowed as part of the dynamic operation of the system. There is a Dynamic Feedback Loop that Affects the Clouds Services Since the cloud is dynamic and has models within which to operate, a dynamic feedback loop allows for constant refinement of resources and services based on the conditions or context. This includes the policies and models related to the services. OpenDI Releases and Realization OpenDI is being developed in three releases or areas of focus. Note the diagram below the areas of overlap are necessary for the OpenDI vision, a typical negative feedback system describes within cybernetics compute research years ago. Illustration 4: Releases Modeled after Cybernetic Feedback
p. 14 However, to meet our needs and time to market, we are developing a flexible architecture that can address all of these areas, and focusing on releases of functional capabilities focused on the areas highlighted in the circle. 1) Control the modeling and centralized control (and reference adapter) implementation 2) Monitoring advanced monitoring and direction on actionable events, correlations, etc 3) Policy centralized yet distributed policy management, and direction around policy implementation We will cover all these areas in this document. However the bulk of the development efforts are focused on 1) at this time. Cloud Architecture Concepts There are two core issues when trying to even describe what a cloud is: 1) What level of abstraction (the service) is provided to the user? 2) Who is that user? In the example of Ning: Social network developer interfaces with Ning via the control APIs to define his/her platform (a social network) Social network user that may not even realize (or care) that Ning was used to create and host the environment. Both represent constituents in the cloud, but they are entirely different in the way the view and use the system. So why is this important? It illustrates the two primary platforms that make up any cloud (or really any service as defined by SOA.) The Service Platform (what the user connects to in the Ning example) The Executive Platform (what the developer connects to in the Ning example)
p. 15 Illustration 5: Service Platform and Executive Platform Some definitions: Stack: A set of dependent services that are required for a running business service. Typically something like this... (from SCDE)
p. 16 Tools: COTS and other homegrown products/scripts/etc that provision, change, report, etc. (e.g. N1 SPS, BMC Patrol, xvm Ops Center) Note not all tools are created equal. Some tools (xvm OC) are a gateway to an entire compute platform that deal with additional integration and help simplify the layers. Illustration 7: Tools and Platform Example
p. 17 Layers: Layers provide functionality in the platform, either exposing services for consumption or providing a mechanism for control, monitoring, etc. In the Services Layers, each can be viewed as part of a stack providing consumers and providers of resources. The Executive Platform Layers provide management capabilities of the Services Layer. Executive Platforms sometimes are associated with a specific layer, but others overlap. For example, an intrusion detection system might monitor virtual instance, a Layer 2 switch, and a application network switch. Illustration 8: Layers and Functional Areas
p. 18 Executive Platforms Multi-layer Orchestration Framework These platforms are familiar. For the IT manager, service platforms might be xvm or VMWare or other technologies that help provide compute, storage, and network resources for consumption. These are the providers to the consumers (users) in this case. The control platform is also familiar. It represents the technologies that IT managers use to monitor, deploy, and otherwise control the systems they manage. Examples could be Tivoli from IBM, OpenView from HP, BMC products, Sun's xvm OpsCenter, etc. What is missing is orchestration. Today this is typically done by several humans, some disjoint tools that help filter and provide better visibility into the systems, and some manual and some limited automated scripts all executed mostly by manual process and triggers, and some formal and some informal processes and procedures that evolve over the lifetime of the services and the entire data center operation. Illustration 9: Orchestration
p. 19 The cloud requires orchestration across these boundaries and layers. In the example of Ning the developer must be able to request a social network environment. This is hosted on storage, networking, and computer systems. There may be different levels of QOS associated with this request if its just an early demo environment, then the need for resources are less than a full fledged social network that is global with thousands of users. The resources provisioned (and further grown to meet the needs, or shrunk) are essentially orchestrated via complex scripts. Illustration 10: Multi-Platform Orchestration
p. 20 In the above example, an application service is layered on two other major infrastructure components the compute systems and application storage, provided by Fishworks (The storage service platform.) Since all three of these systems are orchestrated via a single orchestration bus and a common model, very complex services and service relationships can be handled. This is a key difference in the OpenDI approach. Utilizing technologies based on Java, such as OpenESB, it provides a centralized yet federated system for control, policy, transport, and abstraction of the underlying technologies necessary for these types of operations. Further, it doesn't stop at provisioning. The system can be used to react to other events... 1) load and capacity changes (e.g. service needs more capacity) 2) security issues (e.g. someone hacked a service instance) 3) operational issues (e.g. taking a system out of service for maintenance) In the past, this was done generally by point to point integration, human operators running a script or performing other work at a command line, or even a multi-day process that is manually coordinated across several people. OpenDI can take these actionable triggers, consult policy, and perform the necessary changes to the environment. It can interoperate with human workflow systems ensuring consistent process and change management policies. Legacy Integration Another challenge in the emerging enterprise cloud is legacy integration. The old tools and processes won't go away entirely by a new cloud orchestration platform. Changes of a SOE and service stack, common devices such as switches and storage servers, will still require human intervention and occasional outages. Some example of these tools: Monitoring: Preferred system and application monitoring platforms (e.g. BMC) Enterprise CMDB: Enterprise asset tracking and limited change management (often reporting structure visibility) Human Workflow: Change management, ticketing, trouble shooting, etc. (e.g. Remedy)
p. 21 OpenDI must be able to provide an integration strategy across these tools and provide for acceptable interoperability, reporting, authorization, and other critical functions. Model-driven and Actionable Changes Changes are modeled before they happen. For example, if a service is allowed to grow from 10 to 100 instances, there is a model that provides the necessary structures, data, and process to make that happen. There is a policy defined that controls how much a service can grow, what it can be deployed upon, etc. The system can utilize external reporting systems to provide alerts when resources are becoming scarce, and integrate with other monitoring systems that can provide feedback and ultimately actionable triggers. Cloud Modeling There are several models utilized within OpenDI and the DI approach. 1) Architectural: Provides the higher order rule sets for architecture governance, provides guidelines and constraints within the overall system. This model is more static than the rest. An example of this is the Model maps within the OpenDI Release 1 system or the SDNA model. 2) Service: Define the business service and its components within the architectural models. Includes QOS parameters that can be realized by the system. May be represented as a service catalog or an ontology. 3) Resource: Configuration and state of the resources that provide the services. Includes service configuration at run-time. Also called the DCRM or data center resource model within the OpenDI Release 1 system.
p. 22 Illustration 11: Models, Tools, and Use Cases Cloud Control and Provisioning -- A multi-dimensional problem Granularity One of the big struggling points in the "cloud" is around granularity of changes and how to facilitate them. I'm using some "OpenDI" terms in the following discussion -- refer to http://wikis.sun.com/display/opendi for more info on that.
p. 23 To discuss this appropriately, lets look at how systems are provisioned, using Sun's Standardized Operating Environment (SOE) as an underlying set of principles. An SOE can represent a few things... 1) A contract between consumer and providers that make up a service stack (in the context of computer systems.) 2) An image of such contract 3) Something else... We are using definition 1 the necessary components that allow are bare metal system provide a business service. Illustration 12: Standardized Operating Environment Granularity
p. 24 This illustrates how complex an SOE can be, thusly the deployment (or continuous change) of an application stack. The industry seems focused on the composite image or the image that includes the entire stack, between the network (the application network) and the hardware itself. This is in part due to the focus on OS-level virtualization. The important take away here is that this image is a composite of other entities that make up the entire stack. As you continue to move up the stack, increasing dependencies are made. As you move to the right of the diagram, the more specialized the image becomes. There exists a last mile problem in this model. The final configuration of an application instance or final deployment of the application itself (for example a WAR file in a set of app container clusters) are left to other tools, not the VM management platform. Some tools that perform these tasks are N1 SPS or HP OpsWare, and of course, the human at a CLI (command line interface.) Additionally, the composite model has other issues disk space, network transport, other physics issues (time to deploy an entire 2-4 GB on a spinning disk) are problems. This is especially apparent when a minor change ( e.g. a variable in a config file for an app server) needs to be deployed and the service restarted. One would not want to deploy 10s or 100s of complete composite images to handle this rather simple change. Depending on the complexity of this change... 1) One could use an application provisioning tool to change the variable in the config file itself on the running instance and restart the service. 2) A bare-metal or patch provisioning tool could deploy a new config file with the new variable. Some script could be used to restart the service (or an operator could perform the task) In both cases, the overall management system must be able to record the change so when this service is built again, it includes the new change. It also must be able to phase out the change, and coordinate it with the network as appropriate. Feedback Clouds are a Two-way Street The services are not static and need to be able to re-act to events, for example: Confirmation that provisioning was successful
p. 25 A user requesting a service The system noticing a policy violation and responding (e.g. more capacity) The system noticing an intrusion and executing a strategy to reduce risk and keep the service operating. Most importantly, these are asynchronous. They can occur at any time and are implemented in specific adapters within the OpenDI architecture framework. A Deployment Example Illustration 13: Deployment Example
p. 26 In the above example, there are two primary environments. Shared development and QA/Test plus Production A. Production A has limited deployment choices and supports only clustered applications since the deployment policies require high availability. Service requests are generated via a GUI and an API integrated with a development IDE which allows developers to quickly deploy applications to the dev and test environments and operations maintain higher level controls in the production environ. Request Generator/Description Target Environment Detail Developer deploys new app Development Via Netbeans IDE to OpenDI Adapter, facilitates file transfer and deployment coordination to app hosting environment in dev. Developer stages new app Production Operations receives notification via native trouble ticket tool that new app is ready for deployment. Application must conform to the approved production deployment models (represented as service catalog items) Operator executes change request Production Operations proceeds to make changes based on trouble ticket using OpenDI request GUI Operator takes node out of service Development Request to take node out of service causes development services to migrate to new node. Developers notice a few minutes of connectivity issues. Tester requests additional instances QA for scalability testing Requests resources via a model query and assigns the new resources, system builds new instances and deploys load generation system. Table 1: Some Example Deployment Use Cases Policies for the environments and resource controls have been defined to the system and are stored within the DCRM and model maps. Service changes are modeled in the service catalog and are available via GUIs, IDEs, or APIs. The software components used in this example are described in more detail in the following overview.
p. 27 Section 3 Section 3: Design Overview Major Design Considerations The OpenDI design is driven by some key considerations that are discussed in this section. There are many other factors influencing the design, but for the sake of conciseness, only the big ones are represented here. OpenDI is a key component of the executive platform for controlling dynamic multi-layered systems. Be the Glue OpenDI is not a standalone tool. It is a framework to connect provisioning, management and other operational tools in the enterprise, forming an executive platform. It does this by defining standards for a class of tool to implement. This addresses the concern that individual tools and tool implementations change over time, but the separations of concerns and the scope of responsibilities tend to be more stable. Model Driven All entities known by OpenDI are modeled. Entities include not only hardware, software, and network infrastructure, but also policies, service level definitions, and the mappings of abstract entities, such as service levels and service definitions, to acceptable concrete implementations of these services that embody the policies and constraints of the enterprise. The model is used to understand characteristics and relationships and embody the policies, concerns and constraints of the service management stakeholders. This understandings used to make operational decisions. The impact of those decisions is applied to the model. This provides the linchpin for service design, deployment, management, automation and control that reflects and can be traced to the operational models and concerns of the stakeholders that own the aggregated business, developer and data center management concerns. Distributed OpenDI is a framework for coordinating the activities of multi-vendor, multi-function tools that affect the lifecycle of technology services in the enterprise. As a framework, it should provide a set of common interfaces and taxonomy for each functional area as well as framework services to coordinate cross-function activities (e.g. task management, protocol translation and messaging). As part of embracing a distributed philosophy, all operations are designed to be asynchronous. The framework must account for transient adapters, long-running operations, non-blocking by default, and other characteristics of asynchronous systems.