INTEGRATING CLOUD ORCHESTRATION WITH EMC SYMMETRIX VMAX CLOUD EDITION REST APIs

Transcription

1 White Paper INTEGRATING CLOUD ORCHESTRATION WITH EMC SYMMETRIX VMAX CLOUD EDITION REST APIs Provisioning storage using EMC Symmetrix VMAX Cloud Edition Using REST APIs for integration with VMware vcloud Deploying applications in a multitenant environment EMC Solutions Group Abstract This solution illustrates how public and private cloud providers can use EMC Symmetrix VMAX Cloud Edition as a foundation for their as-a-service offerings by integrating it into their existing multitenant cloud infrastructure based on VMware vcloud. September 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 2

3 Table of contents Executive summary... 4 Business case... 4 Solution overview... 4 Key results/ recommendations... 4 Introduction... 6 Purpose... 6 Scope... 6 Audience... 6 Terminology... 6 Technology overview... 8 VMAX Cloud Edition... 8 VMAX Cloud Edition service levels VMware vcenter Orchestrator VMware vcenter Chargeback VMware WaveMaker VMware vcloud Director VMware vsphere Solution use case Configuration Service provider role Tenant administrator role Tenant user role Optional configuration VMware vcloud Integration Manager Conclusion Summary Findings References White papers Other documentation Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 3

4 Executive summary Business case Service providers, enterprises, and mid-market businesses are increasingly driving a new as-a-service delivery model for IT, becoming providers of public and private clouds. As their customer base expands, they are faced with the challenges of reducing complexity, driving automation, delivering secure multitenancy, improving TCO, and simplifying IT planning. As the concept of cloud computing becomes a reality, application owners are quickly changing their thinking from infrastructurebased deployments to service-level-based deployments. Less consideration is given to the mechanics of the underlying infrastructure and more to performance levels that can be delivered predictably to their business-critical applications regardless of the underlying components. Cloud providers that is, service providers serving multiple tenants in a shared public cloud, or enterprise IT organizations serving multiple internal business units need to be able to deliver the service levels their customers are requesting in the most automated and efficient manner possible. Offering cloud providers an as-a-service platform that delivers automated service-level provisioning with automatic metering, reporting, and billing is an important step toward realizing IT-as-a-service (ITaaS) efficiency. Offering the platform in a way that can be integrated easily into an existing cloud provider environment is critical to realizing this efficiency. Solution overview This solution, developed by EMC Solutions Group, shows how cloud providers can use a shared EMC Symmetrix VMAX Cloud Edition storage array to host their own private clouds as well as provide public services to external customers. Cloud providers can integrate VMAX Cloud Edition into their new and existing as-a-service environment using the REST APIs provided. The VMAX Cloud Edition REST APIs allow the cloud provider to build automation workflows leveraging the orchestration and web portal tools that they have in place to execute common storage management and provisioning tasks. In the use case demonstrated in this paper, a cloud administrator provider gets various service levels of storage from the cloud provider and uses them in a VMware vcloud environment. The cloud administrator then sells the virtual data centers as products using vcloud Integration Manager and a custom portal developed using VMware WaveMaker. Automation is performed using the VMware vcenter Orchestrator engine. VMAX Cloud Edition billing reports are used to determine the base rates for the storage, and VMware vcenter Chargeback is used to create the detailed billing report based on the actual use of various components. Key results/ recommendations This white paper demonstrates how a cloud provider can use the REST-based APIs available in VMAX Cloud Edition to integrate the platform into its existing VMware vcloud environment. By exposing the programmatic interfaces among portal, storage, and vcloud, we are able to represent a seamless deployment of VMAX Cloud Edition for block storage, presenting this storage to the vcloud environment for consumption in a multitenant cloud infrastructure. If the cloud provider is planning to allow changes to the service levels of its provisioned volumes, it should consider using VMware vcloud Director 5.1 and Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 4

5 creating a Provider virtual data center (vdc) that includes all storage profiles for tenant use. The cloud provider should also consider using VMAX Cloud Edition data collection to facilitate reporting and billing by VMware vcenter Chargeback. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 5

6 Introduction Purpose The purpose of this white paper is to provide cloud providers with a detailed example of how they can use VMAX Cloud Edition in the most automated and efficient manner within the parameters of their existing infrastructure based on VMware vcloud. After reading this white paper, cloud providers should understand the benefits of integrating VMAX Cloud Edition into their environments using the available REST APIs to automate storage provisioning and implement multilevel storage for deployments of business-critical applications. Scope The scope of this paper is to provide an overview of an existing multitenant environment based on VMware vcloud and a detailed description of how a VMAX Cloud Edition array can be programmatically integrated into that environment to provide automated storage provisioning for the cloud. This paper also demonstrates how management can be delegated among a cloud provider, customer, and tenant user. The cloud storage in this paper encompasses the Fibre Channel SAN-based block storage provided by VMAX Cloud Edition, the VMFS datastores that are based on those volumes, and the vapps that consume the storage as virtual disks within the organization s vdcs. Audience This white paper is intended for technical engineering staff responsible for deploying and configuring as-a-service offerings in a private, public, and/or hybrid cloud environment. It is assumed that the reader has a strong background in virtualization technologies and overall infrastructure architecture knowledge. With Symmetrix VMAX Cloud Edition management and automation, only a cursory knowledge of storage technologies is necessary, as Symmetrix VMAX Cloud Edition is built specifically to abstract the underlying mechanics of typical storage technologies. Finally, since a large part of this solution involves API integration, a strong knowledge of REST-based APIs and development tools and frameworks is recommended. Terminology This paper includes the terminology defined in Table 1. Table 1. Terminology Term Application programming interface (API) Organization Organization virtual data center (Org vdc) Definition A source-code-based specification intended to be used by software components as an interface to communicate with each other. The unit of multitenancy that represents a single logical security boundary. An organization contains users, virtual data centers, and networks. A subgrouping of compute and storage resources allocated from a Provider vdc and assigned to a single organization. Resources are provisioned to an Org vdc based on vcloud Director resource allocation models. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 6

7 Term Provider virtual data center (Provider vdc) REST API SAN fabric zoning Storage tiering vapp VMware vcloud Integration Manager (vcim) Definition A grouping of compute and storage resources from a single vcenter server instance. A Provider vdc consists of a pool of physical compute resources and one or more datastores. Multiple organizations can share Provider vdc resources. Representational State Transfer application programming interface. REST API defines a set of functions that enable developers to send requests to, and receive responses from, an associated API service through HTTP. The logical grouping of physical interfaces in a storage area network that enables communication between a source (server) and a target (storage array). The assignment of data to different types (tiers) of storage media to meet the service levels and thus reduce total storage cost. A container for a distributed software solution. vapp, which is the standard unit of deployment in vcloud Director, consists of one or more virtual machines and can be imported or exported as an Open Virtualization Format (OVF) package. Software that enables virtual data centers to be sold as products and enables service provider administrators to maintain the distributors and resellers. vcim is available to service providers who are part of the VMware Service Provider Program (VSPP). Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 7

8 Technology overview VMAX Cloud Edition VMAX Cloud Edition enables cloud providers to automate storage provisioning to host private and public clouds as well as legacy applications in a physical environment. Each tenant can have its own view of the management portal and billing information. Tenants are billed based on service levels of storage, and those levels are based on performance characteristics. VMAX Cloud Edition also provides flexibility in automating the provisioning of storage to hosts, including defining and administering SAN fabric zones. VMAX Cloud Edition provides: Secure multitenancy Role-based access control (RBAC) determines the level of access for the provider and tenants. A web portal enables tenant self-service, which reduces the need for provider intervention by enabling delegation of day-to-day management tasks, such as storage provisioning and de-provisioning, to the tenants. The portal manages multiple instances of VMAX Cloud Edition and can be branded per provider, as shown in Figure 1. Figure 1. Tenant portal view Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 8

9 Versatile web portal Users manage their storage assets by submitting service requests to the EMC data center through the web portal, as shown in Figure 2. Interfaces are simplified and abstracted for tenant use. VMAX Cloud Edition automatically takes these logical requests and intelligently translates them into specific resource allocation requests, based on EMC-proven best practices. The requests are then executed against specific storage assets. Figure 2. Tenant Admin portal view The web portal also provides reporting, trending, and monitoring capabilities. Usage reports are tenant-based to facilitate chargeback. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 9

10 VMAX Cloud Edition storage service catalog The levels of storage within VMAX Cloud Edition are described as logical service levels as a part of a service catalog. Each service level is described logically based on its availability and performance characteristics. Users can provision from the service level that best matches their specific application needs, as shown in Figure 3. Figure 3. VMAX Cloud Edition storage service catalog Additional availability profiles can be added to each service level to facilitate business continuity, disaster recovery, mobility, and high availability. REST APIs In addition to providing a web-portal user interface, VMAX Cloud Edition supports REST interfaces, as shown in Figure 4. This enables cloud providers to programmatically integrate VMAX Cloud Edition capabilities directly into their existing orchestration infrastructure. For ease of integration into existing cloud provider environments, VMAX Cloud Edition also provides Web Application Description Language (WADL), which can be accessed at It supports both JSON and XML outputs. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 10

11 Figure 4. VMAX Cloud Edition REST APIs VMAX Cloud Edition service levels VMAX Cloud Edition provides storage service levels of varying availability, capacity, and performance. Each service level is a pre-engineered mix of different drive types, as shown in Figure 5, individually weighted respective to their own quality of service. With different ratios of drive types in each service level, some service levels are weighted in favor of capacity and others in favor of overall performance, which enables cloud providers to charge appropriate prices per level while still maintaining performance and operational benefits. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 11

12 Figure 5. Examples of storage service levels The service levels provide the highest I/O density at the highest-available level where workloads demand it. The I/O density decreases as the service levels move toward serving less performance-intensive workloads where cost is of more importance. There are multiple bands available in each of these service levels, allowing for flexibility in pricing and performance. The service levels illustrated in Figure 3 on page 10 represent a subset of the storage service levels available on VMAX Cloud Edition. Variants of each of these service levels are available from a range of drive packs based on fixed policy mixes. The tenants are provided with the information about various service levels, expected response times, IOPS, and typical use cases. Even if they fail to choose the correct level during provisioning, they can change the service level of the volume at a later time without impact to their applications. Figure 6 shows how a service level can be Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 12

13 changed after an application has been deployed on the storage. Figure 6. Portal service-level change utility Appropriate cost models and rate factors can be applied to these service levels. To facilitate chargeback, allocation reports can be generated on details such as storage consumption. VMAX Cloud Edition performs the SAN fabric zoning, configuring the number of storage paths appropriate for the chosen service level. VMware vcenter Orchestrator VMware vcenter Orchestrator is an IT process automation engine that helps automate cloud processes. It simplifies the integration of VMware vcloud Suite with third-party products and management systems such as VMAX Cloud Edition. vcenter Orchestrator enables workflow definition to automate common tasks with predefined libraries. It also can be extended using third-party plug-ins. In our test environment, we used the REST plug-in for vcenter Orchestrator to define the REST hosts of VMAX Cloud Edition, vcim, and vcenter Chargeback, and we used those web services calls in vcenter Orchestrator workflows. We also used workflows provided by VMware plug-ins for vcenter, vcloud Director, and vcenter Chargeback and extended those workflows as needed. Figure7 depicts a standard vcenter Orchestrator workflow used in our use-case environment. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 13

14 Figure 7. vcenter Orchestrator sequence workflow VMware vcenter Chargeback VMware vcenter Chargeback enables billing-report generation based on various cost models. As resources are shared across multiple tenants, the tenants are charged based on their consumption of the resources. The reports can be generated at the organization or vdc level, or by using various other customizable hierarchies. VMware vcenter Chargeback provides vcenter and vcloud Director data collectors to retrieve resource consumption information automatically. Figure 8 shows vcenter Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 14

15 Chargeback costing options that can be used to provide usage and billing information. Figure 8. vcenter Chargeback VMware WaveMaker VMware WaveMaker provides a rapid Web 2.0 application development environment. It includes a WYSIWYG development studio for creating web applications. Using WaveMaker, we created a sample cloud provider portal to demonstrate the integration capabilities of VMAX Cloud Edition. WaveMaker interacts with vcenter Orchestrator web services to execute workflows and retrieves its output in XML format. Figure 9 shows the WaveMaker configuration options available for the portal page. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 15

16 Figure 9. WaveMaker portal configuration page VMware vcloud Director VMware vcloud Director provides a portal for cloud providers to use to define virtual data centers. A tenant administrator can use the portal to define service catalogs, users, and policies for those virtual data centers. Tenant users can use the portal to create or consume vapps in those virtual data centers. vcloud Director 5.1 allows the use of multiple storage profiles within the same data center, and virtual machines in the vapp can be deployed to specific storage profiles. VMware vsphere VMware vsphere provides the virtualization layer to abstract the underlying hardware resources. VMAX Cloud Edition supports vsphere Storage APIs for Array Integration (VAAI), which provides hardware acceleration for vsphere, by offloading certain storage operations, like cloning and zeroing, to the array for greater efficiency. Figure 10 shows that the datastores provisioned to the vcenter support hardware acceleration. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 16

17 Figure 10. View of vsphere datastores with VAAI support VAAI support can also be confirmed using esxtop. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 17

18 Solution use case Configuration VMAX Cloud Edition can be used by any cloud provider to allow tenants to host private, public, and/or hybrid clouds. In our solution use case, a vcloud provider who uses VMAX Cloud Edition creates virtual data centers and sells them as a product. The cloud provider environment described in this white paper and depicted in Figure 11 is based on VMware vcloud and includes vcloud Director, vcenter Orchestrator, vcenter Chargeback, and vsphere. Additionally, vcim is used to automate sign-up and provisioning of vcloud Director tenant environments. Finally, administrator and user access is provided by a front-end portal based on WaveMaker from VMware. Figure 11. Logical depiction of cloud provider environment The cloud provider using VMAX Cloud Edition gets shared storage of various levels. Using this shared storage, the cloud provider creates Provider vdcs and publishes product templates using vcloud Director. Customers then order virtual data centers as products and deploy their applications within those virtual data centers. VMAX Cloud Edition provides allocation information for various service-level volumes. The cloud provider uses that information to determine the base rate for those volumes. This solution uses vcenter Chargeback to generate billing reports based on the actual use of volumes by the customers of vcloud. VMAX Cloud Edition customers can export this same information into any number of billing systems they may have in place. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 18

19 Service provider role Using administrator credentials, the cloud provider logs in to the portal, as shown in Figure 12. The portal in our case was created using VMware WaveMaker. Figure 12. Service provider portal login window Once logged in, the cloud provider can see the list of tenants and can add and remove tenants. Figure 13 shows the cloud provider view of the sample portal. Figure 13. Cloud provider view of portal Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 19

20 Adding a tenant initiates execution of a vcenter Orchestrator workflow to add a customer organization in vcloud Director. The workflow also creates an administrator account within that organization. Figure 14 shows the organizations created in the test environment vcloud Director instance. Figure 14. vcloud Director organizations As shown in Figure 15, the cloud provider can list the product templates that equate to service levels of storage provisioned from VMAX Cloud Edition. Figure 15. Product templates provided by cloud provider through VMAX Cloud Edition Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 20

21 In addition, the cloud provider can add new product templates by specifying the required storage size and selecting the service level, as shown in Figure 16. Figure 16. Create Product Template Dialog This action initiates execution of the following vcenter workflow: 1. Create a volume on the appropriate service level on VMAX Cloud Edition. 2. Create the fabric zone if needed. 3. Present the newly created volume to the vsphere host. 4. Rescan the vsphere host. 5. Create the datastore. The datastore is then used to create the Provider vdc in vcloud Director. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 21

22 Figure 17 shows the Volume Create request on VMAX Cloud Edition. Figure 17. VMAX Cloud Edition Volume Create request Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 22

23 Note: Previous versions of vcloud Director require a separate vdc for every service level. With vcloud Director 1.5, a single Provider vdc can have multiple storage profiles within it if necessary. Having multiple storage profiles within the same vdc can simplify configurations in the event a tenant ever needs to change the service level of a volume. The cloud provider can view a tenant s consumption information and generate a billing report, as shown in Figure 18. Figure 18. Billing Report request Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 23

24 The reports generated can be viewed using the WaveMaker portal as well as under Archived Reports in vcenter Chargeback, where they can be sent by , generated in PDF, or exported as CSV files or MS Word documents. Figure 19 shows the Archived Reports list in vcenter Chargeback. Figure 19. vcenter Chargeback Archived Reports Tenant administrator role The tenant administrator can log in to the cloud provider portal and view the storage service levels (product templates) available to provision. Figure 20 shows the username format that a tenant administrator uses to log in. Figure 20. Tenant administrator login to cloud provider portal Figure 21 shows the product templates that are available for the tenant administrator to order, along with the corresponding rate card. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 24

25 Figure 21. Tenant administrator view of available product templates The tenant administrator can select any of the product templates and purchase those products, as shown in Figure 22. Figure 22. Tenant administrator s product template order The Current Products tab, shown in Figure 23, displays products (Org vdcs) ordered by the tenant administrator. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 25

26 Figure 23. Tenant administrator s view of products ordered The tenant administrator can use the link provided, as shown in Figure 24, to access the vcloud environment to deploy vapps and add users, catalogs, and so on. Figure 24. Cloud provider s portal link to vcloud environment Figure 25 shows the vcloud Director environment in which the tenant administrator can add users. Figure 25. vcloud Director tenant view Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 26

27 In addition, the tenant administrator can view the vdcs that have been ordered, as shown in Figure 26. Figure 26. Tenant administrator s view of vdcs ordered Finally, the tenant administrator can generate ad hoc reports from vcenter Chargeback using the cloud provider portal, as shown in Figure 27. Figure 28 shows an example of such a report. Figure 27. Ad hoc report request by tenant administrator Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 27

28 Figure 28. Ad hoc report generated from vcenter Chargeback Tenant user role The tenant user uses the vcloud Director portal URL provided by the tenant administrator. Based on user role, the tenant user can create new vapps or use existing ones. When creating or deploying vapps, the user can see the organization s vdcs and have the option to select one of them, as shown in Figure 29. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 28

29 Figure 29. Tenant user s selection of vdc when creating or deploying vapp With vcloud Director 5.1, the tenant user also has the option to identify the storage profile on which to deploy vapps. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 29

30 Optional configuration VMware vcloud Integration Manager For service providers who are part of the VMware Service Provider Program (VSPP), VMware vcloud Integration Manager (vcim) can be used to present the option to order virtual datacenters and present them through a catalog. Otherwise, that function is handled through vcloud Director via the vcenter Orchestrator workflow. vcim enables service providers to sell virtual data centers with their own product numbers and rate card, and to sell any unused capacity through distributors and resellers. Figure 30 shows the storage products available from the service provider as seen in vcim. Figure 30. vcim product catalog vcim also can be used for on-boarding customers, resellers, and distributors, as well as tracking the number of products ordered by customers versus resellers. vcim provides a REST API, which was used in the test environment, along with vcenter Orchestrator, to automate tasks such as on-boarding tenants, defining the products offered, and ordering the products on behalf of the customer. Figure 31 shows the vcim tasks automated through vcenter Orchestrator workflows. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 30

31 Figure 31. vcim tasks available in vcenter Orchestrator Figure 32 shows the product templates on vcim and demonstrates that the Provider vdc is tied to the product template. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 31

32 Figure 32. vcim Figure 33 depicts the logical environment when vcim is used. Figure 33. Logical depiction of cloud provider environment with vcim Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 32

33 Conclusion Summary For cloud providers who want to offer their end users and tenants a foundation for various as-a-service products, VMAX Cloud Edition simplifies and automates the provisioning of world-class storage to support the applications that run a business. The solution use case demonstrates the ease with which VMAX Cloud Edition can be integrated with an existing cloud provider automation environment. Using the RESTbased APIs available with VMAX Cloud Edition, cloud providers can quickly and easily integrate VMAX Cloud Edition with their existing administrative, management, and user portals, and use its orchestration tools to seamlessly provision service-levelbased block storage for use with their most critical applications. Findings VMAX Cloud Edition provides easy and efficient automated storage provisioning in public, private, and hybrid cloud environments, enabling cloud providers to quickly realize a return on their storage investment. End users of VMAX Cloud Edition can easily request storage for their business-critical and lower-level applications based on the required service, and then quickly and easily apply it to their applications. VMAX Cloud Edition REST API calls can easily be executed from within VMware vcenter Orchestrator or any orchestrator that has the ability to make REST calls. By integrating the storage provisioning into existing workflows, the cloud provider can provision volumes, rescan the vsphere host, create datastores, and associate the provisioned storage to Provider vdcs all with a single click of a link on the cloud provider portal. If a cloud provider shares volumes among multiple tenants, it can determine the base rate to charge by using the VMAX Cloud Edition portal along with whatever metering tool is in place in the cloud environment. Finally, cloud providers using a VMware vcloud environment need to plan ahead in their deployment, as having different service levels of storage volumes affects the design of the Provider vdc based on the version of vcloud Director they are using. Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 33

34 References White papers Other documentation For additional information, see the following white paper: Optimizing Cloud Deployment of Virtualized Applications on EMC Symmetrix VMAX Cloud Edition For additional information, see the VMware documentation for the following products: vcloud Director vsphere vcenter Chargeback Manager vcenter Orchestrator WaveMaker Integrating Cloud Orchestration with EMC Symmetrix VMAX Cloud Edition REST APIs 34