EMC ENTERPRISE HYBRID CLOUD 2.5 FEDERATION SOFTWARE- DEFINED DATA CENTER EDITION

Transcription

1 Solution Guide EMC ENTERPRISE HYBRID CLOUD 2.5 FEDERATION SOFTWARE- DEFINED DATA CENTER EDITION Hadoop Applications Solution Guide EMC Solutions Abstract This document serves as a reference for planning and designing a Pivotal Hadoop solution that enables IT organizations to quickly deploy Hadoop as a service (HaaS) on an existing cloud. February 2015

2 Copyright 2015 EMC Corporation. All rights reserved. Published in the USA. Published February, 2015 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC 2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. EMC Enterprise Hybrid Cloud 2.5, Federation Software-Defined Data Center Edition: Hadoop Applications Solution Guide Part Number H

3 Contents Contents Chapter 1 Executive Summary 7 Document purpose... 8 Audience... 8 Solution purpose... 9 Business challenge... 9 Technology solution... 9 Chapter 2 EMC Enterprise Hybrid Cloud Solution Overview 11 Introduction EMC Enterprise Hybrid Cloud features and functionality Automation and self-service provisioning Multitenancy and secure separation Workload-optimized storage Elasticity and service assurance Operational monitoring and management Metering and chargeback Modular add-on components Chapter 3 EMC Enterprise Hybrid Cloud Hadoop as a Service 19 Overview EMC Enterprise Hybrid Cloud HaaS and IaaS Pivotal Hadoop Serengeti VMware vsphere Big Data Extensions Chapter 4 HaaS Component Integration 25 Overview Integrating Hadoop components with EMC Enterprise Hybrid Cloud Big Data Extensions Topology Virtualized Hadoop Configuring the platform Installing and configuring Big Data Extensions Installing and configuring PHD Installing and configuring EMC Enterprise Hybrid Cloud IaaS

4 Contents Chapter 5 Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS 35 Overview Importing and modifying custom vcenter Orchestrator workflows Modifying custom workflows Creating Big Data Extensions Clusters Creating new Big Data Extensions clusters Configuring a Hadoop cluster Creating VMware vcloud Automation Center Catalog Services Accessing VvCloud Automation Center Creating a new service blueprint Chapter 6 Use Cases: EMC Enterprise Hybrid Cloud IaaS 51 Overview IaaS storage services Overview Use case 1: Storage provisioning Use case 2: Select virtual machine storage Use case 3: Metering storage services Summary Monitoring and capacity planning Monitoring Capacity planning Capacity planning example Metering and chargeback Chapter 7 Conclusion 67 Summary Appendix A References 69 VMware references

5 Figures Contents Figure 1. EMC Enterprise Hybrid Cloud key components Figure 2. EMC Enterprise Hybrid Cloud self-service portal Figure 3. EMC ViPR Analytics with VMware vcenter Operations Manager Figure 4. IT Business Management Suite overview dashboard for hybrid cloud.. 16 Figure 5. EMC Enterprise Hybrid Cloud HaaS component overview Figure 6. Pivotal Hadoop (PHD) components Figure 7. Big Data Extensions and Serengeti stack Figure 8. Big Data Extensions and vsphere deployment topology Figure 9. The evolution of virtual Hadoop Figure 10. Figure 11. Configuring the SSO lookup service and management server IP addresses Importing Hadoop binaries into Big Data Extensions management server Figure 12. Removing the default Apache template from Big Data Extensions Figure 13. Importing custom workflows into vcenter Orchestrator Figure 14. Using the validate workflows action Figure 15. How to edit the attributes Figure 16. Editing and creating custom parameter passing Figure 17. Launching scripts from the vcenter Orchestrator Figure 18. Launching of Micro Hadoop Cluster workflow Figure 19. Figure 20. Status of creation of Micro Hadoop cluster from Big Data Extensions (vsphere web client) Status of Micro Hadoop cluster creation from Big Data Extensions vsphere Client Figure 21. Create and name a new Big Data Cluster Figure 22. Advance Service Designer Figure 23. Edit Entitlement window Figure 24. VMware vcloud Automation Center Service Catalog showing Hadoop as a Service Figure 25. Storage Services - Provision cloud storage Figure 26. Provision Cloud Storage select vcenter cluster Figure 27. Storage Provisioning Select datastore type Figure 28. Storage provisioning Choose ViPR storage pool Figure 29. Storage provisioning Enter storage size Figure 30. Provision Storage Storage Reservation for vcloud Automation Center Business Group Figure 31. Set storage reservation policy for virtual machine disks Figure 32. Create new virtual machine storage profile for Tier 2 storage

6 Contents Figure 33. Automatic discovery of storage capabilities using EMC ViPR Storage Provider Figure 34. VMware ITBM chargeback based on storage profile of datastore Figure 35. Choosing virtual machine consumption models and profiles Figure 36. Specifying configuration and projected capacity usage of new virtual machines Figure 37. Capacity summary showing insufficient CPU and RAM resources Figure 38. Specifying number of hosts and amount of CPU and memory Figure 39. Specifying datastore size Figure 40. Compared scenarios Figure 41. Combined scenarios Figure 42. Categorized hybrid cloud environment cost overview Figure 43. vsphere Cluster cost overview Figure 44. Storage cost overview

7 Chapter 1: Executive Summary Chapter 1 Executive Summary This chapter presents the following topics: Document purpose... 8 Audience... 8 Solution purpose... 9 Business challenge... 9 Technology solution

8 Chapter 1: Executive Summary Document purpose Audience This document serves as a reference for planning and designing a Pivotal Hadoop solution that enables IT organizations to quickly deploy Hadoop as a service (HaaS) on an existing cloud. The solution delivers infrastructure as-a-service (IaaS) capabilities to support big data application development. This document introduces the main features and functionality of the solution, the solution architecture and key components, and the validated hardware and software environment. It demonstrates the integration of Pivotal Hadoop Enterprise in the EMC Enterprise Hybrid Cloud solution. The Pivotal Hadoop solution is a modular add-on to the EMC Enterprise Hybrid Cloud solution. EMC Enterprise Hybrid Cloud 2.5, Federation Software-Defined Data Center Edition: Foundation Infrastructure Reference Architecture and EMC Enterprise Hybrid Cloud 2.5, Federation Software-Defined Data Center Edition: Foundation Infrastructure Solution Guide describe the reference architecture and the foundation solution upon which all the EMC Enterprise Hybrid Cloud add-on solutions build. The following documents provide further information about how to implement specific capabilities or enable specific use cases within the EMC Enterprise Hybrid Cloud solution: EMC Enterprise Hybrid Cloud 2.5.1, Federation Software-Defined Data Center Edition: Continuous Availability Solution Guide EMC Enterprise Hybrid Cloud 2.5.1, Federation Software-Defined Data Center Edition: Data Protection Disaster Recovery Solution Guide EMC Enterprise Hybrid Cloud 2.5.1, Federation Software-Defined Data Center Edition: Data Protection Backup Solution Guide EMC Enterprise Hybrid Cloud 2.5, Federation Software-Defined Data Center Edition: Security Management Solution Guide EMC Enterprise Hybrid Cloud 2.5, Federation Software-Defined Data Center Edition: Pivotal CF Platform as a Service Solution Guide EMC Enterprise Hybrid Cloud 2.5, Federation Software-Defined Data Center Edition: Public Cloud Solution Guide This document is intended for executives, managers, architects, cloud administrators, and technical administrators of IT environments who want to build a self-service Pivotal Hadoop-based Enterprise big data platform. Readers should be familiar with VMware vcloud Suite, Pivotal Hadoop, VMware vsphere Big Data Extensions, EMC ViPR, general IaaS defined datacenter concepts, and how a hybrid cloud infrastructure accommodates these technologies and requirements. 8

9 Chapter 1: Executive Summary Solution purpose Business challenge Technology solution The EMC Enterprise Hybrid Cloud solution enables EMC customers to build an enterprise-class, scalable, multitenant infrastructure that enables: Complete management of the infrastructure and application service lifecycle On-demand access to and control of network bandwidth, servers, storage, and security Quick deployment of IaaS components to support HaaS-based services without IT administrator involvement Scalable, elastic, flexible HaaS-based services for maximum asset utilization Access to application services from a single platform for both business-critical and next-generation cloud applications This solution provides the reference architecture and the best practice guidance necessary to integrate the key components and functionality of enterprise HaaS into an underlying EMC Enterprise Hybrid Cloud infrastructure. Today s enterprise demands an agile development platform that can enable the continuous delivery, updating, and horizontal scalability of applications. The Pivotal Hadoop (PHD) platform enables developers to easily deploy, bind, and scale applications and data services. When integrated with VMware vcloud Automation Center, it delivers a self-service Pivotal Hadoop platform that facilitates rapid deployment and instant scaling or updating of Hadoop clusters. HaaS interoperability with the underlying infrastructure needs to accommodate consumable new generation applications while maintaining existing end-to-end service delivery to provide: Efficiency and flexibility Fast, proactive responses for services requests Easy as-a-service model of deployment Adequate visibility into the cost of the infrastructure This EMC Enterprise Hybrid Cloud solution integrates the best of EMC, VMware, and Pivotal products and services, and empowers IT organizations to adopt an as-aservice implementation model of compute and storage infrastructure within the data center. Agile, elastic, on-demand, end-to-end IaaS provisioning is crucial to support a comprehensive, dynamic, and fast-growing big data environment. 9

10 Chapter 1: Executive Summary The key solution components include: EMC ViPR software-defined storage platform VMware vcloud Suite cloud management and infrastructure EMC and VMware integrated workflows VMware NSX for vsphere and vcloud Networking and Security (vcns) technologies VMware vsphere virtualization platform VMware Big Data Extensions with Project Serengeti Pivotal Hadoop (PHD) 10

11 Chapter 2: EMC Enterprise Hybrid Cloud Solution Overview Chapter 2 EMC Enterprise Hybrid Cloud Solution Overview This chapter presents the following topics: Introduction EMC Enterprise Hybrid Cloud features and functionality

12 Chapter 2: EMC Enterprise Hybrid Cloud Solution Overview Introduction The EMC Enterprise Hybrid Cloud solution enables a well-run hybrid cloud by bringing new functionality not only to IT organizations, but also to developers, end users, and line-of-business owners. Beyond delivering baseline infrastructure as a service (IaaS), built on a software-defined data center (SDDC) architecture, the solution delivers feature-rich capabilities to expand from IaaS to business-enabling IT as a service (ITaaS). Backup as a service (BaaS) and disaster recovery as a service (DRaaS) are now policies that users can enable with just a few mouse clicks. End users and developers can quickly access a marketplace of resources for Microsoft, Oracle, SAP, EMC Syncplicity, and Pivotal applications, and can add third-party packages as required. All of these resources can be deployed on private cloud or public cloud services, including VMware vcloud Air, from EMC-powered cloud service providers. The EMC Enterprise Hybrid Cloud solution uses the best of EMC and VMware products and services, and takes advantage of the strong integration between EMC and VMware technologies to provide the foundation for enabling IaaS on new and existing infrastructure for the hybrid cloud. Figure 1 shows the key components of the EMC Enterprise Hybrid Cloud solution. For detailed information, refer to the EMC Enterprise Hybrid Cloud 2.5, Federation Software-Defined Data Center Edition: Foundation Infrastructure Solution Guide. For information on EMC Enterprise Hybrid Cloud modular add-on solutions, which provide functionality such as data protection, continuous availability, and application services, refer to Modular add-on components and to the individual Solution Guides for those add-ons. Figure 1. EMC Enterprise Hybrid Cloud key components 12

13 Chapter 2: EMC Enterprise Hybrid Cloud Solution Overview EMC Enterprise Hybrid Cloud features and functionality The EMC Enterprise Hybrid Cloud solution incorporates the following features and functionality: Automation and self-service provisioning Multitenancy and secure separation Workload-optimized storage Elasticity and service assurance Operational monitoring and management Metering and chargeback Modular add-on components Automation and self-service provisioning The solution provides self-service provisioning of automated cloud services to both users and infrastructure administrators. It uses VMware vcloud Automation Center, integrated with EMC ViPR software-defined storage and VMware NSX or vcns network services, to provide the compute, storage, network, and security virtualization platforms for the SDDC. Cloud users can request and manage their own applications and compute resources within established operational policies. This can reduce IT service delivery times from days or weeks to minutes. Automation and self-service provisioning features include: Self-service portal Provides a cross-cloud storefront that delivers a catalog of custom-defined services for provisioning workloads based on business and IT policies, as shown in Figure 2 Role-based entitlements Ensure that the self-service portal presents only the virtual machine, application, or service blueprints appropriate to a user s role within the business Resource reservations Allocate resources for use by a specific group and ensure that those resources are inaccessible to other groups Service levels Define the amount and types of resources that a particular service can receive during initial provisioning or as part of configuration changes Blueprints Contain the build specifications and automation policies that define the process for building or reconfiguring compute resources 13

14 Chapter 2: EMC Enterprise Hybrid Cloud Solution Overview Figure 2. EMC Enterprise Hybrid Cloud self-service portal Multitenancy and secure separation The solution provides the ability to enforce physical and virtual separation for multitenancy, as strongly as the administrator requires. This separation can encompass network, compute, and storage resources to ensure appropriate security and performance for each tenant. The solution supports secure multitenancy through VMware vcloud Automation Center role-based access control (RBAC), which enables VMware vcloud Automation Center roles to be mapped to Microsoft Active Directory groups. The self-service portal shows only the appropriate views, functions, and operations to cloud users, based on their role within the business. Workloadoptimized storage Elasticity and service assurance The solution enables customers to take advantage of the proven benefits of EMC storage in a hybrid cloud environment. Using ViPR storage services, which leverage the capabilities of EMC VNX and EMC VMAX storage systems, the solution provides software-defined, policy-based management of block- and file-based virtual storage. ViPR abstracts the storage configuration and presents it as a single storage control point, enabling cloud administrators to access all heterogeneous storage resources within a data center as if the resources were a single large array. The solution uses the capabilities of VMware vcloud Automation Center and various EMC tools to provide the intelligence and visibility required to proactively ensure service levels in virtual and cloud environments. Infrastructure administrators can add storage, compute, and network resources to their resource pools as needed. Cloud users can select from a range of service levels for compute, storage, and data protection for their applications and can expand the resources of their virtual 14

15 Chapter 2: EMC Enterprise Hybrid Cloud Solution Overview machines on demand to achieve the service levels they expect for their application workloads. Operational monitoring and management The solution features automated monitoring and management capabilities that provide IT administrators with a comprehensive view of the cloud environment to enable smart decision-making for resource provisioning and allocation. These automated capabilities are based on a combination of EMC ViPR Storage Resource Management (SRM), VMware vcenter Log Insight, and VMware vcenter Operations Manager, and use EMC plug-ins for ViPR, VNX, VMAX, and EMC Avamar systems to provide extensive additional storage detail. Cloud administrators can use ViPR SRM to understand and manage the impact that storage has on their applications and to view their storage topologies from application to disk, as shown in Figure 3. Figure 3. EMC ViPR Analytics with VMware vcenter Operations Manager Capacity analytics and what-if scenarios in vcenter Operations Manager identify overprovisioned resources so they can be right-sized for the most efficient use of virtualized resources. In addition, for centralized logging, infrastructure components can be configured to forward their logs to vcenter Log Insight, which then aggregates the logs from all the disparate sources for analytics and reporting. Metering and chargeback The solution uses VMware IT Business Management Suite to provide cloud administrators with comprehensive metering and cost information across all business groups in the enterprise. ITBM is integrated into the cloud administrator s 15

16 Chapter 2: EMC Enterprise Hybrid Cloud Solution Overview self-service portal and presents a dashboard overview of the hybrid cloud infrastructure, as shown in Figure 4. Figure 4. IT Business Management Suite overview dashboard for hybrid cloud Modular add-on components The EMC Enterprise Hybrid Cloud solution provides modular add-on components for the following services: Application services This add-on solution leverages VMware vcloud Application Director to optimize application deployment and release management through logical application blueprints in VMware vcloud Automation Center. Users can quickly and easily deploy blueprints for applications and databases such as Microsoft Exchange, Microsoft SQL Server, Microsoft SharePoint, Oracle, and SAP. Data protection services EMC Avamar and EMC Data Domain systems provide a backup infrastructure that offers features such as deduplication, compression, and VMware integration. By using VMware vcenter Orchestrator workflows customized by EMC, administrators can quickly and easily set up multitier data protection policies and enable users to select an appropriate policy when they provision their virtual machines. 16

17 Chapter 2: EMC Enterprise Hybrid Cloud Solution Overview Continuous availability A combination of EMC VPLEX virtual storage and VMware vsphere High Availability (HA) provides the ability to federate information across multiple data centers over synchronous distances. With virtual storage and virtual servers working together over distance, the infrastructure can transparently provide load balancing, real time remote data access, and improved application protection. Disaster recovery This add-on solution enables cloud administrators to select disaster recovery (DR) protection for their applications and virtual machines when they provision their hybrid cloud environment. ViPR automatically places these systems on storage that is protected remotely by EMC RecoverPoint technology. VMware vcenter Site Recovery Manager automates the recovery of all virtual storage and virtual machines. Platform as a service The EMC Enterprise Hybrid Cloud solution provides an elastic and scalable IaaS foundation for platform-as-a-service (PaaS) and software-as-a-service (SaaS) services. Pivotal CF provides a highly available platform that enables application owners to easily deliver and manage applications over the application lifecycle. The EMC Enterprise Hybrid Cloud service offerings enable PaaS administrators to easily provision compute and storage resources on demand to support scalability and growth in their Pivotal CF enterprise PaaS environments. Public cloud services The EMC Enterprise Hybrid Cloud solution enables IT organizations to broker public cloud services. The solution has been validated with VMware vcloud Air as a public cloud option that administrators and users can access directly from the solution's self-service portal. End users can provision virtual machines while IT administrators can use VMware vcloud Connector to perform virtual machine migration (offline) to vcloud Air from the on-premises component of their hybrid cloud. 17

18 18 Chapter 2: EMC Enterprise Hybrid Cloud Solution Overview

19 Hadoop as a Service Chapter 3: EMC Enterprise Hybrid Cloud Chapter 3 EMC Enterprise Hybrid Cloud Hadoop as a Service This chapter presents the following topics: Overview EMC Enterprise Hybrid Cloud HaaS and IaaS Pivotal Hadoop Serengeti VMware vsphere Big Data Extensions

20 Chapter 3: EMC Enterprise Hybrid Cloud Hadoop as a Service Overview This chapter identifies and briefly describes the major features and functionality required to support Pivotal Hadoop as a service and promote scalability in the EMC Enterprise Hybrid Cloud environment. EMC Enterprise Hybrid Cloud HaaS and IaaS Project Serengeti VMware Big Data Extensions Pivotal Hadoop (PHD) HaaS Self-Service Portal EMC Enterprise Hybrid Cloud HaaS and IaaS EMC Enterprise Hybrid Cloud HaaS is a solution stack made up of EHC IaaS, integrated with Big Data Extensions and PHD. The self-service aspect of the portal is controlled by VMware vcloud Automation Center as shown in Figure 5. Hadoop is an open-source software program that supports the processing of large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation. PHD is an Apache Hadoop distribution. Deploying a Hadoop cluster using traditional methods is complex and timeconsuming. It typically involves setting up the infrastructure, installing and configuring the operating system, acquiring the respective Hadoop media, installing Hadoop components, and finally creating the Hadoop cluster. This process typically takes weeks and requires a significant skillset. The EMC HaaS offering simplifies the process by using extensive workflow automation in the EHC IaaS backend. Through self-service automation, it is now possible to deploy or expand a Hadoop cluster in minutes using the vcloud Automation Center self-service portal. 20

21 Hadoop as a Service Chapter 3: EMC Enterprise Hybrid Cloud Figure 5. EMC Enterprise Hybrid Cloud HaaS component overview Pivotal Hadoop Pivotal Hadoop (PHD) is an open-source software program that supports the processing of large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation. PHD is an Apache Hadoop distribution. The complete PHD platform contains a number of components that are not specifically used within this solution: YARN (Yet Another Resource Negotiator) a distributed processing framework that can schedule and execute resource requests from multiple applications HBASE a column database that runs on top of the Hadoop Distributed Files System (HDFS) HAWQ HAWQ is a parallel SQL query engine that combines the merits of the Greenplum Database Massively Parallel Processing (MPP) relational database engine and the Hadoop parallel processing framework ZooKeeper a centralized service for maintaining configuration information, naming services, providing distributed synchronization, and providing group services Hive a data warehouse infrastructure built on top of Hadoop infrastructure Hadoop Map Reduce Map Reduce is a programming model for processing and generating large data sets with a parallel, distributed algorithm on a cluster 21

22 Chapter 3: EMC Enterprise Hybrid Cloud Hadoop as a Service Figure 6 shows the PHD components. Figure 6. Pivotal Hadoop (PHD) components Note: YARN, HBASE, HAWQ and HIVE are not referenced in this solution. HAWQ is not installed by default and must be installed separately. This can be automated through the use of vcenter Orchestrator workflows if required. Serengeti Serengeti is an open source project initiated by VMware to enable the deployment and management of Hadoop and big data clusters in a vcenter Server managed environment. The key components are the Serengeti Management Server, which provides a framework for running big data clusters on vsphere, and a command line interface that provides tools and utilities that form an administrative interface for managing and monitoring the cluster environments. VMware vsphere Big Data Extensions VMware vsphere Big Data Extensions is a feature within vsphere to support big data and open source Hadoop distribution workloads. Big Data Extensionsprovides an integrated set of management tools to help enterprises deploy, run, and manage Hadoop on a common virtual infrastructure. Figure 7 shows how Big Data Extensions is an installable virtual appliance plug-in that controls and monitors Hadoop Services. The Big Data Extensions virtual appliance runs on top of vsphere and uses the Serengeti Management Server to control cluster creation by cloning templates through the template server. 22

23 Hadoop as a Service Chapter 3: EMC Enterprise Hybrid Cloud Big Data Extensions is a commercial version of Serengeti, which is an open source project from VMware. Big Data Extensions provides the features of Serengeti in an enterprise format, including: An open source supported version of the Apache Hadoop Distribution The big data extensions GUI which is integrated into vsphere Web Client to perform Hadoop infrastructure and cluster management tasks Elastic-enabled clusters that optimize and provide scaling of physical compute resources in a vsphere environment Figure 7. Big Data Extensions and Serengeti stack 23

24 24 Chapter 3: EMC Enterprise Hybrid Cloud Hadoop as a Service

25 Chapter 4: HaaS Component Integration Chapter 4 HaaS Component Integration This chapter presents the following topics: Overview Integrating Hadoop components with EMC Enterprise Hybrid Cloud Configuring the platform

26 Chapter 4: HaaS Component Integration Overview This section provides guidance on configuring the services required for Hadoop as a Service, specifically Big Data Extensions and PHD, and integrating them with EMC Enterprise Hybrid Cloud IaaS services. Integrating Hadoop components with EMC Enterprise Hybrid Cloud To install and configure Hadoop-as-a-Service components, refer to the appropriate vendor documentation referenced in in the installing and configuring sections for the component in this chapter. The steps discussed assume that the EMC Enterprise Hybrid Cloud has been installed and configured as described in the EMC Enterprise Hybrid Cloud 2.5.1, Federation Software-Defined Data Center Edition: Foundation Infrastructure Solution Guide, and that the IaaS, portal, catalog services, and tenant structure are all in place. Big Data Extensions Topology Big Data Extensions runs on top of Serengeti. Figure 8 shows the virtual appliance that runs the Serengeti Management Server and Template Server. Big Data Extensions provides the GUI for managing Hadoop clusters, communicating through the Serengeti Management Server. Figure 8. Big Data Extensions and vsphere deployment topology With VMware s vsphere Big Data Extensions, you can enable deployment of Hadoop inside your VMware vsphere environment. The Big Data Extensions are distributed as a downloadable OVA-based virtual appliance that is imported into an existing environment. The minimum requirements to support Big Data Extensions are vsphere 5.0 or later and Enterprise or Enterprise plus vsphere licenses. By default, the basic Apache Foundation distribution of Hadoop is also included, but it is very easy to add in other commercial Hadoop distributions such as Pivotal Hadoop, Cloudera Hadoop, Hortonworks Hadoop, or MapR Hadoop. This solution uses the Pivotal Hadoop 26

27 Chapter 4: HaaS Component Integration distribution integrated with the EMC Enterprise Hybrid Cloud IaaS stack to create Hadoop as a Service. After Big Data Extensions is installed, you can begin creating a virtual Hadoop cluster. You can specify a number of configuration options including distribution, topology (basic, compute/storage separation, HBase-only, or custom), and the number and size of the virtual machines for each of the Hadoop roles (for example, name node, client node, and data nodes). Note the options presented in the web interface are only a fraction of what can be invoked through the advanced commandline tools and API. When you start to deploy a Hadoop cluster, Big Data Extensions clones the appropriate virtual machines and automatically builds out the cluster. When you are satisfied with the cluster, you can scale up (increase the size of the virtual machine s memory and CPU resources) or scale out (increase the number of virtual machines). You can configure the cluster to scale automatically as the load alters for additional flexibility and efficiency. Virtualized Hadoop Some of the benefits of virtualizing Hadoop for example, elasticity and multitenancy arise from the increased number of deployment options that become available when Hadoop is virtualized. Figure 9 shows the evolution of virtual Hadoop, from self-contained to a tenant-based model. Figure 9. The evolution of virtual Hadoop The traditional Hadoop model combines compute and data. While this implementation is straightforward, representing how the physical Hadoop model can be directly translated into a virtual machine, the ability to scale up and down is limited because the lifecycle of this type of virtual machine is tightly coupled to the data it manages. Powering off a virtual machine with combined storage and computing means access to its data is lost. Scaling out by adding more nodes would necessitate rebalancing data across the expanded cluster, so this model is not particularly elastic. Separating computing from storage in a virtual Hadoop cluster can achieve compute elasticity, enabling mixed workloads to run on the same virtualization platform and improving resource utilization. It is simple to configure using a HDFS data layer that is 27

28 Chapter 4: HaaS Component Integration Configuring the platform always available, along with a compute layer comprising a variable number of TaskTracker nodes, which can be expanded and contracted on demand. Extending the concept of data-compute separation, multiple tenants can be accommodated on the virtualized Hadoop cluster by running multiple Hadoop compute clusters against the same data service. Using this model, each virtual compute cluster enjoys performance, security, and configuration isolation. While Hadoop performance using the combined data-compute model on vsphere is similar to its performance on physical hardware, providing virtualized Hadoop increased topology awareness can enable the data locality needed to improve performance when data and compute layers are separated. Topology awareness allows Hadoop operators to realize elasticity and multi-tenancy benefits when data storage and computing are separated. Furthermore, topology awareness can improve reliability when multiple nodes of the same Hadoop cluster are colocated on the same physical host. To optimize the data locality and failure group characteristics of virtualized Hadoop: Group virtual Hadoop nodes on the same physical host into the same failure domain, and avoid multiple replicas. Maximize usage of the virtual network between virtual nodes on the same physical host. The virtual network has higher throughput and lower latency than the physical network and does not consume any physical switch bandwidth Installing and configuring Big Data Extensions Refer to VMware vsphere Big Data Extensions Administrator's and User's Guide to install and configure the Big Data Extensions components required for Hadoop as a Service. Configuration task order The following steps outline the high-level tasks you need to perform to install and configure Big Data Extensions: 1. Ensure the environment meets the minimum vsphere requirements, correct licensing is in place, and compute, storage and networking pre-requisites are met. 2. Configure cluster settings, including vsphere HA, Distributed Resource Scheduling, host monitoring, and admission control. 3. Configure network settings using either vswitch, vsphere Distributed Switch (vds), or NSX. Ensure the required ports are configured as part of any firewall policy. 4. Deploy the Big Data Extensions OVF file and assign the management network. When you deploy Big Data Extensions the setup will ask for a destination port group; this is the network that the management network 28

29 Chapter 4: HaaS Component Integration uses to communicate with the server so the port group should be the same as the VLAN ID. If vcenter or Big Data Extensions are unable to communicate with each other, then the integration will fail. Configuring SSO service As part of the configuration process an important step is to configure the SSO service and management server IP addresses. 1. As shown in Figure 10, from the left pane in the Deploy OVF Template page select Customize template. 2. In the VC SSO Lookup Service URL box, type the vcenter Server Fully Qualified Domain Name FQDN in the same format as shown (if the default server name has not been changed). If you do not specify the FQDN here, then the certificate will not be accepted and there will be a connection issue between Big Data Extensions and the Serengeti server later. 3. Under Management Server Network Settings, enter the appropriate IP address settings. Figure 10. Configuring the SSO lookup service and management server IP addresses Starting Big Data Extensions in vsphere After successfully installing and configuring Big Data Extensions within vsphere, power on the Big Data Extensions management server and then register Big Data Extensions within vsphere as the final part of configuration by performing the following steps: 1. Log in to the vsphere client with administrative privileges. 29

30 Chapter 4: HaaS Component Integration 2. Within the vsphere client, locate the Big Data Extensions management server. The management server is located under the datacenter resource pool in which it was deployed. 3. Select and record the management IP address. 4. Register the management server using the register plugin URL: where management-server-ip-address is the IP address you recorded in step Complete the required registration information and then click Submit. The Big Data Extensions icon should now be available in the list of objects within the inventory. Installing and configuring PHD Before installing and configuring PHD, download the following required components and make them available for the installation: Cent OS bit ISO Pivotal Hadoop Tar files Oracle JDK 7, 64 bit rpm for Cent-OS Big Data Extension OVF VMware vsphere Big Data Extensions comes supplied with a default Hadoop distribution from Apache. The HaaS integration requires that Pivotal Hadoop be installed. Get the Pivotal Hadoop media and documentation from and register and obtain the necessary licenses. The following high level tasks outline the process to load the media and create a PHD template within the Big Data Extensions configuration. Installing PHD To create the required installation configuration for Big Data Extensions, use Yum repositories (as opposed to a TAR-ball). When you create a Hadoop cluster that is YUM-deployed, the Hadoop nodes within the cluster then download the Red Hat Package Manager (RPM) packages for the Pivotal Hadoop distribution from the official Yum repositories. The Pivotal Hadoop distribution must be installed in a 64-bit version of the CentOS 6.x operating system. You must use either CentOS 6.2 or CentOS 6.4 to create the Hadoop template virtual machine. The template is used in the cloning process for creating a Hadoop cluster. After you have deployed the Big Data Extensions OVF you must follow the steps to integrate YUM into PHD by creating a YUM repository as outlined below, and then create the template. Creating a Yum repository for PHD The steps for configuring PHD with Big Data Extensions are described in the VMware vsphere Big Data Extensions Administrator s and User s Administration Guide. 30

31 Chapter 4: HaaS Component Integration Creating a Hadoop template virtual machine You must use either CentOS 6.2 or CentOS 6.4 to create the Hadoop template virtual machine. To upgrade from a previous version, refer the chapter titled Create a Hadoop Template Virtual Machine using RHEL Server 6.x in the VMware vsphere Big Data Extensions Administrator s and User s Administration Guide. The following steps outline the procedure for creating a Hadoop template virtual machine: 1. Import the PHD binaries and create PHD media by logging into the Big Data Extensions management server and importing the PHD tar files into an appropriate directory structure on the server. Figure 11shows the binary import process. Figure 11. Importing Hadoop binaries into Big Data Extensions management server 2. Test that the import was successful by accessing the URL path from a browser and ensuring that the expected folders are present. 3. After installing the media into the Big Data Extensions management server, create a new Pivotal Hadoop template. 4. Make the new Pivotal Hadoop template the default template by removing the default Hadoop Apache template from the Big Data Extensions management server, as shown in Figure

32 Chapter 4: HaaS Component Integration Figure 12. Removing the default Apache template from Big Data Extensions Configuring custom resources for Big Data Extensions VMware vsphere Big Data Extensions requires two resources types when automating Hadoop clusters: networking resources and storage resources. Networking resources Networking is used to assign virtual machines IP addresses. Big Data Extensions deploys all nodes of a Hadoop cluster from a single common CentOS template that comes preconfigured with the Big Data Extensions vapp management server. As Big Data Extensions deploys virtual machines into a cluster, it uses either an existing DHCP server or a statically created IP address pool. As part of the deployment process, hostnames are assigned by Big Data Extensions. The hostnames are the same as the IP addresses. For example, if DHCP assigns then the hostname of that virtual machine is Hadoop then uses this hostname for the clusters. Storage resources Big Data Extensions defines two types of storage resources local and shared. Shared storage is useful for management or client servers deployed by Big Data Extensions as shared storage can be protected with technologies such as VMware HA. Within Hadoop there are two types of nodes: master and worker nodes. Master nodes provide tracking functions whereas worker nodes provide job processing 32

33 Chapter 4: HaaS Component Integration capabilities. Because worker nodes are disposable, they do not require top tier storage since Hadoop is designed to deal with node failure. There is also no reason to deploy worker nodes on shared storage. The choice of storage however must be capable of dealing with the required level of performance for the nodes. Allowing Big Data Extensions to use local VMFS storage for worker nodes is analogous to deploying physical worker nodes on commodity storage using direct attached storage. The final stage of configuration is to assign storage resources to Big Data Extensions. This defines how the Hadoop clusters are deployed, either using local or shared datastores. By default Big Data Extensions defines data stores as local. If you need shared datastores, you must configure the datastores accordingly. Refer to Chapter 6 of the VMware vsphere Big Data Extensions Administrator s and User s Guide for details on how to add datastores and networks to a cluster from the vsphere client. Installing and configuring EMC Enterprise Hybrid Cloud IaaS For details. refer to the EMC Enterprise Hybrid Cloud 2.5, Federation Software-Defined Data Center Edition: Foundation Infrastructure Reference Architecture. Detailed installation and configuration information is available only to select EMC personnel and authorized partners. 33

34 34 Chapter 4: HaaS Component Integration

35 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Chapter 5 Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS This chapter presents the following topics: Overview Importing and modifying custom vcenter Orchestrator workflows Creating Big Data Extensions Clusters Creating VMware vcloud Automation Center Catalog Services

36 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Overview The automation of Hadoop clusters is achieved by using custom workflows created with VMware vcloud Orchestrator (vco). This chapter describes how these workflows are configured from within VMware Cloud Automation Center (VMware vcloud Automation Center) to present enterprise organizations with a self-service portal that includes a catalog of pre-configured Hadoop deployment scenarios. Importing and modifying custom vcenter Orchestrator workflows To use HaaS within EMC Enterprise Hybrid Cloud, the administrator must use custom vcenter Orchestrator workflows for deploying HaaS. These workflows offer a choice of cluster sizes that can then be presented as catalog items from the vcloud Automation Center portal. The workflows are imported into VMware vcenter Orchestrator using the vcenter Orchestrator import function to be edited, tested, and packaged according the needs of the organization. Modifying custom workflows This section describes the process for importing the custom workflows into vcenter Orchestrator, so that the Hadoop Administrator can alter them and link them with the big data cluster configurations created in the earlier stages of the process. Importing custom workflows From within the vcenter Orchestrator client, as shown in Figure 13, select Run, click Workflows, and select Import workflow. Browse to the location where you have placed the workflow package and click Open. The imported workflow appears in the folder selected. Figure 13. Importing custom workflows into vcenter Orchestrator 36

37 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Validating workflows After importing the workflows into vcenter Orchestrator, validate them by clicking the name of the folder containing the workflows and then selecting the Validate option from the context menu, as shown in Figure 14. The validation process ensures there are no open ends, unreachable workflow elements, or unused attributes in the workflows, so that they will execute correctly. Figure 14. Using the validate workflows action Customizing HaaS workflows The HaaS workflows provide a framework for deploying each Hadoop cluster configuration of a given size through an automated workflow. The Hadoop administrator should modify the attributes of these workflows to meet the specific needs of the organization. Figure 15 shows how to use the vcenter Orchestrator client to edit the attributes within a workflow. 37

38 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Figure 15. How to edit the attributes Configuring custom parameters To make the workflows dynamic, vcenter Orchestrator uses a combination of attributes and parameters to transfer data when it is processing a workflow. Workflow parameters must receive an input to generate an output or action. An example of configuring a custom parameter is when an input is received from the user or system. The input can then be passed to a command or script that would create a username or password. This in turn can be passed to the Hadoop cluster for authentication. Figure 16 shows how to create a custom username and password for the Hadoop Client node. 38

39 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Figure 16. Editing and creating custom parameter passing Launching a custom script Scripts help to edit the schema, which is the main component of a workflow. Launching individual scripts lets you test the components of the workflow one element at a time, or execute a script at runtime to prepare the data set, for example. Figure 17 shows how to launch scripting from within the workflow by using the Schema panel within the workflow itself. 39

40 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Figure 17. Launching scripts from the vcenter Orchestrator Testing vcenter Orchestrator HaaS custom workflows The previous sections demonstrated how to import the HaaS sample workflows into EMC Hybrid Cloud, specifically the vcenter Orchestrator which is the main orchestration and automation engine for the solution. As shown, once imported, the default workflows can be altered to meet any modifications made to the Hadoop clusters. The workflows can also be modified to pass any additional parameters that may be required, for example, passing the username and password or executing parts of additional scripts components. The final stage in importing and configuring the workflows is to test the workflows that have been imported and modified for each of the HaaS cluster sizes (micro cluster, small cluster, and large cluster). Figure 18 shows how to: Select the specific workflow for a given cluster size Execute the workflow from vcenter Orchestrator View the execution process Verify the execution progress by checking the log files for any error messages 40

41 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Figure 18. Launching of Micro Hadoop Cluster workflow Viewing cluster creation After the vcenter Orchestrator workflow is launched, the cluster creation process starts within vsphere and Big Data Extensions. The management server uses the template server to clone the nodes required to create the cluster in terms of the numbers and types of node that comprise the cluster. To view and verify the cluster creation process, follow these steps: 1. Login to the vsphere web client. 2. Go to the Big Data Extensions and view the actual cluster being created. Figure 19 shows the status of the creation of a micro Hadoop cluster in the Big Data Extensions panel of the vsphere web client. 41

42 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Figure 19. Status of creation of Micro Hadoop cluster from Big Data Extensions (vsphere web client) You can also log in to the vsphere Client Application and view the Hadoop cluster being created. Figure 20 shows the status of the creation of the Micro Hadoop cluster in the vsphere Client Application. 42

43 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Figure 20. Status of Micro Hadoop cluster creation from Big Data Extensions vsphere Client 43

44 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Creating Big Data Extensions Clusters Creating new Big Data Extensions clusters After the vcenter Orchestrator workflows are imported they need to be customized for the different sized clusters according to the requirements of the enterprise. The examples provided describe micro, small, and large Hadoop clusters. The custom workflows define the type of the cluster, including cluster configuration, in terms of the number of master nodes, client nodes, and data nodes for each size. Creating a Hadoop cluster These steps document the procedure for creating a Hadoop cluster within Big Data Extensions, which can then be translated when building a vcenter Orchestrator workflow: 1. In vcenter, under Objects > Data Extensions,click New Big Data Cluster. 2. Follow the steps in the wizard, specifying the appropriate parameters as required. More detail can be found in the VMware vsphere Big Data Extensions Administrator s and User s Guide. Configuring a Hadoop cluster The following sections outline the options and details required during the cluster configuration process. Naming a Hadoop cluster When prompted by the wizard, type a name to identify the cluster. Valid characters for cluster names are alphanumeric and underscores. When choosing a cluster name you should also consider the associated vapp name. Together the vapp and cluster name must be less than 80 characters. Configuring the Hadoop distribution When configuring a Hadoop cluster, you must select the correct Hadoop distribution from the Hadoop distribution list box Change the default from Apache to Pivotal HD, as shown in Figure 21. The distribution name matches the value of the name parameter that was passed to the config-distro.rb script when the Hadoop distribution was configured. For a Pivotal PHD 1.1 cluster, you must configure a valid DNS and FQDN for the cluster's HDFS and MapReduce traffic. Without valid DNS and FQDN settings, the cluster creation process might fail or the cluster is created but does not function. Figure 21. Create and name a new Big Data Cluster 44

45 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Specifying deployment type When prompted by the wizard, select the deployment type for the cluster, either Basic Hadoop Cluster or Data/Compute Separation Cluster. The type of cluster you create determines the available node group selections. Identifying the DataMaster node group The DataMaster node is a virtual machine that runs the Hadoop NameNode service. This node manages HDFS data and assigns tasks to Hadoop TaskTracker services deployed in the worker node group. To identify the group: 1. Select a resource template from the list box or select Customize to create a custom resource template. 2. For the master node, specify shared storage so that the virtual machine is protected with vsphere HA. Identifying the ComputeMaster node group The ComputeMaster node is a virtual machine that runs the Hadoop JobTracker service. This node assigns tasks to Hadoop TaskTracker services deployed in the worker node group. To identify the group: 1. Select a resource template from the list box or select Customize to create a custom resource template. 2. For the master node, specify shared storage so that the virtual machine is protected with vsphere HA. Identifying the HBaseMaster node group (HBase cluster only) The HBaseMaster node is a virtual machine that runs the HBase master service. This node orchestrates a cluster of one or more RegionServer slave nodes. To identify the group: 1. Select a resource template from the list box or select Customize to create a custom resource template. 2. For the master node, specify shared storage so that the virtual machine is protected with vsphere HA. Identifying the Worker node group Worker nodes are virtual machines that run the Hadoop DataNode, TaskTracker, and HBase HRegionServer services. These nodes store HDFS data and execute tasks. To identify the group: 1. Select a resource template from the list box or select Customize to create a custom resource template. 2. For the worker nodes, use local storage. 45

46 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Note: You can add nodes to the worker node group by using Scale Out Cluster, but you cannot reduce the number of nodes. Identifying the Client node group A client node is a virtual machine that contains Hadoop client components. From this virtual machine you can access HDFS, submit MapReduce jobs, run Pig scripts, run Hive queries, and run HBase commands. When configuring the cluster for use with HaaS, you do not configure the Client node group unless any of these configuration items are required outside of the HaaS solution. To identify the group: 1. Select a resource template from the list box or select Customize to create a custom resource template. 2. For the client nodes, use local storage. Note: You can add nodes to the client node group by using Scale Out Cluster, but you cannot reduce the number of nodes. Selecting the Hadoop topology configuration When you create a cluster with Big Data Extensions, it disables automatic migration for the cluster s virtual machines. This prevents vsphere from migrating anything but does not prevent the administrator from migrating nodes unintentionally to other vcenter hosts. It is essential that migrating is not performed from within vcenter as this could break the cluster placement policy. As part of the final cluster configuration you should select the topology configuration that you want the cluster to use: RACK_AS_RACK, HOST_AS_RACK, HVE, or NONE. More information is available in the chapter About Cluster Topology in Chapter 7 of the VMware vsphere Big Data Extensions Administrator s and User s Guide. 46

47 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Creating VMware vcloud Automation Center Catalog Services The focus of customization for this EMC Hybrid Cloud solution is the VMware vcloud Automation Center user self-service portal, where additional functionality is included to enable additional services for cloud users. The final stage of integrating Hadoop as a Service is to present to vcloud Automation Center the HaaS workflows that have been imported and modified so that they can be selected as catalog items. VMware vcloud Automation Center 6.0 provides the extensibility to enable IaaS functionality through Advanced Service blueprints. The IaaS functionality is achieved by exposing custom vcenter Orchestrator workflows that the vcloud Automation Center 6.0 portal can present as a catalog of services for cloud users. You can create custom workflow definitions using vcloud Automation Center Designer. The vcloud Automation Center Designer console provides a visual workflow editor for customizing vcloud Automation Center lifecycle workflows. The extensibility toolkits include a library of activities that serve as building blocks for custom workflows. Using the Advanced Service Designer, you can define new service offerings and publish them to the common catalog as catalog items. Accessing VvCloud Automation Center To create the service blueprints you must access vcloud Automation Center from a browser and log in to vcloud Automation Center. Each tenant has a unique URL to the vcloud Automation Center console: The default tenant URL is in the following format: where hostname is the Fully Qualified Domain Name (FQDN) of a vcloud Automation Center host. The URL for additional tenants is in the following format: where tenanturl is the URL name specified when the tenant is being created. This is the workspace in which the customer creates catalog services. Creating a new service blueprint The following steps demonstrate, at a high level, how to integrate the HaaS workflows into the vcloud Automation Center self-service catalog by showing the creation of: Catalog services Blueprints Custom resources and resource actions For more information, refer to the vcloud Automation Center Extensibility Guide. 47

48 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS To integrate the HaaS workflows into the vcloud Automation Center self-service catalog, follow these steps: 1. From the main vcloud Automation Center portal page, click Advanced Services to list all of the current service blueprints defined. 2. Click the green plus symbol, shown in Figure 22, to create a new service blueprint. Figure 22. Advance Service Designer Follow these steps to create a new service blueprint: 1. Select one of the imported Hadoop Cluster Creation workflows from the list. 2. Name the new service and create a form to support user input for the required parameters. If required, delete the default form and create a new form. 3. Drag and drop any appropriate input fields onto the form. 4. Publish the new service to create the appropriate service definition in the catalog management. 5. Assign a catalog management service to the new advanced service, and create the appropriate entitlement definition in the catalog management, as shown in Figure

49 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS Figure 23. Edit Entitlement window When these tasks are completed, the new service is then available in the service catalog for the cloud administrator. It is possible to replace the default VMware logo icons in the service catalog with more suitable HaaS icons. The replacement of icons is the final stage of customization and ensures that the service catalog items are tailored to a specific function or application. This can be performed from the Catalog Management menu by selecting the Catalog Items list box, selecting the configure an icon option, and then browsing and selecting a new icon. After the configuration stages have been performed within vcloud Automation Center, the service catalog is available to provision HaaS items, as shown in Figure 24. Figure 24. VMware vcloud Automation Center Service Catalog showing Hadoop as a Service 49

50 50 Chapter 5: Creating vcenter Orchestrator Workflows and VMware vcloud Automation Center Catalog Services for HaaS

51 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Chapter 6 Use Cases: EMC Enterprise Hybrid Cloud IaaS This chapter presents the following topics: Overview IaaS storage services Monitoring and capacity planning Metering and chargeback

52 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Overview IaaS storage services This chapter covers EMC Enterprise Hybrid Cloud IaaS and other use cases that can be incorporated to extend the functionality beyond virtual machine provisioning to consume resources. From time to time additional physical resources will be required to support the extension of a Hadoop environment. The following sections show how EHC storage provisioning workflows can be used to create additional resources on demand by provisioning additional storage as required, and how the VMware vcenter Operations tool set can be used to analyze consumed resources, provide capacity planning, increase resources using scenarios that increase physical resources, and increase virtual machine and node capacity. Overview Storage is provisioned, allocated, and consumed by different cloud users in this solution. For vcloud Automation Center IaaS users, the storage services provided in the vcloud Automation Center service catalog provision storage resources that will be allocated to and consumed by other cloud users. Once the storage resources are available, fabric group administrators can assign the resources to business groups. Creators of virtual machine blueprints (business group managers) can then configure their blueprints to use those particular storage resources for the list of virtual machine disks. When they provision virtual machines, cloud users consume the storage and, depending on their entitlements, may choose the storage service for their virtual machines. Use case 1: Storage provisioning This use case demonstrates how ViPR software-defined storage is provisioned for the hybrid cloud from the VMware vcloud Automation Center self-service catalog. 1. To provision block or file storage from the vcloud Automation Center selfservice portal, select the Provision Cloud Storage item from the vcloud Automation Center service catalog, as shown in Figure

53 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Figure 25. Storage Services - Provision cloud storage The storage service blueprint can be created using vcloud Automation Center anything-as-a-service (XaaS) functionality in the vcloud Automation Center Advanced Service Designer. EMC ViPR provisioning workflows, which are presented by vcenter Orchestrator to the vcloud Automation Center service catalog, support storage services. The storage provisioned by the IaaS user enables the fabric group administrator to make storage resources available to their business group. The storage provisioning request requires very little input from the vcloud Automation Center IaaS user. The main inputs required are: Datastore Type: VMFS or NFS Datastore Size vcenter Cluster Storage Tier Most of these inputs, except LUN size, are selected from pre-populated list boxes whose items are determined by the cluster resources available through vcenter and the virtual pools available in ViPR. After entering a description and reason for the storage-provisioning request, enter your password. The vcenter Server will manage multiple ESXi clusters; therefore, you must choose the relevant vcenter cluster to tell the provisioning operation where to assign the storage device. Select a vcenter cluster from the next screen, as shown in Figure

54 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Figure 26. Provision Cloud Storage select vcenter cluster 2. Select the type of datastore you require from the list of available storage types, as shown in Figure 27. A datastore type of VMFS requires block storage, while NFS requires file storage. Other data services such as disaster recovery and continuous availability are displayed as appropriate only if detected in the underlying infrastructure. Figure 27. Storage Provisioning Select datastore type 3. Select from which storage offering the new storage device should be provisioned. The list of available storage offerings is based on the datastore type selected, such as VMFS or NFS, and what matching virtual pools are available from the ViPR virtual array. In this example, a single NFS-based ViPR virtual pool is available to provision storage from, with the available capacity of the virtual pool also displayed to the user, as shown in Figure 28. The storage pools listed have been configured in the EMC ViPR virtual array and their storage capabilities are associated with storage profiles created in vcenter. 54

55 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Figure 28. Storage provisioning Choose ViPR storage pool 4. Enter the size required for the new storage, in GB, as shown in Figure 29. Figure 29. Storage provisioning Enter storage size 5. The fabric group administrator must reserve the new Storage Pool for use by the business group, as shown in Figure 30. Figure 30. Provision Storage Storage Reservation for vcloud Automation Center Business Group When the automated process sends an notification to the fabric group administrator that the storage is ready and available in vcloud Automation 55

56 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Center, the fabric group administrator can then assign capacity reservations on the device for use by the business group. In this example, a number of required input values, such as LUN or datastore name, have been masked from the user during the storage provisioning request process. Some of these values are locked-in and managed by the orchestration process and logic to ensure consistency. In addition to the initial provisioning of storage to the ESXi cluster at the vsphere layer, this solution provides further automation and integration of the new storage up into the vcloud Automation Center layer. The ViPR storage provider automatically tags the storage device with the appropriate storage profile based on its storage capabilities. The remaining automated steps in this solution are, through the vcloud Automation Center: rediscovery of resources under vcenter endpoint storage reservation policy assigned to new datastore fabric group administrator notification of availability of new datastore Use case 2: Select virtual machine storage This use case demonstrates how cloud users can consume the available storage service offerings. This use case is part of the broader virtual machine deployment use case, but here it relates directly to how the business group manager and users can manage the storage service offerings available to them. VMware vcloud Automation Center business group managers and users can select the appropriate storage for their virtual machine through the VMware vcloud Automation Center user portal. For business group managers, the storage type for the virtual machine disks can be set during the creation of a virtual machine blueprint. As shown in Figure 31, the relevant storage reservation policy can be applied to each of the virtual disks. Figure 31. Set storage reservation policy for virtual machine disks 56

57 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS After the storage reservation policy is set, the blueprint will always deploy this virtual machine and its virtual disks to that storage type. If more user control is required at deployment time, the business group manager can elect to allow business group users to reconfigure the storage reservation policies at deployment time by selecting the checkbox Allow user to see and change storage reservation policies. Use case 3: Metering storage services This solution uses VMware IT Business Management Suite (ITBM) to provide chargeback information on the storage service offerings for the hybrid cloud. Through its integration with VMware vcenter and VMware vcloud Automation Center, ITBM enables the cloud administrator to automatically track utilization of storage resources provided by EMC ViPR. The EMC ViPR VASA provider in vcenter automatically captures the underlying storage capabilities of LUNs provisioned from virtual pools on the EMC ViPR virtual array. Storage profiles are created based on these storage capabilities, which are aligned with the storage service offerings. This integration enables ITBM to automatically discover and group datastores based on predefined service levels of storage. In this solution we created a separate virtual machine storage profile for each of the storage service offerings, as shown in Figure 32. Figure 32. Create new virtual machine storage profile for Tier 2 storage The storage capabilities are shown automatically in vsphere, as shown in Figure 33, where Tier 2 EMC ViPR storage is supporting a datastore. 57

58 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Figure 33. Automatic discovery of storage capabilities using EMC ViPR Storage Provider Note: Storage capabilities are only visible in the traditional vsphere client and not in the web client. Also, the web client uses virtual machine storage policies in place of virtual machine storage profiles. After the EMC ViPR Storage Provider has automatically configured the datastores with the appropriate storage profiles, the data stores can be grouped and managed in ITBM in line with their storage profile. Figure 34 shows that the cost profiles created in vcenter are discovered by ITBM. This allows the business management administrator to group tiered datastores provisioned with ViPR and set the monthly cost per GB as needed. Figure 34. VMware ITBM chargeback based on storage profile of datastore Summary VMware vcloud Automation Center can provide a storefront for storage services to be used by cloud users. These service catalog items deploy EMC ViPR software-defined storage services based on the usage of multiple service offerings of block and file storage across EMC VNX and VMAX storage arrays. Each service offers varying levels of availability, capacity, and performance to satisfy the operational requirements of different lines of business. This solution combines EMC ViPR with EMC array-based FAST-enabled storage service offerings across the EMC storage arrays with VMware vsphere to simplify storage operations for hybrid cloud consumers. 58

59 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Monitoring and capacity planning Monitoring The VMware vcenter Operations Management Suite has functions that can help HaaS administrators to achieve the following goals: Eliminate or significantly reduce the manual problem-solving effort in the environment. Proactively manage core service and cloud infrastructure performance, and utilize infrastructure resources optimally. Provision proactive warnings regarding performance issues before problems affect the end user. Real-time performance dashboards enable service providers to meet their SLAs by highlighting potential performance issues before end users notice these issues. Infrastructure maintenance and operations teams need the end-to-end visibility and intelligence to make fast, informed operational decisions to proactively ensure service levels in cloud environments. They need to get promptly to the root cause of performance problems, optimize capacity in real time, and maintain compliance in a dynamic environment of constant change. The vcenter Operations Management Suite offers many features and functions to deliver quality of service, operational efficiency, and continuous compliance for your dynamic cloud infrastructure and business critical applications. Capacity planning This section describes in detail the capacity planning functions that can help you to predict the impact on underlying infrastructure of new HaaS deployments or of upgrading current HaaS instances with new services. Forecasting capacity risks in vcenter Operations Manager involves creating what-if scenarios to examine the demand and supply of resources in the cloud infrastructure. A what-if scenario is a supposition about how capacity and load might change if certain conditions, influenced by an increased or decreased number of ESX hosts, storage resources, or virtual machines in environment, occur, without making actual changes to your virtual infrastructure. If you implement the scenario, you know in advance what your capacity requirements are. To create a what-if scenario, you can use models and profiles based on current resource consumption in the existing environment. Alternatively, you can manually define amounts of virtual machine RAM, storage, CPU, and utilization in a new consumption profile, as shown in Figure 35, to predict the potential impact of growth. 59

60 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Figure 35. Choosing virtual machine consumption models and profiles To define a new virtual machine profile, you can make detailed specifications that give you the option to include and predict specific resource utilizations, reservations, and limits in order to get as accurate a projection as possible, as shown in Figure 36. Figure 36. Specifying configuration and projected capacity usage of new virtual machines Figure 37 shows that there are insufficient resources for a planned deployment scenario consisting of either 50 or 85 new virtual machines. In this case, we can easily provision new vsphere hosts using vcloud Automation Center services as described in previous sections. 60

61 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Figure 37. Capacity summary showing insufficient CPU and RAM resources Before you provision new hardware resources, you can create hardware change scenarios to determine the effect of adding, removing, or updating the hardware capacity in a vsphere cluster. You can create a scenario that models changes to hosts and datastores, as shown in Figure 38 and Figure 39. Figure 38. Specifying number of hosts and amount of CPU and memory 61

62 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Figure 39. Specifying datastore size The what-if scenario capacity planning function allows you compare how adding different amounts of virtual machines and hardware will impact your actual environment, as shown in Figure 40. Figure 40. Compared scenarios Capacity planning example In a planning exercise, assume that you: Have a request to deploy an additional 45 Hadoop node instances in the existing HaaS. Plan to purchase blade servers compliant with a certain specification. Want to deploy an additional 25 Hadoop clusters. 62

63 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS In Figure 41, each column shows how an individual change affects resources in your environment. The Combined Scenarios column shows you the cumulative effect of hardware purchasing and an overall expansion of 70 virtual machines. Figure 41. Combined scenarios Metering and chargeback VMware ITBM provides cloud administrators with comprehensive metering and cost information across physical and virtual resources in the EMC Enterprise Hybrid Cloud environment. Besides working out the cost of physical components such as storage, compute, and networking resources, you can also include and configure other factors that affect the overall cost of your cloud environment, such as operating system licensing, maintenance, labor, and environmental facilities costs, as shown in Figure

64 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Figure 42. Categorized hybrid cloud environment cost overview ITBM is integrated into the vcloud Automation Center portal for the Hadoop administrator and presents a dashboard overview of the hybrid cloud infrastructure. VMware ITBM Standard Edition uses its own reference database, which has been preloaded with industry-standard data and vendor-specific data to generate the base price for virtual CPU (vcpu), RAM, and storage values. These prices, which default to the cost of CPU, RAM, and storage, are automatically consumed by vcloud Automation Center, where they can be changed as appropriate by the cloud administrator. This eliminates the need to manually configure cost profiles in vcloud Automation Center and assign them to compute resources. ITBM is also integrated with vcenter and can import existing resource hierarchies, folder structures, and vcenter tags to associate EMC Hybrid Cloud resource usage with business units, departments, and projects. Infrastructure resources consumed by HaaS instances and hosted applications are provided by dedicated vsphere clusters with associated vsphere hosts and datastores. ITBM provides you with detailed information about: Number of vsphere hosts in the vsphere cluster and the number of virtual machines on each host CPU and RAM capacity and utilization of the vsphere cluster Overall cost of the compute resources provided by the dedicated vsphere cluster 64

65 Chapter 6: Use Cases: EMC Enterprise Hybrid Cloud IaaS Cluster cost by virtual machine The Clusters tab provides you with insight into the cost of the vsphere cluster resources consumed by Hadoop cluster instances. You can monitor costs while provisioning new hosts, as shown in Figure 43. Figure 43. vsphere Cluster cost overview The Datastores tab provides insight into the cost of the storage resources consumed by an HaaS instance. The name of a datastore provisioned by vcloud Automation Center storage services inherits a cluster name prefix as part of its published name. Performing a sort by datastore name gives you a list of the names and costs of the datastores provisioned and assigned to hosts in the vsphere cluster, as shown in Figure 44. Figure 44. Storage cost overview 65