EMC BACKUP-AS-A-SERVICE

White Paper EMC BACKUP-AS-A-SERVICE EMC Avamar, VMware vcloud Director, and VMware vcenter Orchestrator Provide portal-based backup management Deliver single click backup and restore for vcloud Director Provide backup services for cloud and traditional hosted environments EMC Solutions Group Abstract This white paper provides information on using EMC technology to create a portal based multitenant Backup-as-a-Service offering which is portal based and extremely easy to use. In addition, design considerations related to its implementation and how the required components are integrated are discussed. December 2012

Copyright 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H10508.1 2

Contents Executive summary... 4 Business case... 4 Solution overview... 4 Key results... 5 Introduction... 6 Purpose... 6 Scope... 6 Audience... 6 Terminology... 6 Backup-as-a-Service... 7 Overview... 7 Self-service portal... 7 Portal implementation... 8 Channel sales model... 10 Design Considerations... 12 Orchestration tool... 14 vco PowerShell... 16 Reporting capabilities... 17 EMC Avamar... 18 Overview... 18 Multitenant support... 19 CLI and API support... 19 MCCLI examples... 20 Configuration database access... 21 Limitations and workarounds... 21 Conclusion... 22 Summary... 22 Findings... 22 References... 23 White papers... 23 Product documentation... 23 3

Executive summary Business case Service providers face the challenge of providing easy to use backup solutions that integrate seamlessly with their hosted VMware vcloud Director (vcd) environments. Providing a simple portal-based graphical solution which allows their technical and non-technical vcd customers to easily backup and restore virtual machines is critical to the successful adoption of this type of offering. In addition, any portal-based Backup-as-a-Service (BaaS) solution of this type must integrate into existing orchestration and management infrastructures. Ideally, the integration of all the different systems must result in a single management interface for the customer s and service provider s administrators. Existing dedicated, standalone, disk-based or tape-based backup offerings don t provide the ease of use or deep integration with vcd that customers require. Therefore the service provider must offer a Backup-as-a-Service solution which provides that ease of use and seamless integration with vcd. This BaaS solution provides service providers with the ability to offer backup services to all of their vcd customers, regardless of their technical abilities or vcd usage model. Solution overview This white paper describes a scalable backup solution for a VMware vcloud Director environment, including the backup components and associated proof-of-content portal and orchestration integration. This solution can be used to provide backup services for public or private cloudbased VMware vcloud Director environments. In addition, this solution can be used in the environments that are not provided as a service such as VMware s vcloud Director environments within a single or multi-organizational enterprise. For this white paper, the service provider or enterprise components are co-located within one geographic data center environment. This white paper validates the integration of the solution s components and provides broad guidelines about how this type of solution can be built and integrated into the service provider s environment. Key solution components include: EMC Avamar 6.1 Provides centralized and scalable backup environment with deduplication and replication capabilities. VMware vcenter Orchestrator Provides an IT Process Automation engine that helps automate the cloud solution and integrate the VMware vcloud Suite with other orchestration systems. VMware vcloud Director - Orchestrates the provisioning of software-defined datacenter services as complete virtual datacenters that are ready for consumption in a matter of minutes. WaveMaker - Provides a rapid application development environment for building, maintaining and modernizing business-critical Web 2.0 applications. 4

PostgreSQL Provides a powerful, open source object-relational database system used for storing solution-specific information. Key results Backup-as-a-Service enables service providers to fundamentally change the way in which they provide backup services for customers who have purchased their hosted vcloud Director environments. By leveraging an in-house BaaS infrastructure that is integrated with vcloud Director, service providers can provide uniform data backup capabilities and offer differentiated offerings across their customer base. This solution demonstrates that BaaS: Can provide a simple one-click backup experience Can be leveraged through direct or channel sales Improves flexibility and simplifies application deployment Enables users to focus on revenue generating activities and other projects instead of equipment logistics 5

Introduction Purpose Scope This white paper describes the architecture of an EMC BaaS solution based on EMC Avamar and VMware vcloud Director. It also discusses how service providers can leverage the BaaS framework to deploy backup services. This framework allows service providers to adapt their service portfolio to their customers dynamic business requirements. The scope of this white paper is to demonstrate integration of Avamar and vcloud Director in a service provider environment. This solution presents a reference implementation of a portal-based channel BaaS offering that provides simple portal based configuration and recovery actions. In this white paper general configuration and operational procedures are outlined. However, for detailed product installation information, refer to the user documentation for those products. This white paper has been designed to contribute to, but not replace, other operational elements such as systems integration, customization, run books, maintenance, and support. If the need arises for operational guidance outside the scope of this paper, contact your EMC Global Services representative. Audience Terminology This white paper is intended for EMC employees, partners, and customers including IT planners, system architects and administrators, and any others involved in evaluating, acquiring, managing, operating, or designing a BaaS infrastructure environment leveraging EMC technologies. Throughout this white paper we assume that you have some familiarity with the concepts and operations related to backup and virtualization technologies, and their use in cloud and data center infrastructures. Table 1 defines some of the key terms used in this paper. Table 1. Term Tenant URL API CLI MCCLI MCGUI Terminology Definition A customer of compute/backup services. A service provider will have multiple tenants within their BaaS infrastructure. Uniform Resource Locator Application Programming Interface Command Line Interface Management Console Command Line Interface Avamar Management Console Graphical User Interface 6

Backup-as-a-Service Overview Backup-as-a-Service (BaaS) uses cloud infrastructure to backup data to a shared, rather than dedicated, backup infrastructure. Service providers can offer BaaS to customers who need a flexible, on-demand backup infrastructure but prefer not to purchase, configure, or maintain it by themselves. This solution focuses on demonstrating how a service provider could provide an integrated and easy-to-use vcloud Director based BaaS offering for those customers looking for simplicity in their backup and recovery procedures. This solution focuses exclusively on a BaaS solution for channel and direct vcd environments unlike ESG s previous Backup-as-a-Service offering, titled EMC Backupas-a-Service: EMC Avamar, EMC Data Protection Advisor, and EMC Homebase, which focused on providing a general BaaS solution. By focusing the solution more specifically, additional capabilities and functionality were able to be implemented. Fundamentally, as with any BaaS offering, this solution enables customers to consume cloud infrastructure in much the same manner as they consume electrical power. Users consume and pay for power without needing to understand or maintain the component devices and infrastructure required to provide the service. In a BaaS environment, customers can draw on the elastic resources that cloud infrastructure delivers and pay only for the backup service they consume. The BaaS environment typically consists of: Hosted vcloud Director environments Self-service portal Secure multitenant-enabled shared infrastructure Self-service portal The integration of any as-a-service offering by a service provider is a key part of their solution development and delivery mechanism. Only by integrating a new as-a- Service offering into their existing portal can they continue to offer their services in a cost-effective and scalable fashion. Allowing tenants to sign up for new services, change service levels, and perform basic tasks through a web-based portal is critical to maintaining scalability. In addition, some service providers want to use their portals not only for tenant access but also as the mechanism used by their staff to manage and administer the environment. Regardless, the ability to integrate any new as-a-service offering into the provider s existing environment is critical. The focus of this solution, developed as a proof-of-concept portal implementation, is to demonstrate the simplicity of an Avamar-based vcloud Director backup solution. Figure 1 provides a screen view of the customer virtual machine management page. 7

Figure 1. PoC Customer Virtual Machine Management view VMware vcenter Orchestrator (vco) was used as an orchestration tool along with VMware s web portal to provide a proof-of-concept web-based portal implementation. This VMware-based solution is pictorially represented in all figures of this white paper. A web-based portal with underlying orchestration simplifies administration along with management, and thereby avoids requiring users to learn and use the full featured administrative consoles of the underlying applications. A web-based portal also allows service providers to limit and audit what functions are available for each user. The goal of this proof-of-concept was to demonstrate the functionality and usage of a portal. Portal implementation For this use case, several different components were used: 1. VMware vcenter Orchestrator was used, leveraging its graphic user interface (GUI) workflow. Each service provider will need to customize and integrate the command-line interface (CLI) and/or application programming interface (API) capabilities into their own service portal offering. In our solution vco functioned as the broker between Avamar, vcloud Director, and the PostgreSQL database by using the following interfaces: SSH calls to Avamar using its SSH plug-in REST calls to VMware vcloud Director to get customer information Database calls to PostgreSQL database to retrieve and/or modify database table information While our solution leveraged vco for these calls, they could be done through (web) server side scripting (some DLLs if it is Microsoft-based, or some jar files if Java-based). In addition, the SSH calls could be done from the client side as well as through server side scripting. VMware vcenter Orchestrator (vco) is the central broker for the entire solution. For each customer it will access the database and retrieve customer 8

information. Once retrieved from PostgreSQL it will request from vcd details on the all VMs associated with that customer and store the resulting information in the database. Then it will check the backup configuration of each customer and VM. For any newly added VMs, vco will check to determine if auto protect has been selected. If it has, the VM will need to be protected (backed up) automatically for the customer. vco will then add the new VM to the customer s Avamar domain and default backup schedule. Customers who did not select auto protect for their VMs can manually protect their VMs by selecting Protect in the portal. vco is also integral to the billing information displayed by the portal. It retrieves, on a per-customer basis, the Primary Protected Bytes and Unique Bytes information from the Avamar database through SQL queries. Once retrieved this information is added to tables in the PostgreSQL database. 2. PostgreSQL Database was used to store distributor, reseller, and customer information. In addition, it was used as a repository for all the customerspecific information around VMs and backups retrieved from vcd and Avamar. By using PostgreSQL tables in this fashion, the WaveMaker portal development was simplified as the portal could display information from the tables with minimal formatting and filtering. 3. WaveMaker (aka Portal) was used to simplify and speed the development of the web portal portion of this solution. By leveraging WaveMaker s capabilities it was possible to implement the web-portal in days instead of weeks or months. 4. Avamar Virtual Edition Server - To integrate Avamar, command shell scripts were created, which were used by vco to execute (over SSH connections to the Avamar server) MCCLI commands. The exact integration and functionality provided is further discussed in subsequent sections. In addition, examples of scripts we used to enable the vco workflows are shown later in this document. The orchestration capabilities used by this solution were modeled on the work done in the white paper titled : EMC Avamar, EMC Data Protection Advisor, and EMC HomeBase. An overview of the techniques and tools used will be provided here but for additional information and other examples please refer to that white paper. 5. VMware vcloud Director integration with a backup solution was the primary use-case for this solution. The solution made no changes to vcd and only accessed it through its RESTful interfaces. The communication and interdependencies between the components is illustrated in the following diagram: 9

Figure 2. Component communication As the above diagram represents, vco is the central orchestration component communicating with all the other components. Channel sales model This solution implements a channel-based BaaS sales model in which the service provider sells capacity to distributors. The solution was architected to illustrate how a channel-based BaaS solution might be designed and implemented. A three-tiered model was constructed based on this architecture. The first tier, referred to as the distributor tier, would purchase capacity from the service provider and, in turn, resell that capacity to distributors. These distributors would sell the backup capacity to customers who would either manage it themselves or have the distributor manage the backups for them. This is graphically represented in the following diagram: Figure 3. Hierarchical view of channel implementation 10

The following figure illustrates the distributor view of the resellers purchasing backup capacity from them: Figure 4. Distributor Management View of Resellers The above illustrates the Reseller View for the Orographic Distributor Corp and lists the resellers Orographic is selling to along with their allocated and consumed space. The following figure illustrates the reseller view of their customers: Figure 5. Distributor Management View of Resellers Figure 5 illustrates the Customer View for the Nene Reseller Inc and lists the customers Nene is providing backup capacity to. This overview also details whether an individual customer is a managed customer and their total amount of Protected Bytes, also known as primary protected storage, as well as how much unique data each customer has. 11

The following figure illustrates the customer s view of the virtual machines and vapps in the vcd environment: Figure 6. Customer View of Virtual Machines The above illustrates what a customer, Mercury Themometers Inc in this case, will see in the VMs View of their portal. This view lists the individual VM s included in Mercury s vcd environment and the protected status of each VM. For simplicity the amount of protected bytes, or primary protected storage, is listed for each VM as well as whether or not the VM is powered on or off within the vcd environment. Design Considerations One major design consideration for this solution was implementing a solution which provided a simple-to-use portal suitable for non-technical users. In addition to being easy-to-use the portal had to provide automated discovery of any newly created vapps and VMs in addition to self-service restoration and backup management. Secure multitenancy to the back-end infrastructure was provided through the use of authentication and access controls within the solution. Once a customer creates a new VM or vapp through vcd it is automatically detected and listed as unprotected in the portal as illustrated below by Uranus_VM12 : Figure 7. Automatic Detection of a newly added VM 12

Once a vapp or VM is detected, protecting it is a simple matter of highlighting the VM and clicking on the Protect button as is illustrated below: Figure 8. Customer Protecting a Virtual Machine Similarly, unprotecting a virtual machine is as easy selecting the VM to be unprotected and clicking the unprotect button as shown by the following figure: Figure 9. Customer Unprotecting a Virtual Machine While this portal didn t include an option to auto-protect newly added vapps and VMs, it would be simple to implement that feature. 13

This solution also simplified the steps required to restore a VM, thereby enabling customers to perform self-service recovery operations. To restore a VM, a customer need only select that VM from the VM s View, click on the Restore button, and then pick which of the backups should be restored. The following figure illustrates this: Figure 10. Customer Restoring a Virtual Machine Orchestration tool An orchestration tool allows you to define a workflow and the operations needed to execute it on demand. For example, it can provision the server using Cisco UCS Manager plug-ins, deploy the storage using automated processes, configure the network, update CMDB, and provision the provider vdc and organization vdc. There are various orchestration tools available to perform all or some of the orchestration functions. Choosing an orchestrator depends on what functionality or infrastructure integration is needed, existing systems, price, and other considerations. For this use case, we focused on vco. 14

vco uses an open and flexible plug-in architecture to automate provisioning and operational tasks across both VMware and third-party applications, as shown in Figure 11. Figure 11. VMware vcenter Orchestrator architecture The process for developing a workflow involves a series of phases: 1. Provide general information about the workflow. 2. Create the input parameters. 3. Create the logical flow of the workflow by laying out and linking the schema. 4. Bind the input and output parameters of each element to workflow attributes, creating the necessary parameters and attributes as you define each element. 5. Create supporting scripts for scriptable tasks or custom decision elements. 6. Create the layout and behavior of the input parameters dialog box that the user sees when they run the workflow by creating the workflow presentation. 7. Validate the workflow. 15

An overview of this workflow is shown below. Figure 12. Example of designing a workflow using vco vco PowerShell The reference implementation also leveraged the vcenter Orchestrator Windows PowerShell plug-in for simple and rapid prototyping. Windows PowerShell is a command-line shell and scripting language designed for system administration, and as such it has wide-spread industry support. There are PowerShell scripts already written for many common tasks, and vco users can easily use and reuse these scripts. The vco PowerShell plug-in is used to call PowerShell scripts and commandlets (cmdlets) from Orchestrator actions and workflows, and to work with the result. For Avamar integration, the PowerShell script will use SSH to connect to the Avamar server, run the MCCLI commands, and return the output. PowerShell requires Windows to run, so we have a Windows machine with PowerShell installed on it (PowerShell host). Connection between the PowerShell plug-in and remote host machine is established using Secure Shell (SSH). For this project, we used the SSH plug-in of vco to create workflows that gather user input and then call the underlying CLI commands and shell scripts. A production implementation would also need to strictly enforce user authorization checks and validate user input. 16

Reporting capabilities For this solution none of the standard reports included with Avamar were used. Instead, the solution leveraged Avamar s read-only ODBC interface to directly extract backup usage. After extracting it from Avamar the information was uploaded into the solution s internal PostgreSQL database every few minutes. The portal then displayed the information as represented by the figure below. The billing model used by the solution was based purely on primary protected data which is the amount of actual disk capacity used by each protected virtual machine. Figure 13. Sample reporting page 17

EMC Avamar Overview EMC Avamar provides scalable backup and restore capabilities with integrated data deduplication and support for multisite replication. It also supports multitenant implementations through the use of domains. Avamar deduplicates backup data across sites and servers to reduce total disk storage, making backup a cost-effective solution. Backup data can also be encrypted in-flight and at-rest for security and privacy reasons. With the release of Avamar 6.1, Change Block Tracking (CBT) support has been added for VMware client recoveries, in addition to the existing CBT backup support that was added in Avamar 5.0. Similarly, Avamar 6.1 included an automatic function to loadbalance across multiple Avamar VMware proxies to simplify and speed-up VMware backups and recoveries. Figure 14 illustrates the Avamar Web administrative portal. Figure 14. Standard Avamar administrative GUI This solution utilized the Avamar Virtual Edition (AVE) for testing and simulation purposes. This implementation was deployed as a virtual machine within VMware. It is intended for smaller deployments up to 2 TB but is functionally compatible with a full multinode Avamar grid deployment scaling to over 100 TB of deduplicated storage. 18

Multitenant support Avamar provides domains, a built-in paradigm for segregating user data. This Avamar management feature is not tied to Internet domains. Each domain is logically segregated within the Avamar system, with backup metadata for each client assigned and accessible via that domain. By utilizing domains and other actions within Avamar, a set of actions can be restricted to hosts, clients, and groups within a specific domain or sub-domain. By organizing clients within this hierarchy, it is possible to use Avamar s built-in reporting capabilities (with some modifications) to generate status and statistical reports about backup related operations. When implementing user access controls in the service portal, each customer should be assigned a domain or sub-domain within the Avamar hierarchy. This hierarchy should be enforced on all backup calls placed by the user to the Avamar system through the portal. Whichever service portal is used to connect to the Avamar system, Console Command Line Interface (MCCLI) should be granted access to the appropriate levels of the Avamar hierarchy. This is how multitiered access controls could be implemented at both the service portal and Avamar levels if required for audit or compliance reasons. CLI and API support Through the use of the Avamar Management MCCLI, service providers can provide customized access to the backup, restore, configuration, and reporting aspects of Avamar without requiring direct access to the Avamar Management Console GUI (MCGUI). The MCGUI is a Java software application that can be installed on a Windows or Linux client. For this project, we utilized the vco SSH plugin to create workflows that gather user input, connect to the AVE server, run the required MCCLI commands or shell scripts, and then return any output or error codes. One important aspect of implementing portal integration around Avamar MCCLI involves identity management and access controls. When the SSH plugin connects to the MCCLI application, it runs as a privileged Avamar administrator that can access any and all available commands. The commands are not run as the portal user. It is the responsibility of the portal code to validate the input and parse the returning MCCLI attributes to determine what information can be presented to the requesting individual. For example, if a tenant administrator requests to see all domains within the Avamar instance, the MCCLI request will return all domains not just those that are within that tenant s domain. It is the responsibility of the portal code to review and edit the values passed to and returned from the MCCLI, to validate what subset of domains the requesting user is permitted to see. It may also be necessary for the portal to make multiple MCCLI calls on behalf of a particular user to first determine what information they are permitted to see and then actually request that information. In this way, the portal, which is vco in this solution, manages identities and their access to the Avamar environment. The following figure shows a workflow design in vco. 19

Figure 15. Designing workflow in vco MCCLI examples The following are two sample MCCLI commands. In these examples, ROOT is the tenant s top-level domain, which can be / for service provider administrators creating a new tenant. Example script using MCCLI to create a new domain: /usr/local/avamar/bin/mccli domain add -domain= ${ROOT} -location= ${NAME} - email= ${EMAIL} - contact= ${CONTACT} --name= ${DOMAIN} Example script using MCCLI to add a host to a domain: /usr/local/avamar/bin/mccli client add --location= ${LOCATION} contact= {CONTACT} --domain= ${ROOT}${DOMAIN} name= ${HOST} The MCCLI returns errors, status codes, and messages on the execution of each command. And the command output is also returned as shown in the following code example: # /usr/local/avamar/bin/mccli domain add --name="/cust001" 0,22527,Domain added. Attribute Value 20

--------- -------------------------------------------------------- ----------------------- domain <appdircomponent contact="" domain="/" email="" id="" name="cust001" phone=""/> # echo $? 0 # /usr/local/avamar/bin/mccli domain add --name="/cust001" 1,22541,Domain already exists. # echo $? 1 The error code and message numbers can be used to quickly parse and process the output from each MCCLI command. Only a few of the scripts included in this solution are provided here. Additional detail and other sample scripts are available in the EMC White Paper titled: EMC Backup-asa-Service: EMC Avamar, EMC Data Protection Advisor, and EMC Homebase. Full documentation for configuring Avamar using MCCLI is provided in the Avamar Management Console Command Line Interface (MCCLI) Programmer Guide. Configuration database access Limitations and workarounds It is possible to directly access the Enterprise Management Server (EMS) or Management Console Server (MCS) databases in a read-only manner to provide direct access to the Avamar configuration. Querying the database directly may allow more customization of the service provider s portal integration. The database views exposed are documented in the Avamar Administration Guide. One of the challenges involved in deploying Avamar in service provider environments is the requirement that each Avamar client should have a unique IP address with which to communicate with the Avamar backup server. This unique IP address is required to establish bi-directional communication between the backup client and the Avamar server. A unique IP address is not required to backup the client. For more details on how to architect solutions refer to the BRS white paper Creating Backup as a Service (BaaS) Solutions Leveraging EMC Avamar as well as the product documentation. 21

Conclusion Summary This solution provides service providers with a simple to use, integrated carrier-grade, scalable, multitenant backup service, which can easily and simply backup and restore vcloud Director environments. This solution also allows service providers to provide robust backup protection leveraging EMC Avamar which can: deduplicate data stored in virtual disks leverage Change Block Tracking (CBT) in VMware environments significantly reduce storage consumption enable replication of virtual disks across data center locations This solution provides a reference implementation for delivering backup services that leverage a service provider s existing orchestration and portal infrastructure. EMC BaaS provides the reporting capabilities that customers demand, including backing up job status, restoring job status, and daily compressing rate reports. Findings We found the following key results during the testing of this solution: The EMC BaaS solution with EMC Avamar and VMware s vcloud Director can provide an extremely easy to use multitenant backup service on a service provider cloud platform. The EMC BaaS solution with VMware vcloud Director and vcloud Orchestrator can integrate Avamar with industry-leading orchestration and portal solutions. The EMC BaaS solution successfully backed up and restored a newly created virtual machine in approximately 3 minutes. The backup and restore support of virtual machine image backups provided the ability to restore the entire virtual machine as well as individual files and folders. 22

References White papers Product documentation For additional information, see the white papers listed below. EMC documents are available on the EMC online support website. : EMC Avamar, EMC Data Protection Advisor, and EMC Homebase Compute-as-a-Service Understanding EMC Avamar with EMC Data Protection Advisor Applied Technology Applied Best Practices for Service Provider Billing For additional information, see the product documents listed below. VMware vcloud Director Documentation (VMware.com) VMware vsphere Documentation (VMware.com) VMware vcenter Orchestrator Documentation (VMware.com) Avamar 6.1 Management Console Command Line Interface (MCCLI) Programmer Guide 23