EMC BACKUP-AS-A-SERVICE

White Paper EMC BACKUP-AS-A-SERVICE EMC AVAMAR, EMC DATA PROTECTION ADVISOR, AND EMC HOMEBASE Deliver backup services for cloud and traditional hosted environments Reduce storage space and increase backup speeds Provide portal-based backup management EMC Solutions Group Abstract This white paper provides information on creating a Backup-as-a-Service platform using EMC technology such as EMC Avamar, EMC Data Protection Advisor, and EMC HomeBase. It also explores the design considerations related to the platform s implementation, and provides information on how to integrate various components in that infrastructure. March 2012

Copyright 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H10508 2

Contents Executive summary... 5 Business case... 5 Solution overview... 5 Key results/ recommendations... 6 Introduction... 7 Purpose... 7 Scope... 7 Audience... 7 Terminology... 7 What is Backup-as-a-Service?... 8 Overview... 8 Self-service portal... 8 Portal implementation... 9 Design considerations... 10 Orchestration tool... 10 Developing a workflow... 11 vco PowerShell... 12 Reporting capabilities... 13 EMC Avamar... 14 Overview... 14 Multi-tenant Support... 15 CLI and API Support... 15 Workflows... 15 MCCLI examples... 16 Configuration Database Access... 17 Limitations and workarounds... 17 EMC Data Protection Advisor... 18 Overview... 18 Reporting... 19 CLI and API support... 20 Scheduling reports... 20 On-demand reports... 21 EMC HomeBase... 23 Overview... 23 CLI and API Support... 24 3

Avamar Scripts... 25 Overview... 25 General script notes... 25 Service provider tasks... 25 List all Avamar domains and sub-domains present in the system... 25 Create an Avamar domain... 25 Deleting an Avamar domain... 26 Tenant admin tasks... 26 Add a machine to the Avamar domain... 26 List client s domain name... 27 Delete client from a domain... 27 Create a default dataset... 27 Create a custom dataset... 28 Create a retention policy... 28 Create a schedule... 29 Create a group... 30 Tenant admin master script... 30 Tenant user tasks... 31 Add machines to the existing backup group.... 31 Conclusion... 33 Summary... 33 Findings... 33 About EMC Proven Solutions... 34 Take the next step... 34 References... 34 White papers... 34 Product documentation... 34 4

Executive summary Business case Service providers face the challenge of offering robust backup services to protect their customers data for both consumers of cloud-based services and traditional hosting services, while deploying the backup solution in a scalable fashion. Similarly, the BaaS solution must integrate into existing orchestration and management infrastructures. Ideally, the integration of all the different systems must result in a single management interface for the customers and service provider s administrators. Service providers can offer Backup-as-a-Service as an alternative to existing dedicated, stand-alone, disk- or tape-based backup offerings; while integrating customer service catalogs into an easy-to-deploy platform. EMC s BaaS solution provides service providers with the ability to offer backup services to all of their customers, regardless of whether they are consumers of cloudbased services or traditional hosting services. Solution overview This white paper describes a carrier-class backup solution for virtual and physical servers, including the backup components and associated portal and orchestration integration. This solution can be used to provide backup services for: Backups at the application, file system, or virtual machine image level within a multitenant service provider cloud environment Bare-metal backup of physical servers within service provider data centers In addition, this solution can be used in the following environments that are not provided as-a-service: Backups at the application, file system, physical servers, or virtual machine image level within a traditional hosting environment Backups for application, file system, or virtual machine image level within a single or multi-organization enterprise For this solution use case the service provider, or enterprise, components are colocated within one geographic data center environment. This white paper validates the integration of the solution s components and provides broad guidelines about how this type of solution can be built and integrated into the service provider s environment. Key solution components include: EMC Avamar 6.0 Provides centralized and scalable backup environment with deduplication and replication capabilities. EMC Data Protection Advisor 5.8 Creates reports on storage utilization and backup coverage. EMC HomeBase 6.6 Automates platform configuration logging and provides restore and migration capabilities for physical and virtualized systems. 5

Key results/ recommendations Backup-as-a-Service enables service providers to change the way in which they provide backup services to their customers. By leveraging an in-house BaaS infrastructure, service providers can provide uniform data backup capabilities and also offer differentiated offerings across their customer base, allowing them to: Improve flexibility and simplify application deployment. Enable end-users to focus on revenue generating activities and other projects instead of equipment logistics. Create a strong foundation to leverage the benefits of other services such as backup, data protection, and more. 6

Introduction Purpose Scope This white paper describes the architecture of the (BaaS) solution based on EMC Avamar, EMC Data Protection Advisor, and EMC HomeBase. It also discusses how service providers can leverage the EMC BaaS framework to deploy backup services. This framework allows service providers to adapt their service portfolio to their customers dynamic business requirements. Throughout this white paper we assume that you have some familiarity with the concepts and operations related to backup and virtualization technologies, and their use in cloud and data center infrastructures. This white paper discusses multiple EMC products as well as those from other vendors. Some general configuration and operational procedures are outlined. However for detailed product installation information, please refer to the user documentation for those products. Audience Terminology This white paper is intended for EMC employees, partners, and customers including IT planners, system architects and administrators, and any others involved in evaluating, acquiring, managing, operating, or designing a Backup-as-a-Service infrastructure environment leveraging EMC technologies. Table 1 defines some of the key terms used in this paper. Table 1. Terminology Term Tenant URL API CLI Definition A customer of compute/backup services. A service provider will have multiple tenants within their BaaS infrastructure. Uniform resource locator Application programming interface Command line interface 7

What is Backup-as-a-Service? Overview Backup-as-a-Service (BaaS) uses cloud infrastructure to back up data to a shared, rather than dedicated, backup infrastructure. Service providers can offer BaaS to their customers who want a flexible, on-demand backup infrastructure without having to purchase, configure, or maintain it themselves. Much like an electric power utility, in which end-users consume and pay for power without needing to understand or maintain the component devices and infrastructure required to provide the service, customers can draw upon the elastic resources that cloud infrastructure delivers and pay only for what they need. A BaaS environment typically consists of: Self-service portal Backup clients Secure multitenant enabled shared infrastructure Self-service portal The integration of any as-a-service offering by a service provider is a key part of their solution development and delivery mechanism. Only by integrating any new as-a- Service offering into their existing portal can they continue to offer their services in a cost-effective and scalable fashion. Allowing tenants to sign up for new services, change service levels, and perform basic tasks through a web-based portal is critical for maintaining scalability. In addition, some service providers wish to use their portals not only for tenant access but also as the mechanism used by their staff to manage and administer the environment. Regardless, the ability to integrate any new as-a-service offering into the provider s existing environment is critical. This solution initially developed a proof-of-concept portal implementation, as shown in Figure 1, using simple web/shell scripts. We then went further and used VMware vcenter Orchestrator (vco) as an orchestration tool along with the VMware web portal to provide a more capable proof-of-concept web-based portal. This VMware-based solution is shown in the figures throughout this white paper. A web-based portal with underlying orchestration simplifies administration and management, and thereby avoids requiring users to learn the full-featured administrative consoles of the underlying applications. This also allows service providers to limit and audit the functions that are available to each user. The goal of this proof-of-concept was to demonstrate what functionality a portal could provide and how. 8

Figure 1. Example of simple web page listing of scripts Portal implementation For this use case we used VMWare vcenter Orchestrator and leveraged its GUI for each integration of Avamar and Data Protection Advisor (DPA) action. Service providers will need to customize and integrate the CLI and API capabilities into their own specific service portal offering. To integrate Avamar we created command shell scripts for vco to execute MCCLI commands over SSH connections to the Avamar server. DPA reports can be scheduled and stored in a folder where they can be picked up by the portal. Alternatively, DPA 5.x supports XML formatted commands for accessing DPA reports in raw format. In this use case we integrated Avamar and DPA functionality into the portal. The integration is further discussed in subsequent sections. In addition, examples of scripts we used to enable the vco workflows are shown in Avamar Scripts. There are various portal and service catalog options available which perform all or some of the portal and catalog functions. Choosing a portal/catalog depends on what functionality is needed, existing systems, price, and other considerations. For this use case, we created simple shell/web scripts using CLI options to initiate backup/restore/configuration actions. We integrated these into vco to provide the portal interface shown in Figure 2. 9

Figure 2. Example implementation of BaaS self-service portal using vco Design considerations One major design consideration for this solution is enforcing secure multitenancy on a shared back-end infrastructure. User authentication and access controls are available within each component of the overall solution. We chose to enforce user authentication and authorization at the service portal rather than at the point of interaction with each component. We felt this would be the most compatible implementation, as service providers would already have existing authentication mechanisms in place for their portals and would not need to integrate authentication with each product. This means that all interactions between the portal and the underlying servers use a shared authentication mechanism. The service portal must then enforce user access controls. This eliminated additional complexities such as password and account synchronization between the underlying servers and the portal. This increases the complexity of the portal side of the implementation, as it must control user access and perform input validation before calling the underlying scripts. A production implementation may require additional considerations including using a tiered account strategy to control portal access to certain systems. One example may be using different portals for customer and infrastructure machines or for audit/compliance reasons. Orchestration tool An orchestration tool allows you to define a workflow and the operations needed to execute it on demand. For example, it could provision the server using Cisco UCS Manager plug-ins, deploy the storage using automated processes, configure the network, update CMDB, provision the provider vdc and organization vdc, and so on. There are various orchestration tools available which perform all or some of the orchestration functions. Choosing an orchestrator depends on what functionality or 10

infrastructure integration is needed, existing systems, price, and other considerations. For our use case testing we focused on vcenter Orchestrator. VMware vcenter Orchestrator uses an open and flexible plug-in architecture to automate provisioning and operational tasks across both VMware and third-party applications, as shown in Figure 3. Figure 3. VMware vcenter Orchestrator architecture Developing a workflow The general process for developing a workflow is as follows: 1. Provide general information about the workflow. 2. Create the input parameters. 3. Create the logic of the workflow by laying out and linking the schema. 4. Bind the input and output parameters of each element to workflow attributes, creating the necessary parameters and attributes as you define each element. 5. Create supporting scripts for scriptable tasks or custom decision elements. 6. Create the layout and behavior of the input parameters dialog box that the user sees when they run the workflow by creating the workflow presentation. 7. Validate the workflow. An overview of this workflow is shown in Figure 4. 11

Figure 4. Example of designing a workflow using vco vco PowerShell Our reference implementation also leveraged the vcenter Orchestrator Windows PowerShell plug-in for simple and rapid prototyping. Windows PowerShell is a command-line shell and scripting language designed for system administration, as such it has wide-spread industry support. There are PowerShell scripts already written for many common tasks, and vco users can easily use and reuse these scripts. The vco PowerShell plug-in is used to call PowerShell scripts and commandlets (cmdlets) from Orchestrator actions and workflows, and to work with the result. For Avamar integration, the PowerShell script will SSH to the Avamar server, run the MCCLI commands, and return the output. PowerShell requires Windows to run, and so we have a Windows machine with PowerShell installed on it (PowerShell host). Connection between the PowerShell plug-in and remote host machine is established using SSH. For this project, we used the SSH plug-in of vco to create workflows that gather user input and then call the underlying CLI commands and shell scripts. A production implementation would also need to strictly enforce user authorization checks and validate user input. This is discussed in further detail in the Avamar section. Examples of the PowerShell scripts we used are shown in Avamar Scripts. 12

Reporting capabilities The reports included with Avamar and Data Protection Advisor (DPA) provide an overall view of the backup and storage environment. Figure 5 illustrates DPA reports which were integrated into the portal for our reference implementation by scheduling those reports for pickup and display by the portal. Figure 5. Sample list of DPA reporting page in vco 13

EMC Avamar Overview EMC Avamar provides scalable backup and restore capabilities with integrated data deduplication and support for multisite replication. It also supports multitenant implementations through the use of domains. Avamar deduplicates backup data across sites and servers to reduce total disk storage by up to 50 times, enabling costeffective long-term retention on Avamar data store servers. Backup data can also be encrypted in-flight and at-rest for security and privacy. Avamar 6.0 supports Change Block Tracking (CBT) for VMware client recoveries in addition the existing CBT backup support. Avamar 6.0 can also automatically loadbalance across multiple Avamar VMware proxies to simplify and speed-up VMware backups and recoveries. Figure 6 shows the Avamar administrative portal. Figure 6. EMC Avamar Administrator interface This proven solution uses the Avamar Virtual Edition (AVE) for testing and simulation. This implementation is deployed as a virtual machine within VMware. It is intended for smaller deployments up to 2 TB, but it is functionally comparable to a full multinode Avamar grid deployment scaling to 100 TB or more of deduplicated storage. 14

Multi-tenant Support Avamar segregates user data using domains (these are an Avamar management feature and are not tied to Internet domains). Each domain is logically segregated within the Avamar system, with backup metadata for each client assigned and accessible through that domain. By using domains, reporting and other actions within Avamar can be restricted to hosts, clients, or groups within a specific domain or sub-domain. By organizing clients within this hierarchy, it is possible to use Avamar reporting capabilities to generate status and statistical reports about backup related operations. When implementing user access controls in the service portal, each customer should be assigned a domain or sub-domain within the Avamar hierarchy. This hierarchy should be enforced on all backup calls that each user places to the Avamar system through the portal. Whichever user the service portal uses to connect to the Avamar MCCLI should be granted access to the appropriate levels of the Avamar Hierarchy. This is how multitier access controls can be implemented at both the service portal and Avamar levels if required for audit or compliance reasons. CLI and API Support Through the use of the Avamar Management Console Command Line Interface (MCCLI) service providers can provide customized access to the backup, restore, configuration, and reporting aspects of Avamar without requiring direct access to the Avamar Management Console GUI (MCGUI). The MCGUI is a Java software application that can be installed on a Windows or Linux client. Workflows For this project we used the vco SSH plug-in to create workflows that do the following: Gather user input Connect to the AVE server Run the required MCCLI commands or shell scripts Return any output or error codes One important aspect of implementing portal integration around Avamar MCCLI is identity management and access controls. When the SSH plug-in connects to the MCCLI application it runs as a privileged Avamar administrator which can access any available commands. The commands are not run as the portal user. It is the responsibility of the portal code to validate the input and parse the returning MCCLI attributes to determine what information can be presented to the requesting individual. For example, if a tenant administrator requested to see all domains within the Avamar instance the MCCLI request would return all domains not just those that are within that tenant s domain. It is the responsibility of the portal code to review and edit the values passed to and returned from the MCCLI to validate the sub-set of domains the requesting user is permitted to see. It may also be necessary for the portal to make multiple MCCLI calls on behalf of a particular user to first determine what information they are permitted to see and then actually request that information. In this way the 15

portal, which is vco in our solution, manages identities and the access they have into the Avamar environment. Figure 7 shows a workflow design in vco. Figure 7. Designing a workflow in vco MCCLI examples Figure 8 and Figure 9 show two sample MCCLI commands. In these examples, ROOT is the tenant s top level domain, which could be / for service provider administrators creating a new tenant. /usr/local/avamar/bin/mccli domain add -domain= ${ROOT} -location= ${NAME} - email= ${EMAIL} - contact= ${CONTACT} --name= ${DOMAIN} Figure 8. Example script using MCCLI to create a new domain /usr/local/avamar/bin/mccli client add --location= ${LOCATION} contact= {CONTACT} --domain= ${ROOT}${DOMAIN} name= ${HOST} Figure 9. Example script using MCCLI to add a host to a domain 16

The MCCLI returns error and status codes and messages upon execution of each command. And command output is also returned as shown in Figure 10. # /usr/local/avamar/bin/mccli domain add --name="/cust001" 0,22527,Domain added. Attribute Value --------- -------------------------------------------------------- ----------------------- domain <appdircomponent contact="" domain="/" email="" id="" name="cust001" phone=""/> # echo $? 0 # /usr/local/avamar/bin/mccli domain add --name="/cust001" 1,22541,Domain already exists. # echo $? 1 Figure 10. Example using MCCLI to show status and return codes The error code and message numbers can be used to quickly parse and process the output from each MCCLI command. Currently, the ability to integrate Avamar-based VMware client recovery with a portal is limited in Avamar 6.0. It is possible to access all Avamar VMware client backup capabilities through the MCCLI just not all the MCCLI recovery actions. Full documentation for configuring Avamar using MCCLI is provided in the Avamar Management Console Command Line Interface (MCCLI) Programmer Guide. Configuration Database Access Limitations and workarounds It is possible to directly access the Enterprise Management Server (EMS) or Management Console Server (MCS) databases in a read-only manner to provide direct access to the Avamar configuration. Querying the database directly may allow more customization of the service provider s portal integration. The database views exposed are documented in the Avamar Administration Guide. One of the challenges involved in deploying Avamar in service provider environments is the requirement that each Avamar client should have a unique IP address to communicate with the Avamar backup server. This unique IP address is required to establish bidirectional communication between the backup client and the Avamar server. A unique IP address isn't required to just back up the client, but is required for restoration operations. For more details on how to design solutions refer to the EMC white paper, Creating Backup as a Service (BaaS) Solutions Leveraging EMC Avamar, as well as the product documentation. 17

EMC Data Protection Advisor Overview EMC Data Protection Advisor (DPA) is a sophisticated reporting and analytics platform that provides customers with full visibility into the effectiveness of their data protection strategy. It performs this by monitoring all of the technologies that a customer uses to protect their data including backup software, storage arrays and file servers. The DPA reporting engine provides customizable reports to highlight problems with the environment, and enables customers to perform: Capacity management Service level reporting Chargeback Change management Troubleshooting The DPA Predictive Analysis Engine provides customers with early warning of problems that might be about to occur, and generates alerts allowing customers to resolve problems sooner, reducing business impact. Figure 11 shows a typical DPA view. Figure 11. Storage environment viewed through EMC Data Protection Advisor 18

Reporting DPA provides standard Avamar specific reports such as client count, daily backup data, job status, and so on. These standard reports can be used by service providers to monitor the health of their backup environment. In a multitenant environment DPA is able to run reports on each tenant (each Avamar domain ). This can be done by DPA as it is aware of the association between clients and the domain that each client belongs to. Similarly, as clients are added and removed from domains the reports that DPA runs will reflect that information. Figure 12 shows a DPA multitenant view. Figure 12. DPA multitenant view 19

EMC HomeBase is also integrated with DPA for reporting purposes. It will automatically configure DPA for a new tenant s client which DPA will then include in future reports for billing. DPA can also be used to generate reports on the success and failure of HomeBase installations and backups along with whether profiles were successfully captured from existing and new clients CLI and API support DPA provides the following mechanisms through which its output can be integrated into a web-based portal, including: Scheduling reports to run automatically and their output stored in a location which can be accessed by the portal Directly running reports from the command line and specifying where the report output will be stored Scheduling reports The recommended approach for making DPA reports available to the portal is as follows: Schedule the reports to be run on a regular basis. Store the output of the reports in a hierarchical file-system sorted by tenant and report, and which can be accessed by the portal. Have the portal code scan for new reports when those pages of the portal are accessed. Figure 13 shows a sample screen of the DPA Portal webpage as well as the actual reports, which had previously been scheduled. 20

Figure 13. DPA de-dupe rate distribution report On-demand reports The second option for integrating report output into the portal is by providing users with the ability to directly execute a report. The user selecting this option will have to wait for the report to be run by the DPA engine but will get an up-to-the-minute report. In this case the portal code will execute the script and once complete display the resulting report to the user. This mechanism should be used sparingly and only if necessary as it will be very difficult to predict how long the report will take to run. Using this option for reports which take more than a few minutes to run is strongly discouraged. Users should be warned that the portal will not display the report until it has been completed, and the next portal page will not appear instantaneously as when displaying already-run reports. Figure 14 shows a sample portal screen and the subsequent report. 21

Figure 14. DPA SLA client summary 22

EMC HomeBase Overview EMC HomeBase provides fast, repeatable, bare-metal server recoveries and migrations across dissimilar hardware. HomeBase automatically creates and stores server configuration profiles based on your schedules and retention policies, and can apply these profiles to new hardware to recover a server, readying it for immediate operations. HomeBase also provides server configuration and change reporting capabilities based on its profiling technology. HomeBase integration with Avamar provides complete business resiliency, while reducing the amount of storage required to enable full system recovery when compared to traditional imaging solutions. Where imaging solutions generate images that are thousands of megabytes in size, HomeBase creates configuration profiles of just a few megabytes and restores all other needed files from the existing Avamar backup. This combination provides a fast, comprehensive server recovery solution with minimal storage requirements. In addition, the integration of HomeBase with Avamar allows fully automated and unattended one-click restores of supported Windows and RHEL servers across dissimilar hardware platforms and between physical and virtual server stacks. HomeBase profiling is initiated using the Avamar pre-scripting capability during the backup, and full system recoveries are driven from the HomeBase Server console. HomeBase 6.6 adds a variety of capabilities for further automating recovery to VMware virtual machines and for increased multitenant security, including: Multitenancy for recovery sessions, ensuring that an administrator initiating recoveries through the HomeBase portal can only see their specific clients. vsphere integration to automatically provision a virtual machine with specifications (CPU, memory, disk, and so on) matching the source physical server as part of the process when recovering to VMware-based virtual systems. Figure 15 shows the HomeBase user interface. 23

Figure 15. Standard EMC HomeBase administrative user interface Because HomeBase easily integrates with existing backup workflows, server configuration recovery information is always synchronized with data recovery information, ensuring reliable and simple server recovery. HomeBase is easily integrated into DPA with a few simple steps, enabling DPA to automatically detect new HomeBase enabled servers and include these in future reports for billing as well as reports on the status of HomeBase profiles for a client. CLI and API Support The HomeBase server is designed using the latest Service Orientated Architecture (SOA). The HomeBase server provides a REST based API to make its operating system and hypervisor provisioning capability available to internal and external integrators. Using this flexible API, HomeBase allows server recovery workflows to be easily integrated with data backup workflows, ensuring that server recovery information is always in sync with data recovery information. Similarly, this REST-based API can be used to integrate HomeBase into a service provider s portal as well as automating agent installation and configuration options. In HomeBase 6.6 the REST API does not support recovery operations. These can only be done through the HomeBase portal. The REST API is thoroughly documented in the EMC HomeBase user documentation. Our solution did not do any integration of HomeBase into the Portal. 24

Avamar Scripts Overview This section describes examples of the scripts we used to integrate Avamar with our Backup-as-a-Service solution platform. Note: These scripts are presented as examples only. Any scripts used in your own environment must be written for your specific application. EMC does not endorse or support these scripts beyond informational purposes. General script notes All of the example scripts presented here are shell scripts, placed on the Avamar (Linux) server. The complete path is required to run them in the vcenter Orchestrator. All scripts run the Avamar MCCLI command line utility with required arguments. They are run by the vco SSH plug-in. All scripts run as the root user of Avamar server. For production environments, a different security approach may be required. For information about building your own custom solutions using MCCLI, refer to the Avamar Management Console and Command Line Interface (MCCLI) Programmer s Guide. Throughout this section, domain refers to the Avamar domain, not the Active Directory domain. The Avamar domain is similar to a folder. All objects related to that account (tenant) reside in that folder. Security can be set on Avamar domains to restrict tenants ability to see other tenant information. It is expected that the Avamar client is already installed on all the client machines before a machine can participate in the backup program. One way is to provision the VM image with the Avamar client already installed. If an existing machine does not have the client, it must be installed first, before it can participate in the backup program. Avamar client is available from the Avamar server itself. Service provider tasks List all Avamar domains and sub-domains present in the system This script lists all domains and sub-domains in a given Avamar domain. If the recursive option is removed, it only gets the sub-domains of a given domain. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: /Tenants) echo Listing domains of $1 <path>/avamar/bin/mccli domain show --recursive=true --domain=$1 Create an Avamar domain This is the first step for provisioning a tenant backup space in the Avamar system. All tenants object (sub-tenants, machines names, backup policies, schedules, and so on) reside in this domain. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: Tenants/Tenant-01) 25

(Assumes tenants Avamar domain is already existing) echo Adding the Avamar Domain $1 <path>/avamar/bin/mccli domain add --name=$1 <path>/avamar/bin/mccli domain show --name=$1 Deleting an Avamar domain To delete a domain all objects need to be deleted first. The force option can be used without doing so, but that must to be used with caution as it will delete all child domains and the machines participating in those domains, policies, groups, schedules, and datasets present in those domains. To use the force option, check the MCCLI programming guide. Input Arguments in sequence $1 = Complete root domain path where the domain need to be deleted is present, without the domain name itself(ex: /Tenants) $2 = Just the name of the Avamar Domain to be deleted (ex: Tenant- 01) echo Deleting the Avamar Domain $2 from $1 <path>/avamar/bin/mccli domain delete --name=$2 --domain=$1 <path>/avamar/bin/mccli domain show --name=$1/$2 --recursive=true Tenant admin tasks The tasks described in Service provider tasks can also be added as tenant admin tasks too, as they must manage their own sub-domains/sub-tenants and the objects under that. But security needs to be set at the tenant level so that they can t see other tenant information. Add a machine to the Avamar domain Adding a machine to the Avamar domain is a two step process. First it needs to be added to the domain and then it needs to be activated (invited in Avamar terms which can be done from the client side or from the server side, but can only be done from the server side in this solution). Adding a machine does not automatically backup the machine. Adding the machine name lets the server assign a unique ID for the client to participate in all the backup operations. When the machine is added to a group, then only the machine backup happens as defined in the dataset. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: Tenants/Tenant-01) $2 = Complete machine name (ex:tenantmachinename) echo Adding the Client $2 to the Avamar Domain $1 <path>/avamar/bin/mccli client add --name=$1/$2 <path>/avamar/bin/mccli client show --domain=$1 <path>/avamar/bin/mccli client invite --name=/$1/$2 <path>/avamar/bin/mccli client show --domain=$1 26

List client s domain name This script gets the complete domain path of the machine. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: Tenants/Tenant-01) $2 = Complete or partial machine name (ex:winxptest) echo Listing client and its domain name <path>/avamar/bin/mccli client show --domain=$1 grep $2 Delete client from a domain Deleting a client from a domain is the same as removing the machine from the entire backup system. If it is added again, it will be treated as a new machine and will have a new unique ID. Also, all backups related to that machine will be marked for deletion. To move between the domains, the move operation should be used (refer to the MCCLI programming guide). Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: Tenants/Tenant-01) $2 = Complete or partial machine name (ex:winxptest) echo Deleting the Client $2 from the Avamar Domain $1 <path>/avamar/bin/mccli client delete --name=$1/$2 <path>/avamar/bin/mccli client show --domain=$1 The tenant admin must set up the following: Dataset (the data to be backed up) Retention policy (how long a backup must be kept in the system) Schedule (when and what interval the backup needs to be performed) Group (to have all these objects plus the machine names participating in particular backup program). Usually these are set once, and future machines follow the same backup pattern as the other machines in the same group. Create a default dataset This is required to define what to back up. In this script we are backing up the complete machine. For default dataset details, refer to the MCCLI programming guide. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: /Tenants/Tenant- 01) $2 = Dataset name (to easily identify _DS is added in the script, but this is not required.) 27

echo Creating a DEFAULT dataset called $2_DS in the Avamar Domain $1 <path>/avamar/bin/mccli dataset add --name=$1/$2_ds <path>/avamar/bin/mccli dataset show --recursive=true --domain=$1 grep $2_DS Create a custom dataset A default or custom dataset is required to define what to back up. In Create a default dataset we backed up the complete machine. In this script we can define a particular file, folder, database, or anything that is supported by Dataset definitions. For dataset definition details, refer to the MCCLI programming guide. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: /Tenants/Tenant- 01) $2 = Dataset name (to easily identify _DS is added in the script, but this is not required.) $3 = Target folder to backup (C:/Temp, do not use back slash, C:\temp is not recognized.) echo Creating a custom dataset called $2_DS in the Avamar Domain $1 <path>/avamar/bin/mccli dataset add --name=$1/$2_ds -- alldata=false echo Adding Windows File System Plugin to the Dataset <path>/avamar/bin/mccli dataset add-target --name=$1/$2_ds -- target=$3 --plugin=3001 echo Listing the Dataset just created <path>/avamar/bin/mccli dataset show --domain=$1 grep $2_DS Create a retention policy A retention policy is required to define how long a backup must be retained. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: /Tenants/Tenant- 01) $2 = Dataset name (to easily identify _RP is added in the script, but this is not required.) $3 = Enter the number of day or months or years the Policy has to expire after, from today. Example: To expire this policy after 5 days, just input "5D" without quotes. Similarly 13W for 13 weeks 3Y for 3 years An exact date can also be mentioned, but the format, YYYY-MM-DD echo Creating a Retention Policy called $2_RP in the Avamar Domain $1 <path>/avamar/bin/mccli retention add --domain=$1 --name=$2_rp -- basic=$3 echo Listing the Retention Policy details that is just created <path>/avamar/bin/mccli retention show --name=$1/$2_rp 28

Create a schedule A schedule is required to define when to perform the back up, and at what interval. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: /Tenants/Tenant- 01) $2 = Dataset name (to easily identify _RP is added in the script, but this is not required.) $3 = Either one of the following argument is required. To back up at specific intervals [--hours=string]: Set the time of day for a daily schedule in 24-hour format. Example: --hours=2,5,7,10,23 To back up on selected weekdays [--days=string]: Set the days of week for a weekly schedule, or the day of month for a monthly schedule. Valid values are M[onday], Tu[esday], W[ednesday], Th[ursday], F[riday], Sa[turday], and Su[nday]. Example: --days=m,tu,f,sa To back up on a particular day of the month [--nth-day=string]: Set the nth day of a month for a monthly schedule. Valid values are 1, 2,..., 28, and last. Example: --nth-day=12,23,last To back up on a particular week of the month [--week=string]: Set the week of the month for a monthly schedule. Valid values are first, second, third, fourth, and last Example: -week=second Optional arguments [--desc=string]: You can enter textual description of the schedule [--duration=string]: Back up window in format HH:MM. Example: --duration=5:00 [--start=string] Start time in format HH:MM (24 hour format) Example: --start=13:30 [--tz=string] Time zone for start time defaults to time zone of machine. Example: --tz=cst OR --tz=america/toronto echo Creating a schedule called $2_SCH in the Avamar Domain $1 <path>/avamar/bin/mccli schedule add --name=$1/$2_sch $3 <path>/avamar/bin/mccli schedule show --name=$1/$2_sch 29

Create a group A group is required to organize the dataset, retention policy, and schedule in addition to the machine names participating in this backup plan. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: /Tenants/Tenant- 01) $2 = Dataset name (to easily identify _RP is added in the script, but this is not required.) $3 = Boolean value (true/false) - Making this value true will immediately enable the scheduled backups. Making it false keep everything ready for future usage. echo Creating a Group called $2_GRP in the Avamar Domain $1 echo This is used to hold Dataset, Retention Policy, Schedule and the MachineNames to be backed up. <path>/avamar/bin/mccli group add --domain=$1 --name=$2_grp -- enabled=$3 echo Listing the Group details that is just created <path>/avamar/bin/mccli group show --name=$1/$2_grp Tenant admin master script The following script performs the domain, dataset, retention, and scheduling tasks. Input Arguments in sequence $1 = Complete Avamar domain name with path (ex: /Tenants/Tenant- 01) $2 = string Name used to create <string>_ds, <string>_rp, <string>_grp, <string>_sch $3 = Machine name to backup. echo Creating a dataset called $2_DS in the Avamar Domain $1 <path>/avamar/bin/mccli dataset add --name=$1/$2_ds <path>/avamar/bin/mccli dataset show --recursive=true grep '$2_DS' echo Creating a schedule called $2_SCH in the Avamar Domain $1 <path>/avamar/bin/mccli schedule add --name=$1/$2_sch -- hours=11,12,15,18,23 <path>/avamar/bin/mccli schedule show --name=$1/$2_sch echo Creating a Retention policy called $2_RP in the Avamar Domain $1 <path>/avamar/bin/mccli retention add --name=$1/$2_rp <path>/avamar/bin/mccli retention show --name=$1/$2_rp echo Creating a Group called $2_GRP in the Avamar Domain $1 <path>/avamar/bin/mccli group add --name=$1/$2_grp -- dataset=$1/$2_ds --enabled=true --retention=$1/$2_rp -- schedule=$1/$2_sch <path>/avamar/bin/mccli group show --name=$1/$2_grp echo Adding the machine to the group $2_GRP 30

<path>/avamar/bin/mccli group add-client --client-name=$1/$3 -- name=$1/$2_grp <path>/avamar/bin/mccli group show-client-members --name=$1/$2_grp Tenant user tasks These scripts show examples of tenant user tasks. Add machines to the existing backup group. This script adds the machine names to a group that is already defined by the tenant admin. This script does the following: Searches for the machine Gets the domain of the machine Finds the respective group and adds the machine to the group. If the group is already activated, the back up happens with the other machines in that group. This script can also be performed by the tenant admin. It is also possible to create a script that adds a bulk number of machines to the group. For more information about bulk adding, refer to the MCCLI programming guide. Input Arguments in sequence $1 = Exact Tenant User s machine name #!/bin/bash #IFS is used to split the input at a pattern export IFS=" " # accept the case insensitive machine name as input and convert to upper case macname=`echo $1 tr [:lower:] [:upper:]` echo "macname=$macname" export MACHINE="foo" export DOMAIN="bar" # check if there a machine exists in the entire avamar domains listmachines=`<path>/avamar/bin/mccli client show --recursive=true grep -i $macname` #lop thorugh each machine and see if it matches with the machine name passed as input argument for eachmachinename in $listmachines; do <path>/avamar/bin/mccli client show --recursive=true grep -i $macname read eachmachinename validdomainname junk #convert each line to upper case test=`echo $eachmachinename tr [:lower:] [:upper:]` # echo "test=$test" #check if it matches with the machinename passed as input if [ "$test" == "$macname" ] then #if matches, accept this as valid machine name #echo $eachmachinename found validmachinename=$eachmachinename 31

# echo "validmachinename=$validmachinename" # echo "validdomainname=$validdomainname" # echo "MACHINE=$MACHINE DOMAIN=$DOMAIN" MACHINE=$validMachineName DOMAIN=$validDomainName # echo "MACHINE=$MACHINE DOMAIN=$DOMAIN" # next fi # get the complete path (Avamar Domain Name) of the machine name in Avamar system # checks if the first letter is / #if [[ $test == /* ]] #then # equal this to the domain name # echo $eachmachinename found validdomainname=$eachmachinename #fi done MACHINE=$validMachineName DOMAIN=$validDomainName # echo "MACHINE=$MACHINE DOMAIN=$DOMAIN" # echo "MACHINE=$MACHINE DOMAIN=$DOMAIN" <path>/avamar/bin/mccli group show-client-members -- name=${domain}${domain}_grp <path>/avamar/bin/mccli group add-client --clientname=${domain}/${machine} --name=${domain}${domain}_grp <path>/avamar/bin/mccli group show-client-members -- name=${domain}${domain}_grp 32

Conclusion Summary This solution provides service providers with an integrated carrier-grade, scalable, multitenant backup service which can backup and restore physical and virtual machines. As organizations increase their use of out-sourced data centers, their backup challenges can also grow. Service providers who already offer cloud-based services or traditional hosting services are ideally positioned to provide local BaaS for customers to round out their other as-a-service offerings. allows service providers to provide robust backup protection leveraging EMC Avamar and HomeBase technologies. EMC BaaS can also deduplicate data stored in virtual disks, significantly reducing storage consumption and enabling replication of virtual disks across data center locations. This solution provides a reference implementation for delivering backup services that leverage a service provider s existing orchestration and portal infrastructure. EMC BaaS leveraging EMC Data Protection Advisor technology provides the enhanced reporting capabilities that customers demand including backup job status, used capacity; restore job status, and daily compression rate reports. Findings We found the following key results during the testing of this solution: The EMC BaaS solution with EMC Avamar, EMC Data Protection Advisor, and EMC HomeBase supported per-customer backup services on a service provider multitenant cloud platform. The EMC BaaS solution with VMware vcloud Director and vcloud Orchestrator can integrate Avamar and Data Protection Advisor with industry-leading orchestration and portal solutions. The EMC BaaS solution successfully backed up and restored user data over LAN networks. The backup and restore support was all encompassing, including: files, applications, system backups, virtual machine image backups, and bare-metal backup of physical servers. 33

About EMC Proven Solutions Take the next step EMC Proven Solutions help customers identify and overcome business challenges by reducing risk and time-to-value of their information infrastructure. EMC leverages its expertise and proven technologies with its strategic relationships with Cisco, Microsoft, Oracle, SAP, and VMware to deliver solutions that support our customers business and technical requirements. All solutions are rigorously tested and documented with reference architectures and best practices designed to reduce the total cost of ownership of the infrastructure and increase IT Efficiency. EMC offers a portfolio of consulting and professional services for service providers and their customers to assist in balancing workloads across service delivery models ranging from legacy physical architectures and virtualized infrastructures through on and off-premise cloud architectures. The EMC Cloud Advisory Service with Cloud Optimizer helps customers develop a strategy for optimizing the placement of application workloads. By assessing three factors economics, trust and functionality organizations can maximize their cost savings and business agility gained through the use of private and public cloud resources. References White papers Product documentation For additional information, see the white papers listed below. EMC documents are available on the EMC online support website. Compute-as-a-Service (EMC) Understanding EMC Avamar with EMC Data Protection Advisor Applied Technology (EMC) For additional information, see the product documents listed below. VMware vcloud Director Documentation VMware vsphere Documentation VMware vcenter Orchestrator Documentation Avamar 6.0 Management Console Command Line Interface (MCCLI) Programmer Guide (EMC) EMC Data Protection Advisor API Reference (EMC) 34