Prepared for: How to Become Cloud Backup Provider

Transcription

1 Prepared for: How to Become Cloud Backup Provider

2 Contents Abstract... 3 Introduction... 3 Purpose... 3 Architecture... 4 Result... 4 Requirements... 4 OS... 5 Networking... 5 Database... 5 Permissions... 5 Hardware recommendations... 5 Installation of Linux server... 6 Useful tips... 6 To set network... 7 Installing dependencies... 7 Openstack installation... 8 Useful tips... 8 Integration of Openstack object storage with S Useful tips CloudBerry Managed Backup configuration and usage About Openstack Foundation About CloudBerry Lab Appendix A: CloudBerry Lab stand alone products usage... 16

3 Abstract In this white paper we will explain how one can install and configure cloud storage software and use it for backup purposes. Document can be used by an individual or a company who wishes to become a cloud backup provider and utilize its own hardware for cloud object based storage. We will use Openstack object storage software along with CloudBerry Lab products. Introduction Object based storage is considered a leading emerging technology, and many of its features are deemed ideal for cloud storage. Benefits of object storage include massive scalability, geographic independence, multi tenant features and the ability to use off the shelf hardware. More companies are considering object based storages as a part of their infrastructure, for instance for backup purposes. They choose it for its benefits and price. There is a number of public object based cloud storages offered on the market, such as Amazon S3, Glacier, Windows Azure, Rackspace, and many others. However, there are still many companies who already have existing hardware and want to transform a part of it or transform it entirely in a cloud storage, utilizing latest technologies. It s also a huge playground for Managed Service Providers (MSP), who are willing to provide cloud storage and backup services to their final customers at the same time. New technologies, such as object based storage require new tools, like products of CloudBerry Lab. Purpose This white paper may be used by Managed Service Providers who are willing to expand their portfolio by providing cloud backup services to their customers using their own private cloud storage and CloudBerry Managed Backup. There are two ways to provide cloud storage to the final customers from MSPs point of view. First one is to create new account on public cloud storage provider, like Amazon, for example, and resell it. However, there is a number of MSPs who run their own hardware and want to utilize its capacity to construct their own private cloud storage built on their own resources. In this white paper, we will briefly explain the second way: how this can be done by using Openstack object storage software. We will explain how to use CloudBerry Managed Backup to offer online backup service to the final customer. As an add on, we will show how this cloud storage can be accessed and used by stand alone products of CloudBerry Lab for object based storages.

4 Architecture It is recommended to install the Openstack software in a cluster of three or more nodes. For this documentation, we deployed a single node installation of Openstack Havana in a virtual machine running Red Hat Enterprise Linux 6.4. Result We were able to install and configure Openstack object storage software on a stand alone Linux server and use it together with CloudBerry Managed Backup, Backup and Explorer products. Thus, we proved a concept that any individual or company, MSP especially, may use this basic scenario to become a Managed Backup Provider to the cloud storage, built on their own hardware resources. We encourage everyone to go through the following sections to get the basic idea of the technical requirements, tips, and general steps of the installation.

5 Requirements OS OpenStack Object Storage currently runs on Ubuntu, RHEL, CentOS, Fedora, opensuse, or SLES. Please check this link to find Openstack guides for your version of Linux. Networking 1Gpbs or 10 Gbps is suggested internally. For OpenStack Object Storage, an external network should connect the outside world to the proxy servers, and the storage network is intended to be isolated on a private network or multiple private networks. Database For OpenStack Object Storage, a SQLite database is part of the OpenStack Object Storage container and account management process. You should also install MySQL as a pre requirement. Permissions You can install OpenStack Object Storage either as root or as a user with sudo permissions if you configure the sudoers file to enable all the permissions. Hardware recommendations Server Recommended Hardware Notes Object Storage object servers Object Storage container/account servers Processor: dual quad core Memory: 8 or 12 GB RAM Disk space: optimized for cost per GB Network: one 1 GB Network Interface Card (NIC) Processor: dual quad core The amount of disk space depends on how much you can fit into the rack efficiently. You want to optimize these for best cost per GB while still getting industry-standard failure rates. At Rackspace, our storage servers are currently running fairly generic 4U servers with 24 2T SATA drives and 8 cores of processing power. RAID on the storage drives is not required and not recommended. Swift's disk usage pattern is the worst case possible for RAID, and performance degrades very quickly using RAID 5 or 6. As an example, Rackspace runs Cloud Files storage servers with 24 2T SATA drives and 8 cores of processing power. Most services support either a worker or concurrency value in the settings. This allows the services to make effective use of the cores available. Optimized for IOPS due to tracking with SQLite databases.

6 Server Recommended Hardware Notes Memory: 8 or 12 GB RAM Object Storage proxy server Network: one 1 GB Network Interface Card (NIC) Processor: dual quad core Network: one 1 GB Network Interface Card (NIC) Higher network throughput offers better performance for supporting many API requests. Optimize your proxy servers for best CPU performance. The Proxy Services are more CPU and network I/O intensive. If you are using 10g networking to the proxy, or are terminating SSL traffic at the proxy, greater CPU power will be required. Installation of Linux server For this single node installation we used Red Hat Enterprise Linux. We installed it on the VMware VM. You can go through these procedures in order to prepare your server for Openstack installation. Useful tips Confirm that: The machine s hostname is not set to localhost. The machine s hostname is not mapped to the loopback address ( ) in /etc/hosts. The host is assigned a static IP address. Do not use DHCP. To Set Host Name # vim /etc/sysconfig/network

7 To set network Don t forget to disable Network Manager as described in pre requirements! # nano /etc/sysconfig/network-scripts/ifcfg-eth0 To disable firewall # service iptables save # service iptables stop # chkconfig iptables off Installing dependencies On apt based systems, 1. apt get update 2. apt get install curl gcc memcached rsync sqlite3 xfsprogs git core libffi dev python setuptools 3. apt get install python coverage python dev python nose pythonsimplejson python xattr python eventlet python greenlet pythonpastedeploy python netifaces python pip python dnspython python mock On yum based systems, 1. yum update 2. yum install curl gcc memcached rsync sqlite xfsprogs git core libffi devel xinetd pythonsetuptools 3. yum install python coverage python devel python nose python simplejson python xattr pythoneventlet python greenlet python pastedeploy python netifaces python pip python dnspython python mock This installs necessary system dependencies; and most of the python dependencies. Later in the process setuptools /distribute or pip will install and/or upgrade some other stuff it s getting harder to avoid. You can also install anything else you want, like screen, ssh, vim, etc. You also may subscribe to alternative repositories, like EPEL (Extra Packages for Enterprise Linux) If you use RedHat, you can subscribe to optional channels. This may help with Puppet installation.

8 Openstack installation Here we won t show the complete procedure, just give a brief idea and some useful tips, which might help you setting up the test Openstack node. You can download manuals/installation guide from here. At a high level, your single node Openstack installation will proceed as follows: 1. Fulfill the pre requirements for the installation 2. Configure the Identity Service 3. Configure The Image Service 4. Configure Compute Services 5. Add a Dashboard 6. Add Object Storage NB! Note that you can install Openstack on one node. Please follow this link to learn more. You'll be able to add more nodes to your Openstack cloud later, if you choose. Useful tips NB! Current version of Havana documentation misses the step of creation of Object Storage service and addition of the endpoint. You can follow this steps: keystone service-create --name=swift --type=object-store --description="object Storage Service" Property Value description Object Storage Service id 272efad2d cbb911c1e5a5a6ed name swift type object-store $ keystone endpoint-create \ --region RegionOne \ --service-id=272efad2d cbb911c1e5a5a6ed \ --publicurl ' \ --internalurl ' \ --adminurl ' Property Value

9 adminurl id e32b3c4780e51332f9c128a8c208a5a4 internalurl publicurl region RegionOne service_id 272efad2d cbb911c1e5a5a6ed For RedHat, CentOs, Fedora several steps would be different: the memcached config is in `/etc/sysconfig/memcached`, not `/etc/sysconfig/memcached.conf` Use service openstack-swift-proxy start, instead of service proxy-server start To set up DNS on the main DNS server, you will need 3 entries (in our case x, hostname openstack, default domain openstack.example): openstack IN A xxx openstack.example IN A xxx *.openstack.example IN A xxx After the installation check: #swift-init all status You should get the similar output: container updater running (6528 /etc/swift/container server.conf) account auditor running (6529 /etc/swift/account server.conf) object replicator running (6530 /etc/swift/object server.conf) proxy server running (6531 /etc/swift/proxy server.conf) container replicator running (6532 /etc/swift/container server.conf) object auditor running (6533 /etc/swift/object server.conf) object expirer running (6534 /etc/swift/object expirer.conf) container auditor running (6535 /etc/swift/container server.conf) container server running (6536 /etc/swift/container server.conf) account server running (6537 /etc/swift/account server.conf) account reaper running (6538 /etc/swift/account server.conf) container sync running (6540 /etc/swift/container server.conf)

10 account replicator running (6541 /etc/swift/account server.conf) object updater running (6542 /etc/swift/object server.conf) object server running (6543 /etc/swift/object server.conf) You can also run: #swift --os-username username --os-tenant-name tenant-name --os-password password -V 2.0 -A stat To verify that your Openstack Object Storage is running well. You should get the similar: Account: AUTH_17ad08b3cc224a428e6b09a043170f48 Containers: 0 Objects: 0 Bytes: Accept Ranges: bytes X Timestamp: Content Type: text/plain; charset=utf 8 Integration of Openstack object storage with S3 To use Openstack object storage you may use a separate middleware, which is located here. You may simply download it a zip file, and install using the following commands: sudo python setup.py install or sudo pythin setup.py develop After this you should add the following lines to the proxy server.conf file (please adjust it to your environment): [pipeline:main] pipeline = healthcheck cache swift3 s3token authtoken keystoneauth proxy server [filter:swift3] use = egg:swift3#swift3 [filter:s3token] paste.filter_factory = keystone.middleware.s3_token:filter_factory service_port = 5000

11 service_host = openstack.example auth_port = auth_host = auth_protocol = http auth_token = d0edc7087f ae9 admin_token = d0edc7087f ae9 To generate new credentials (access and secret keys) for s3 object storage use the following procedure: get tenant id with #keystone tenant-list get admin id with #keystone user-list And use this: #keystone --os-auth-url --os-username your_admin_name --os-tenant-name your_tenant_name --os-password your_password ec2-credentials-create user_id 7946ddaeb1b949778c1fb882596a tenant_id 17ad08b3cc224a428e6b09a043170f48 As an output you should receive the table with access and secret keys: Property Value access 9a4be325c0ec4c5682b5cf48735e82c2 secret SECRETSECRETSECRETSECRETSECRET! tenant_id 17ad08b3cc224a428e6b09a043170f48 user_id 7946ddaeb1b949778c1fb882596a Useful tips If you get a following error after installation of the s3 middleware and your proxy server fails to start: message_format = _("Expecting to find %(attribute)s in %(target)s." NameError: name '_' is not defined Then this is a known bug, you can fix it like described here.

12 CloudBerry Managed Backup configuration and usage First go to your Managed Backup Console page and provide your credentials. You can go to My Account > Switch to my own account and choose Amazon s3 compatible account (Pic 1, 2). Then select storage provider and provide cloud storage credentials (Pic.3) that we ve copied from Openstack installation. After finishing the wizard, you will be able to backup your clients to your own cloud storage. To make it happen, please create an arbitrary user (Pic.4), this user will receive a download link for backup client (Pic.5, 6). After installation which is very similar to installation of stand alone CloudBerry Backup, this user becomes the first user of your managed backup service. He may download the backup client and use his username and password in order to backup his files to your object storage..please check Appendix A to get the basic idea how to configure stand alone products of CloudBerry Lab with this new storage. Pic.1. Selecting a private cloud object storage, based on Openstack (S3 compatible).

13 Pic.2. Selecting a private cloud object storage, based on Openstack. Pic.3. Specifying credentials of a private cloud object storage for Managed Backup account

14 Pic. 4. Creating the first customer s profile. Pic.5. Customer s link to download Backup client.

15 Pic.10. Customer receives an with initial password and the link to Backup client. About Openstack Foundation. OpenStack is a cloud computing project aimed at providing an infrastructure as a service (IaaS). It is free and open source software released under the terms of the Apache License. The project is managed by the OpenStack Foundation, a non profit corporate entity established in September 2012 to promote OpenStack software and its community. The technology consists of a series of interrelated projects that control pools of processing, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering its users to provision resources through a web interface. (by Wikipedia) About CloudBerry Lab CloudBerry Lab was established by a group of experienced IT professionals with the mission to help organizations in adopting Cloud computing technologies by closing the gap between Cloud vendors propositions and consumer needs through development of innovative low cost backup solutions.

16 Appendix A: CloudBerry Lab stand alone products usage CloudBerry Lab also develops stand alone products: Backup robust backup product, Explorer file manager, Drive product, which maps cloud storage as a virtual disk with the Windows environment. If you set everything up, you can now use your cloud storage with CloudBerry Explorer, Backup and Drive. Please go to File > S3 compatible > S3 compatible (Pic.11) and enter your Service Point name, Access and Secret keys (Pic.12). As the process is very similar for other products, we want to show it. Pic.11. Connecting CloudBerry Explorer to new cloud object storage.

17 Pic.12. Specifying credentials of a private cloud object storage for CloudBerry Explorer.