HP CloudSystem Foundation and Enterprise Software: Recommended Backup and Restore Procedures

Transcription

1 HP CloudSystem Foundation and Enterprise Software HP CloudSystem Foundation and Enterprise Software: Recommended Backup and Restore Procedures This document outlines the detailed steps to back up and restore HP CloudSystem Foundation and Enterprise Software management appliance virtual machines. This document also details the synchronization actions that occur when restoring HP CloudSystem management appliance virtual machines. The recommended approach HP CloudSystem management appliances run in virtual machines. The management appliances include CloudSystem Foundation base appliance the primary appliance for the CloudSystem Foundation product, where Openstack technologies and other services run. Software Defined Network appliance the appliance running HP SDN Controller, which implements the OpenStack Networking (Neutron) service. Network node appliances three appliances running the Neutron network node service. VMware vcenter proxy appliance(s) one or more appliances running Openstack proxy services that integrate with vcenter Server for communication with ESX virtual machine instances. CloudSystem Enterprise appliance when CloudSystem Enterprise is deployed, this appliance runs the additional Enterprise software. Together, these are referred to as the management appliance virtual machine.

2 The recommended approach leverages the capabilities of the hypervisor on which the management appliance virtual machines are installed. Only the backup and restoration of the management appliances are covered here. This document does not describe the cloud (consisting of KVM hosts and/or ESX clusters) managed from CloudSystem Foundation or CloudSystem Enterprise appliances. For associated software, such as VMware vcenter Server and ESX clusters, HP OneView, or HP Insight Control server provisioning (HP ICsp), consult the documentation available online for these products. The backup procedure requires shutting down all of the appliance virtual machines, creating and copying the backup to a backup store, and starting the virtual machines. Backups should be scheduled whenever significant changes are made to the CloudSystem managed environment as noted in Synchronizing once restore is complete. The restore procedure requires that the appliance virtual machine backup be copied to the same hypervisor management clusters where the appliances ran originally. Refer to Appendix A for specific scenarios and short cuts that could be taken during backup. An overview of the backup procedure 1. Shut down all of the appliance virtual machines. 2. Create a clone or OVA template (based on the underlying hypervisor detailed below) 3. Copy the virtual machine clone files to a backup store. 4. Restart all of the appliance virtual machines. 5. Restart the Nova compute service and hp-ovs-agent service on all KVMmanaged nodes.

3 When to back up management appliances Some categories of operations are lost if these changes are not part of the backup. Therefore, HP recommends that you backup the management appliances after any significant change to the managed clusters or to CloudSystem. These changes include: Adding new compute nodes (KVM hosts, ESX clusters) Registering a new vcenter Server Changing the credentials of the management appliance virtual machines Adding new users (administrators or self-service users) to CloudSystem Foundation or Enterprise Adding new networks to CloudSystem Foundation Modifying virtual machine instances in CloudSystem Foundation or Enterprise Back up an ESX management cluster 1. Gracefully shut down all the virtual machines in the management appliance in the following order: a. CloudSystem Foundation base appliance b. SDN appliance c. Network node appliances d. vcenter Server proxy appliance(s) (if any) e. Enterprise appliance (if installed) 2. Create an OVA template for each of the appliances. a. From the system designated as the backup store, log in to the vcenter Server console. b. Select a VM. c. Select File->Export->Export to OVF template d. In the subsequent screen, provide a name for the template, a directory where the OVA template needs to be copied, and select Choose Single file (OVA) as format. e. Repeat steps a through d for every virtual machine in the appliance management cluster. 3. Start the virtual machines in the same order as they were shut down. a. Foundation base appliance. (Wait until the login screen appears in the browser before continuing.) b. SDN appliance. (Wait until the login screen appears in the hypervisor console before continuing.) The following virtual machine appliances can be started soon after the above appliances have come up. c. Network node appliances d. vcenter proxy appliance(s) (if any) e. Enterprise appliance (if installed)

4 4. If the managed environment includes KVM managed nodes, for each KVM node, restart the following services to enable the nodes to regain connectivity to the CloudSystem appliances. > service openstack-nova-compute restart > service hp-ovs-agent restart > service hp-sdn-agent restart Back up a KVM management host 1. Gracefully shut down all the virtual machines in the management appliance. a. Log in to the KVM management host that is hosting the management appliances. b. On the command line, type virsh list to view a list of appliances. c. Then type virsh shutdown <appliance_vm>. 2. Shut down the appliances in the following order: a. CloudSystem Foundation base appliance b. SDN appliance c. Network node appliances d. vcenter Server proxy appliance(s) (if any) e. Enterprise appliance (if installed) 3. Note the path(s) where the virtual machine appliances are present. You will need this information to place them back at the same location during restore. 4. Create a clone for each virtual machine. a. Log in to KVM management host. b. cd /CloudSystem/images (This is the default path where the images are stored. If the images are copied to a different directory during the creation of the appliance, (in deployer.conf), cd to that folder.) c. virt-clone o <appliance_vm> --auto-clone This command does the following. Creates a clone of the appliance VM with name <appliance_vm>clone. The clone will get created in the same directory as the original files. Creates.qcow2 files in /Cloudsystem/images with names in the form <appliance_vm>-clone.qcow2. The Foundation base appliance creates an additional.qcow2 file called <appliance_vm>-glance-clone.qcow2. d. Create a copy of the VM configuration.xml file virsh dumpxml <appliance_vm>-clone > <appliance-vm>-clone.xml

5 Note: The above command creates a dumpxml file in the directory from where the command is executed. Therefore, it is recommended to execute these commands from the directory where the images are available. For the Foundation base appliance, you do not need to generate a dumpxml file for <appliance_vm>-glance-clone.qcow2. 5. Copy all cloned files to backup system. 6. Start the virtual machines in the same order as they were shut down. a. Foundation base appliance. (Wait until the login screen appears in the browser before continuing.) b. SDN appliance. (Wait until the login screen appears in the hypervisor console before continuing.) c. Network node appliances d. vcenter Server proxy appliance(s) (if any) e. Enterprise appliance (if installed) Log in screen for CloudSystem Foundation 7. After a successful copy, delete the cloned VMs using the following commands a. virsh destroy <appliance_vm>-clone b. virsh undefine <appliance_vm>-clone 8. If the managed environment has KVM managed nodes, for each KVM node, restart the following services to enable the nodes to regain connectivity to the CloudSystem appliances. > service openstack-nova-compute restart > service hp-ovs-agent restart > service hp-sdn-agent restart An overview of the restore procedure 1. Copy the virtual machine clones or OVA template files from the backup store to the management cluster. Log in screen for the SDN appliance from the VMware vcenter Server Console The KVM management console would show similar information 2. Create all the appliance virtual machines from the clones or OVA templates. 3. Start the appliance virtual machines. 4. Restart the Nova compute service and hp-ovs-agent service on all the KVM managed nodes. 5. If there are issues in connecting to the storage system(s), make changes to connect them to the appliance. 6. After the appliance virtual machines come up completely, start the resynchronization process (described in Synchronization once restore is complete section below).

6 Restore an ESX management cluster 1. On the same ESX management cluster where the appliances were running earlier, copy the OVA templates from the backup store. The system from where you execute this should be able to see the backed up files a. On the vcenter Server or using a VI Client, select File >Deploy OVF template, and give the path and name of the template. b. Choose the appropriate datacenter, cluster, resource pool and storage options. c. Choose Thin Provision as the disk format. d. Choose defaults for other screens, and press Finish to complete the deployment. 2. Power on the virtual machine appliances in the following order. The first two virtual machine appliances should be started within a minute of each other. a. Foundation base appliance b. SDN appliance The following virtual machine appliances can be started soon after the above appliances have come up. c. Network node appliances d. vcenter Server proxy appliance(s) (if any) e. Enterprise appliance (if installed) 3. If the managed environment includes KVM managed nodes, for each KVM node, restart the following services to enable the nodes to regain connectivity to the CloudSystem appliances. > service openstack-nova-compute restart > service hp-ovs-agent restart > service hp-sdn-agent restart Restore a KVM management cluster 1. Log on to the KVM management host. a. Copy the clone.xml and.qcow2 files from the backup host to the same location from where the files were backed-up (as noted in point 2 in the Back up a KVM Management host section). b. cd /CloudSystems/images 2. Configure and start the appliance virtual machines. a. Optional (only required if the clone was deleted and undefined from the KVM host): virsh define <appliance_vm>-clone.xml b. virsh start <appliance_vm>-clone The first two virtual machine appliances should be started within a minute of each other in the following order.

7 i. Foundation base appliance ii. SDN appliance The following virtual machine appliances can be started soon after the above appliances have come up. iii. Network node appliances iv. vcenter Server proxy appliance(s) (if any) v. Enterprise appliance (if installed) 3. For each KVM node, restart the following services to enable the nodes to regain connectivity to the CloudSystem appliances. > service openstack-nova-compute restart > service hp-ovs-agent restart > service hp-sdn-agent restart Correcting connection to storage system(s) When restoring either ESX clusters or KVM management compute nodes, issues can arise when connecting the storage systems to the restored appliance. If this happens, you can see information about this type of error in the Activity screen in the CloudSystem Console. Use the corrective information as described in the Activity logs to address these errors. 1. Log in to the CloudSystem Console. 2. Navigate to the Activity screen and check for alert information. 3. Navigate from the main menu to Block storage drivers. 4. Edit the affected block storage driver and import the 3Par certificate when prompted. (See the HP CloudSystem Administrator Guide for more information.) Repeat this process for every storage system that fails to connect back to the CloudSystem Foundation base appliance. Synchronizing once restore is complete After the appliance management virtual machines are booted up, all relevant services in the managed nodes (KVM hosts and vcenter Server proxy appliances) are restarted. It is recommended to synchronize the management appliance virtual machines with the managed environment. To synchronize, follow the steps outlined below. Note: The following instructions do not apply to the Enterprise appliance. Execute these steps on the Foundation appliance and managed nodes only. 1. Wait for the management appliance virtual machine to populate the information from the managed environment. In an environment with around 5000 instances, this can take about two hours. 2. Log in to the console using the steps provided in Allow local logins in the HP CloudSystem Administrator Guide. 3. Establish the SSL certificate trust relationship as described in Manage security in the HP CloudSystem Administrator Guide.

8 4. Verify that the state for all services is smiley :-). If any service is not smiley, ssh onto the corresponding hostname listed under Host column and execute service <service_name> restart. ~]# nova-manage service list Binary Host Zone Status State Updated_At nova-compute kvmhost2.hpiscmgmt.local nova enabled :-) :45: healthnmon-virtproxy p01-esx-iscapp02-proxy-1.hpiscmgmt.local internal enabled :-) :45: healthnmon-collector p01-esx-iscapp02 internal enabled :-) :45: iscevents p01-esx-iscapp02 internal enabled :-) :45: iscrecovery p01-esx-iscapp02 internal enabled :-) :45: nova-conductor p01-esx-iscapp02 internal enabled :-) :45: nova-cert p01-esx-iscapp02 internal enabled :-) :45: nova-scheduler p01-esx-iscapp02 internal enabled :-) :45: healthnmon-virtproxy p01-esx-iscapp02 internal enabled :-) :45: nova-compute p01-esx-iscapp02-proxy-1.hpiscmgmt.local nova enabled :-) :46: nova-consoleauth p01-esx-iscapp02 internal enabled :-) :46: From the command prompt, execute the command service iscrecovery resync This starts the recovery service in Restore mode. The resynchronization with the managed environment starts. 6. Resynchronization actions are logged on the Activity screen in the CloudSystem Console as Alerts prefixed with Resync alert: 7. After the synchronization is complete, the Activity screen will show the following message: For any resource that is not connected, the recovery service will attempt the operation periodically for 24 hours. Once this message appears, the system is ready to use.

9 Recovery Actions: Synchronization among managed nodes and appliances During synchronization, an attempt is made to indicate alerts and state changes that occur after a restore for various instances, images and volumes. Note: The following operations do not apply to the Enterprise appliance. They are restricted to the Foundation appliance and managed nodes. Instances after restore operation Instance state Instance available at the time of backup Instance not available at the time of backup Instance state changed after Restore In-Progress Instance (Incomplete task associated with an instance) Instance exists on the host Is host active and connected? No change No Resychronization actions Instance marked as deleted in the CloudSystem database. Resync Alert message in the activity log indicating the instance is marked as deleted. Unknown No No change. Wait for pre-defined period (24 hours) for the compute node to come up. After the compute node is connected, change the state in the CloudSystem database to indicate the instance is deleted. Resync Alert message in the activity log indicating the instance is "orphaned". No No action required. No change. Wait for pre-defined period (24 hours) for the Unknown No compute node to come up. After the compute node is connected, change the state in the CloudSystem database to indicate the instance is "orphaned". Instance state in the database updated to match the state in the hypervisor. In the case of instance being stopped in the database, the OpenStack synchronization logic would forcefully stop the instance on the hypervisor. No change. Wait for specific period for the host to come No up. After the compute node is connected, the instance state in the database is updated to match the state in the hypervisor. Change the state in the CloudSystem database to" active" if instance is running, or "shutoff" if instance is shut down. Note: If the resynchronization of an instance is in progress, its associated state is not reflected in the CloudSystem user interface on the Instances screen. However, you can see the current state of the instance by logging in to the CloudSystem Portal. (See HP CloudSystem Administrator Guide.)

10 Images after restore operations Because the OpenStack Glance image disk is also backed up and restored, there should be no changes in the image state between backup and restore. All the changes (addition, deletion and modification) done to the images since the last back up are lost. The Glance image disk is returned to the backed-up state at the time of restore. Image state Image available at the time of backup Image exists after backup (on the original appliance) No (Image deleted) Actions after restore No change The image will be available Image not available at the time of backup Image size changed after backup "In-Progress" Image create or delete task "In-Progress" Image create or delete task (Image created) No No The image will not be available No action required Image is of the same size to that at the time of backup Image state in the CloudSystem database updated to mark available Image state in the CloudSystem database updated to mark deleted See also Appendix B: Backup of OpenStack Glance images. Volumes after restore operation Volume status Volume not created at the time of backup Volume "available" at the time of backup Volume exists on the array Volume attached to an instance Actions post-restore No Volume not monitored. No information is provided as there is no information about these volumes at the time of backup. No No change No NA Volume marked as deleted in the CloudSystem database. Resync Alert message in the Activity log indicating the image is marked as deleted. Volume state and instance state updated to match the current state Unknown Unknown No change. Volume 'In use' at the time of backup No change, No Alert message. Volume status in the database will change to "Available". No Resync alert indicates that the volume is detached from the instance. Unknown Unknown No change, No Alert message.

11 Volume not "available" at the time of backup No Volume will not be managed. The information is available in the iscrecovery log file. This file is obtained by generating audit logs from the CloudSystem user interface. For more information on how to generate audit logs, refer to the HP CloudSystem Administrator Guide. You will see the following message for every "orphaned" volume. The volume <Volume_Name> in storage system is no longer managed by CloudSystem 8.1 Specific updates to volumes - size, display name in the comment field In-Progress volume create or delete task Not applicable The CloudSystem database will be updated with the latest information from the storage system. Volume state in the CloudSystem database is updated to mark available. Notes: 1. If the size of a volume that is attached to an instance changes after the backup, the corresponding changes to the size are not reflected in the restored volume. 2. Changes done to a VMFS based volume s name after backup won t get synced after a restore. 3. If a vcenter cannot be reached during recovery, volumes that were deleted after the backup, will get synced up only on the next (periodic) run of recovery service after the vcenter becomes available again. 4. Any changes to the size of a VMFS backed volume done directly from the vcenter won t get synced after a restore.

12 Appendix A: Minimizing backup time and space Backing up every management appliance virtual machine requires time and storage space. The following shortcuts can be employed to reduce storage consumption and downtime during backup. Back up the following management virtual machines every time you back up CloudSystem appliances. Foundation base appliance SDN appliance Enterprise appliance (if installed) Back up the following virtual machines at least once Network node appliances vcenter proxy appliance(s) a. When a new vcenter proxy appliance is created b. When an ESX cluster compute resource is activated or deactivated. Appendix B: Backup of OpenStack Glance images The Openstack Glance image disk is backed up during the hypervisor-based backup as described earlier. However, a Glance disk is expected to grow (in the order of terabytes) over time. For the base appliance, the OVA templates or.qcow2 files could eventually consume a huge quantity of backup space. To minimize the impact of image growth on storage space and time necessary to create a backup file, consider the following approaches, which can be applied when there are no changes in the glance images managed by the appliance since the last backup. For ESX Hypervisor 1. If the OVA template is of comparable size to that of the base disk, use the approach as described in Back up an ESX management cluster. 2. If the Glance disk file is relatively larger, choose to clone the virtual machine. a. From the vcenter Server user interface, right-click on the Foundation appliance VM and select Clone. b. After the clone is created, copy all the disk files except the glance.vmdk file. Ensure that the glance.vmdk file is copied only when the number of images in the Glance store has changed. c. Delete the clone. 3. During the restore operation: a. Copy all the files from the backup to be restored. Copy the latest glance.vmdk file to the same folder.

13 For KVM hypervisor b. Start the virtual machine instance. 1. During the first clone of the appliances, ensure that all the files are copied to the backup store as described in Back up a KVM management cluster. 2. During subsequent backups, copy the glance.qcow2 image only if there are changes to the Glance image store. 3. During restore, copy the latest set of virtual machine files from the backup store. Copy the glance.qcow2 image if the latest set of the virtual machines do not have the file copied already. 4. Start the virtual machine from the clone. Appendix C: Network synchronization The current design architecture for the network controller node requires deleting new entities with state unknown at the time of backup. Therefore, once the network node is restored, the virtual machine instances for which networks were added or updated after the restore will have their networks disconnected. Keeping a list of virtual machine instances created since the last backup was created makes it easier to clone those instances (using hypervisor utilities) for use in later deployment through CloudSystem. The allowed address pairs configured will not be resynchronized after the restore operation. Therefore if any instance gets assigned the IP of the stale entry in the allowed address pairs, that rule will get applied. Appendix D: Useful manual recovery actions Inconsistencies can occur as a result of a restore if the state of the managed resources at the time of backup and restore are different. In cases where the backup does not contain enough data about the current managed resources, a manual recovery should be performed. Alerts and audit messages provided through CloudSystem Console can be used to identify the recovery action that should be performed. Cloud Management Subnet If the cloud connectivity subnet range is added or modified after backup, it needs to be updated again. Ensure that the same range information and metadata is entered after the restore. Provider and Private networks Changes to network information after the last backup would be lost after the restore operations. Any new ports or networks created after the last backup need to be re-created. Instances

14 If any new instances (or orphaned instances) were created after the backup, they cannot be brought back into the CloudSystem Console. An alert will be displayed on the Activity page of the CloudSystem Console for each of these instances after restore. Such an instance can be re-provisioned by taking a snapshot of the instance from the hypervisor. The newly provisioned instance will have a different set of IP addresses from the original. The intent here is to preserve the data in the instance and to enable management of the instance through the CloudSystem appliance. Step 1: Take a snapshot of the VM and optionally remove the instance from the host Instances on KVM hosts 1. Log in to the host for the instance specified in the alert. 2. Find the disk path of the instance from the domain name obtained in the alert. The following shows a sample alert: virsh dumpxml <domain_name> The disk path can be found in the file attribute at /devices/disk/source of the output XML. By default, the disk path is in the following format: /var/lib/nova/instances/<domain_name>/disk Using the disk path specified above, create a snapshot of the image file as follows: qemu-img snapshot -c <snapshot_name> <disk_path> If the instance is not deployed from an ami format then, qemu-img convert -f qcow2 -O qcow2 -s <snapshot_name> <disk_path> <output file_path> or else, if the instance is deployed from an ami format, capture a snapshot as a raw image: qemu-img convert -f qcow2 -O raw -s <snapshot_name> <disk_path> <output file_path> Repeat the above steps, for all disk attachments of the instance. 3. To remove the instance from the host, use the domain name of the instance obtained from the alert and execute the following commands on the host: virsh destroy <domain_name> virsh undefine <domain_name>

15 Instances on ESX Clusters 1. Log in to the vcenter Server registered for the cluster specified in the alert. (The alert message would be similar to the alert for a KVM orphaned instance) 2. Navigate to the VM on the cluster. From the summary page, browse to the datastore for the instance. 3. Download all the vmdk files (disk attachments) for the specified instance. Step 2: Upload the snapshot/vmdk files obtained onto CloudSystem Portal as a Glance image. Step 3: Provision a new instance using the base image through CloudSystem Portal. For multiple disk attachments, provision new volumes using the respective image as the volume source. Attach the volumes onto the new instance provisioned. Learn more about HP CloudSystem: and Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.