Backing up and restoring HP Insight Software v3.10 Central Management System (Windows)

Transcription

1 Backing up and restoring HP Insight Software v3.10 Central Management System (Windows) Abstract... 2 Introduction... 2 Scope... 3 Preparing your environment for backup... 4 Saving your environment Preparing your environment for restore Restoring your environment Completing the recovery process References Appendix A HP Insight mxsync Utility man page Appendix B Sample mxsync.log Appendix C Insight Software Backup File and Directories Appendix D Sample Script Appendix E VM-based Activate and Deactivate Actions Appendix F Add and Remove Domain Groups Appendix G Troubleshooting For more information... 78

2 Abstract The purpose of this white paper is to assist IT professionals in performing backup and recovery of HP Insight Software v3.10 running on a Microsoft Windows -based Central Management Server (CMS). HP Insight software suites are designed to address the most difficult issues encountered by IT infrastructures, such as cost, time, energy, and change. With flexible software that unifies the management of hardware resources and scales to businesses of all sizes, HP Insight software helps you: Take control of your time by simplifying management of HP ProLiant and HP BladeSystem infrastructures Reduce risk by identifying and resolving issues faster Drive business growth through flexible infrastructure deployment and optimization Lower cost through more efficient use of staff and facilities Introduction If the HP Insight software Central Management Server (CMS) environment becomes lost or corrupted, it may be necessary to restore your environment from a backup copy. The purpose of this white paper is to assist IT professionals in defining backup and recovery procedures for HP Insight Software v3.10 running on a Microsoft Windows -based Central Management Server (CMS). HP Insight Software media provides an integrated installer to quickly and accurately deploy, configure, integrate, and update HP Insight software suites and individual management products. Note: The details described in this paper are specific to the Insight software components included on the HP Insight Software DVD v3.10. The Insight Software Matrix 1.01 update is required to support the HP Insight mxsync Utility. See the Scope section for additional details. The Preparing your environment for backup section in this white paper describes the steps to quiesce the HP Insight software environment prior to performing a backup. In the section Saving your environment the steps to perform a backup of the HP Insight software files, directories and databases that contain user changeable state and configuration data are described. This partial backup process can be performed between full system backups. The steps to restore the environment are described in the Preparing your environment for restore and Restoring your environment sections. Finally the Completing the recovery process section describes a set of best practices to resynchronization the environment after the data has been restored. HP recommends periodically testing your backup and restore procedures in a non-production environment. Note: This paper describes the backup and restoration of a same system for fault insurance purposes only. It does not apply to system migration or software upgrades. 2

3 Scope This version of the white paper describes the procedures to backup and restore the Insight Software components listed in Table 1. The Insight Software Matrix 1.01 Update to Insight Software DVD v3.10 is required to support the HP Insight mxsync Utility. To download the Insight Software Matrix 1.01 Update, search for HPMTRX at Table 1. Components covered by the procedures described in this document. Insight Software Component Version Comment HP Integrated Lights-Out 2 Advanced for BladeSystem HP Insight Software Integrated Installer 3.10 HP Systems Insight Manger (HP SIM) HP Insight Power Manager (IPM) 2.0 HP Rapid Deployment Pack (RDP) 3.83 A full system backup should be performed whenever changes are made to the RDP environment. HP Insight Software Advisor 3.10 HP Performance Management Pack (PMP) HP Virtual Machine Management Pack (VMM) This document covers saving and restoring the VMM configuration data stored on the CMS. Please refer to HP ProLiant Essentials Virtual Machine Management Pack for details on backing up and restoring Virtual Machines. HP Insight Server Migration software for ProLiant (SMP) 3.70 HP Virtual Connect Enterprise Manager (VCEM) System Management Homepage (SMH) 3.0 HP Version Control Repository Manager (VCRM) WMI Mapper 2.6 HP Insight Dynamics VSE suite for ProLiant HP Insight Orchestration (HP IO) HP Insight Recovery (HP IR) HP Systems Insight Manager July 2009 hotfix for Windows is required. This hot fix is included in the HP BladeSystem Matrix 1.01 update. 2 This version of the product is included in the HP BladeSystem Matrix 1.01 update. 3

4 Preparing your environment for backup If the HP Insight software Central Management Server (CMS) environment has been lost or corrupted, it may be necessary to restore your environment from a backup copy. HP recommends periodically performing a full system backup. The section Saving your environment describes the steps to perform a backup of only the HP Insight software files, directories and databases that contain user changeable state and configuration data. This partial backup process can be performed between full system backups. To successfully restore, using a partial backup, the CMS must be running the same version of the software that was running when the full system backup was performed. A full system backup should be performed whenever the software on the CMS is updated. 1. Ensure all in-progress Insight Recovery jobs have completed. You can view the status of Insight Recovery jobs by selecting Insight Recovery from the Tools menu in HP SIM. Select the Jobs tab to view the status of jobs. A job with the in progress icon in the Status column has not yet finished. Figure 1. Wait for all Insight Recovery jobs to complete. Failover Job: Running 4

5 2. Ensure no HP Insight Orchestration requests are in-progress. Requests pending approval and scheduled requests may remain scheduled. You can view the status of HP IO requests by selecting Insight Orchestration from the Tools menu in HP SIM and then selecting the Requests tab. Click on the End Date column name to sort the request list. Requests with an empty End Date have not yet finished. Figure 2. Wait for HP Insight Orchestration in-progress requests to complete. Scheduled requests or requests pending approval do not need to complete. 5

6 3. Ensure all in-progress logical server jobs have completed before stopping the HP Logical Server Automation service. Logical server jobs that are in-progress when the service stops will fail. These jobs will need to be resubmitted after restarting the service. You can view the status of logical server jobs by selecting VSE Management from the Tools menu in HP SIM. Use the Reports menu to select the Logical Server Job Status item. Click on the % Complete column name to sort the list of jobs. Figure 3. Wait for all logical server jobs to complete. 6

7 4. Ensure all in-progress Virtual Connect jobs have completed. You can view the status of Virtual Connect jobs by selecting Integrated Consoles -> Virtual Connect Enterprise Manager (VCEM) from the Tools menu in HP SIM. Select the Jobs tab to view the status of jobs. Click on the End Time column name to sort the job list. Jobs displaying an empty End Time have not yet finished. Figure 4. Wait for all in-progress Virtual Connect jobs to complete. 7

8 5. Ensure all in-progress Virtual Machine tasks have completed. You can view the status of Virtual Machine tasks by selecting View Task Results from the Tasks & Logs menu in HP SIM. Click on the End Time column name to sort the task list. Tasks displaying an empty End Time have not yet finished. Figure 5. Wait for all in-progress Virtual Machine tasks to complete. 8

9 6. Ensure all in-progress installation and configuration tasks have completed (e.g., Configure and Repair Agents tasks). You can view the status of the tasks by selecting View Task Results from the Tasks & Logs menu in HP SIM. Click on the End Time column name to sort the task list. Tasks displaying an empty End Time have not yet finished. Figure 6. Wait for all in-progress installation and configuration task to complete. 9

10 7. If you are using the CMS to manage gwlm on Integrity nodes, you can issue the gwlm history --flush command to gather the latest historical data from the gwlm agents before performing a backup. This step may be skipped if you are not using the CMS to manage gwlm on Integrity nodes. Figure 7. Use gwlm history command to gather latest historical data. 8. The capcollect command enables the user to collect historical utilization data from systems licensed for Capacity Advisor for analysis on the Central Management Server (CMS). Data collection via Capacity Advisor can be controlled to occur at different times by doing one or more of the following actions: use the automated nightly collection, schedule a collection to occur periodically or one time only or launch an immediate collection. To avoid interrupting data collection, HP recommends scheduling the Insight software backup so that it does not overlap with the Capacity Advisor data collection. 10

11 9. Although not required, HP recommends waiting for all actively running migrations to complete before stopping the HP Server Migration Pack services. Migrations that are actively running when the services stop will fail. These jobs will need to be resubmitted after restarting the services. You can view the status of the currently running migrations by selecting View Status/Logs tab in HP Server Migration Pack Universal Edition. Figure 8. HP Server Migration Pack - Running Migration Status Screen 10. Before starting a full system backup, HP recommends waiting for all HP Rapid Deployment Pack (RDP) jobs to complete. This step is not required when performing a partial backup. You can use the HP ProLiant Essential Rapid Deployment Pack console, Start -> All Programs ->Altiris -> Deployment Solution -> Console, to view the status of RDP jobs. 11

12 11. Stop the Insight Software services describe in Table 2 in the order defined by the Stop Order column. A sample script to stop the services is provided in Appendix D. Note: If, in your environment, you have implemented a policy to automatically restart stopped services, this policy will need to be temporarily disabled to avoid restarting the services during the backup. Table 2. Insight Software services to stop before backup. Stop Order HP Insight Software Component Service Display Name 1 HP Insight Orchestration (HP IO) RSCentral 2 HP Insight Orchestration (HP IO) RSJRAS 3 HP Insight Orchestration (HP IO) RSScheduler 4 HP Insight Orchestration (HP IO) HP Insight Orchestration 5 The HP Insight Dynamics VSE suite for Proliant HP Logical Server Automation 6 The HP Insight Dynamics VSE suite for Proliant HP Application Discovery 7 The HP Insight Dynamics VSE suite for Proliant HP Global Workload Manager Central Management Server 8 The HP Insight Dynamics VSE suite for Proliant HP Agentless Data Collector Service 9 HP Virtual Machine Management Pack (VMM) HP Virtual Machine Management Service 10 HP Systems Insight Manager (HP SIM) HP Systems Insight Manager 11 HP Insight Server Migration Software for ProLiant HP Insight Server Migration software Application Service 12 HP Insight Server Migration Software for ProLiant HP Insight Server Migration software Web Service 13 HP Version Control Repository Manger (VCRM) HP Version Control Repository Manager 14 System Management Homepage (SMH) HP System Management Homepage The following Insight Software services may remain running during backup. HP Insight Software Component Service Display Name Comment OpenSSH Services for HP Systems Insight Manager WMI Mapper for HP Systems Insight Manager WMI Mapper for HP Systems Insight Manager OpenSSH Server Pegasus WMI Mapper Pegasus WMI Mapper Indications Service Indications sent while the WMI Mapper services are not running may be lost. Indications sent while the WMI Mapper services are not running may be lost. 12

13 Saving your environment If the HP Insight software Central Management Server (CMS) environment has been lost or corrupted, it will be necessary to restore your environment from a backup copy. HP recommends periodically performing a full system backup. A partial backup of the HP Insight software files, directories and databases that contain user changeable state and configuration data can be performed between full system backups. Important: To successfully restore the HP Insight software files, the CMS must be running the same version of the software that was running when the full system backup was performed. A full system backup should be performed whenever the software on the CMS is updated. The first part of this section describes the steps to perform a full system backup of the CMS. Table 3 contains a high-level description of these steps. Table 3. Overview of the steps to perform a full system backup. Step mxsync-svcutil stop backup Action 1 mxsync-svcutil manual 2 Using the tool of your choice, perform a full system backup of the CMS. 3 mxsync-svcutil auto mxsync-svcutil start 1. By default, the Windows services startup type of the Insight software services is set to Automatic. When performing a full system backup, HP recommends temporarily changing the startup type to Manual so that the startup type in the backup copy is Manual. In the event a restore is necessary, this will prevent the services from starting before the recovery process is complete. Table 4 provides a list of the Insight software service names. At a minimum, the state of the HP Insight Orchestration service must be changed to Manual. A sample script to change the startup type of the services to Manual is provided in Appendix D. 2. Using the tool of your choice, perform a full system backup of the CMS. 3. After the backup is complete, the startup type of the Insight software Windows services should be reset to Automatic for normal operations. A sample script to change the startup type to Automatic is provided in Appendix D. 13

14 Table 4. List of HP Insight software service names. HP Insight Software Component Service Display Name Service Name HP Insight Orchestration (HP IO) RSCentral RSCentral HP Insight Orchestration (HP IO) RSJRAS RSJRAS HP Insight Orchestration (HP IO) RSScheduler RSScheduler HP Insight Orchestration (HP IO) HP Insight Orchestration hpio The HP Insight Dynamics VSE suite for Proliant The HP Insight Dynamics VSE suite for Proliant The HP Insight Dynamics VSE suite for Proliant The HP Insight Dynamics VSE suite for Proliant HP Virtual Machine Management Pack (VMM) HP Logical Server Automation HP Application Discovery HP Global Workload Manager Central Management Server HP Agentless Data Collector Service HP Virtual Machine Management Service HP Logical Server Automation AppDisc HP Global Workload Manager CMS hp_agentless_service hpvmmsvc HP Systems Insight Manager (HP SIM) HP Systems Insight Manager HP Systems Insight Manager HP Insight Server Migration Software for ProLiant HP Insight Server Migration Software for ProLiant HP Version Control Repository Manger (VCRM) System Management Homepage (SMH) OpenSSH Services for HP Systems Insight Manager WMI Mapper for HP Systems Insight Manager WMI Mapper for HP Systems Insight Manager HP Insight Server Migration software Application Service HP Insight Server Migration software Web Service HP Version Control Repository Manager HP System Management Homepage OpenSSH Server Pegasus WMI Mapper Indications Service Pegasus WMI Mapper hpsmpsvc hpwebsmpsvc cpqsrhmo SysMgmtHp OpenSSHd WbemConsumer WMI Mapper 14

15 The remainder of the section describes the steps to perform a partial backup of the HP Insight software files, directories and databases that contain user changeable state and configuration data. This partial backup process can be performed between full system backups. Table 5 contains a highlevel description of these steps. Table 5. Overview of the steps to perform a partial backup. Step mxsync-svcutil stop backup Action 1 Using the file management tool of your choice, save the directories and files listed in Table 7. 2 Using the tool of your choice, perform a full backup of the databases listed in Table 6. 3 mxsync-svcutil start 4 Schedule a Windows task to save the HP Insight Orchestration network definition files between backups. This step is optional but recommended in environments where HP IO services are frequently created and deleted. To save the HP Insight Software data files and databases, proceed as follows: 1. Using the file management tool of your choice, save the directories and files listed in Table 7. For each occurrence of a bracketed variable (e.g., <SIM> ), substitute the actual installation path (e.g., C:\Program Files\HP\Systems Insight Manager). Table 6 describes the default installation directories for the HP Insight Software DVD v3.10. Although recommended, it is not necessary to save the log directories and files marked below as Optional. Note: You must use a file management tool that preserves file and directory permissions (such as the Windows Backup tool). 15

16 Appendix C contains a fully qualified path list of the directories and files described in Table 7. This list was created using the default installation directory settings described in Table 6. Table 6. Default Insight Software Installation Directories Symbol Component Default Installation Directory <ALC> HP IO <SystemDisk>Program Files\HP\ALC <HP> <SystemDisk>Program Files\HP <HPIO> HP IO <SystemDisk>Program Files\HP\Insight Orchestration <HPIR> HP IR <SystemDisk>Program Files\HP\Insight Recovery <HPOO> HP IO <SystemDisk>Program Files\HP\Operations Orchestration <OpenSSH> HP SIM <SystemDisk>Program Files\HP\OpenSSH <SIM> HP SIM <SystemDisk>Program Files\HP\Systems Insight Manager <SMH> SMH <SystemDisk>hp\hpsmh <SMP> SMP <SystemDisk>Program Files\HP\Server Migration Pack - Universal Edition <SystemDisk>Program Files (x86)\hp\server Migration Pack - Universal Edition <SystemDisk> C:\ <VCEM> VCEM C:\Program Files\HP\Virtual Connect Enterprise Manager <VCRMRep> VCRM <SystemDisk>Repository <VMM> VMM <SystemDisk>Program Files\HP\Virtual Machine Management <VSE> ID-VSE <SystemDisk>Program Files\HP\Virtual Server Environment <VSEData> ID-VSE <SystemDisk>Documents and Settings\All Users\Application Data\HP\Virtual Server Environment <UserProfile> HP SIM <SystemDisk>Documents and Settings\{username} <WMIMapper> HP SIM <SystemDisk>Program Files\The Open Group\WMI Mapper 16

17 Table 7. Insight Software Data Directories and Files Directory or File Optional Description <ALC>\Folders\ No HP Insight Orchestration configuration files <HP>\VMMTemplateLib\ No HP Virtual Machine Management Pack data files. Note: This directory is used to store the virtual machine backup and virtual machine template information created using Virtual Machine Management Pack. <HPIO>\conf\ No HP Insight Orchestration configuration files <HPIO>\logs\ Yes HP Insight Orchestration log files <HPIO>\data\ No HP Insight Orchestration network definition files <HPIR>\conf\ No HP Insight Recovery configuration files <HPIR>\logs\ Yes HP Insight Recovery log files <HPIR>\STORAGE\EVA\bin\clxevacfg No HP Insight Recovery credentials <HPIR>\STORAGE\EVA\conf No HP Insight Recovery configuration files <HPIR>\STORAGE\EVA\log Yes HP Insight Recovery log files <HPOO>\Central\conf\ No HP Operations Orchestration configuration files <HPOO>\Central\logs\ Yes HP Operations Orchestration log files <HPOO>\Central\rasrepo\ No HP Operations Orchestration data files <HPOO>\Central\rcrepo\ No HP Operations Orchestration data files <HPOO>\RAS\Java\Default\webapp\logs\ Logs where HP IO RAS java usage gets logged <HPOO>\Studio\conf\ No HP Operations Orchestration Studio configuration files <HPOO>\Studio\logs\ Yes HP Operations Orchestration Studio log files <HPOO>\Studio\rasrep\ No HP Operations Orchestration Studio data files <HPOO>\Studio\studiorep\ No HP Operations Orchestration Studio data files <OpenSSH>\etc\ No Save the OpenSSH keys. <SIM>\bin\CLIClientConfig.cfg No CLI configuration file <SIM>\bin\server_cert.pem No Certificate used by the Command Line Interface (CLI) <SIM>\config\ No HP Systems Insight Manager configuration files <SIM>\Install PSP\ No HP ProLiant Support Packs (PSP) data files <SIM>\log\Alc*.log Yes HP Insight Orchestration log files <SIM>\logs\mx.log Yes HP Systems Insight Manager audit log file <SIM>\mibs\ No Save any existing user-supplied or modified Management Information Bases (MIBs). <SIM>\output\ Contains job result output <SIM>\patch\ No Program updates (required if you want to reinstall HP SIM), this may be empty 17

18 Directory or File Optional Description <SIM>\tools\ No Save any existing user-supplied or modified tool definition files (TDEFs). <SMH>\certs\ No Directory containing the imported HP SIM certificates. <SMH>\conf\ No System Management Homepage configuration files <SMH>\data\cgibin\vcrepository\cpqsrhmo.log <SMH>\data\cgibin\vcrepository\cpqsrhmo.settings Yes No HP Version Control Repository Manager log file. HP Version Control Repository Manager configuration file. Note: The cpqsrhmo.settings file is encrypted. If a full system backup is not available and it becomes necessary to reinstall VCRM, the archived copy of this file will not be usable. The Administrator will need to manually reconfigure the VCRM settings using the configure the repository and automatic update settings in VCRM. <SMH>\data\cgi-bin\vcrepository\vcainf.xml No HP Version Control Repository Manager configuration file. <SMH>\data\webapp-data\ No System Management Homepage webapp related data. <SMH>\logs\ Yes System Management Homepage log files. <SMH>\session\ No System Management Homepage session directory. Subdirectories are used to store temporary session data. <SMH>\tomcat\ No System Management Homepage directory containing tomcat configuration files, log files, executables, other files of java based plug-ins and also working files produced by tomcat. Note: This directory may not exist. <SMH>\webapp\ No System Management Homepage webapp integration xml files. <SMP>\bin\hpvmm.conf No HP Server Migration Pack Universal Edition configuration file <SMP>\db\ No HP Server Migration Pack Universal Edition online hibernate database file <SMP>\log\ Yes HP Server Migration Pack Universal Edition log files <SystemDisk>\hp\sslshare No Shared HP OpenSSL certificate directory <UserProfile>\.ssh\ No OpenSSL user certificate directory <VCEM>\cfg\mvcd-config.xml No HP Virtual Connect Enterprise Manager configuration file <VCRMRep> No HP Version Control Repository Manager repository directory. Note: You can use VCRM to determine the location of the Repository. Select the configure the repository and automatic update settings in VCRM. The location will be display in the Directory field in the Repository Directory dialog. Please refer to the "HP Version Control Repository 18

19 Directory or File Optional Description Manager" document for additional information. <VMM>\bin\hpglobal.conf No HP Virtual Machine Management Pack configuration file <VMM>\bin\hpvmm.conf No HP Virtual Machine Management Pack configuration file <VMM>\bin\hpvmmdb.conf No HP Virtual Machine Management Pack configuration file <VMM>\bin\windows\hpvmmsvc.conf No HP Virtual Machine Management Pack configuration file <VMM>\bin\windows\hpvmmweb.conf No HP Virtual Machine Management Pack configuration file <VMM>\log\ Yes HP Virtual Machine Management Pack log files <VSE>\conf\ No HP Insight Dynamics VSE configuration files <VSE>\config\ No HP Insight Dynamics VSE configuration files <VSE>\data\ No HP Insight Dynamics VSE data files <VSE>\lbin\capadcollector.nodes No HP Insight Dynamics VSE configuration files HP Capacity Advisor node list configuration file <VSE>\lbin\capadcollector.properties No HP Insight Dynamics VSE configuration files HP Capacity Advisor node list property file <VSE>\logs\ Yes HP Insight Dynamics VSE log files <VSE>\lsadb\ No HP Insight Dynamics VSE logical server data files <VSE>\userprefs No HP Insight Dynamics VSE configuration files <VSEData>\ No HP Insight Dynamics VSE data directory. Note: The <VSEDataLocation> entry refers to the value of the HKEY_LOCAL_MACHINE\SOFTWARE\Hewl ett-packard\virtual Server Environment\DataLocation registry entry. <WMIMapper>\cimserver_planned.conf No OpenPegasus WMI Mapper configuration file <WMIMapper>\client.pem No OpenPegasus WMI Mapper client certificate <WMIMapper>\file.pem No OpenPegasus WMI Mapper server certificate key <WMIMapper>\logs\ Yes OpenPegasus WMI Mapper log files <WMIMapper>\server.pem No OpenPegasus WMI Mapper server certificate <WMIMapper>\ssl.cfg No OpenPegasus WMI Mapper configuration file 19

20 2. Using the tool of your choice, perform a full backup of the databases listed in Table 8. The database backup must be sufficient to restore the entire database, including schema as well as data. Table 8. - Insight Software Databases Component HP Systems Insight Manager (HP SIM) HP Performance Management Pack (PMP) HP Virtual Machine Management Pack (VMM) The HP Insight Dynamics VSE suite for Proliant HP Insight Orchestration (HP IO) HP Insight Orchestration (HP IO) HP Insight Recovery (HP IR) Database Insight_v* PMP_V* hpvmmsqldb gwlm hpio Oo lsdt Security Note: The archived files and databases contain confidential information including credential data. Access to this data should be careful controlled. 3. Start the Insight Software services described in Table 2 in the reverse of the order described by the Stop Order column. A sample script to stop the services is provided in Appendix D. 4. HP recommends scheduling a Windows task to frequently save the HP Insight Orchestration network definition files between backups. These files contain the current status of the HP IO networks and are stored in the <HPIO>\data\ directory. There is one XML file, Subnet_<network name>.xml, for each network. During the restore process, it will be possible to use these files to reduce the effort required to identify the IP addresses that were allocated after backup. Note: To support automatic provisioning, HP Insight Orchestration maintains a pool of available IP addresses for each network defined in HP IO. After restore, IP addresses allocated after the last backup, but before the catastrophic failure occurred, will be incorrectly marked as available and need to be fixed up. Please refer to the mxsync HP Insight Orchestration Synchronization section in Appendix A for additional details. 20

21 Preparing your environment for restore Prior to performing a restore, the following steps may be used to quiesce the environment. 1. If your environment is still operational, you may want to follow the steps defined in the section Preparing your environment for backup to allow in-progress operations to complete. 2. If running, stop the Insight Software services describe in Table 9 in the order defined in the Stop Order column. A sample script to stop the services is provided in Appendix D. Note: If, in your environment, you have implemented a policy to automatically restart stopped services, this policy will need to be temporarily disabled to avoid restarting the services during the recovery process. Table 9. Insight Software services to stop prior to restore. Stop Order HP Insight Software Component Service Display Name 1 HP Insight Orchestration RSCentral 2 HP Insight Orchestration RSJRAS 3 HP Insight Orchestration RSScheduler 4 HP Insight Orchestration HP Insight Orchestration 5 HP Virtual Server Environment Management Software HP Logical Server Automation 6 HP Virtual Server Environment Management Software HP Application Discovery 7 HP Virtual Server Environment Management Software HP Global Workload Manager Central Management Server 8 HP Virtual Server Environment Management Software HP Agentless Data Collector Service 9 HP Virtual Machine Management Pack HP Virtual Machine Management Service 10 HP Systems Insight Manager HP Systems Insight Manager 11 HP Insight Server Migration Software for ProLiant HP Insight Server Migration software Application Service 12 HP Insight Server Migration Software for ProLiant HP Insight Server Migration software Web Service 13 HP Version Control Repository Manger (VCRM) HP Version Control Repository Manager 14 System Management Homepage (SMH) HP System Management Homepage 15 OpenSSH Services for HP Systems Insight Manager OpenSSH Server 16 WMI Mapper for HP Systems Insight Manager Pegasus WMI Mapper Indications Service 17 WMI Mapper for HP Systems Insight Manager Pegasus WMI Mapper 21

22 Restoring your environment The section describes the steps to restore your environment from backup. Table 10 contains a highlevel description of the restore and recovery steps. Table 10. Overview of restore and recovery steps. Step Action mxsync-svcutil stop restore Restoring your environment 1-4 Using the tools of your choice, restore the Insight software environment from backup. 1 mxsync-e Completing the recovery process Note: Execution mode (-e) should only be enabled if a backup copy of the Insight software databases and data files is available. 2 mxsync-svcutil auto 3 mxsync-svcutil start 4-8 Where necessary, manually re-issue the operations that were performed after backup. 1. If necessary, first restore the CMS environment from the full system backup. Warning: Before proceeding, please be sure that all Insight software services are stopped. Refer to Table 9 for a list of services to stop. A sample script to stop the services is provided in Appendix D. 2. Using the file management tool of your choice, restore the directories and files listed in Table 7. You must use a tool that preserves file and directory permissions (such as the Windows Backup tool). 3. Using the tool of your choice, restore the databases listed in Table 8. You must restore each database using the same database name that existed when the database was saved. 4. If you have saved a more recent version of the HP Insight Orchestration network files, <HPIO>\data\Subnet_<network name>.xml, overwrite the restored versions with the more up-to-date copies. 22

23 Completing the recovery process Changes to managed resources that occur after backup but before the failure occurred, referred to in this document as post-backup activity, can affect the recovery process. Examples of post-backup activity include activating or deactivating a logical server, placing an HP IO service on Standby, deleting an HP IO service or logical server, and adding or deleting an HP IR recovery group. To complete the recovery process, the Insight software solution stack components need to resynchronize. Note: The purpose of the mxsync tool is to place the solution stack in a consistent state. Most operations that occurred after backup will need to be manually re-entered. HP recommends backing up the environment frequently. 1. Run the mxsync command line utility and follow the displayed instructions. The mxsync command line utility is provided as an aid to guide the Administrator through the process of synchronizing the Insight software environment with the current state of managed resources. Appendix A contains a description of the HP mxsync Utility. Appendix B contains the output from a sample run of mxsync. Note: The installation directory for the mxsync utility is <HPSIM>\mxsync\bin directory. The fully qualified path for the mxsync command is <HPSIM>\mxsync\bin\mxsync. If, as a post-backup activity, a VC Domain was added to or removed from a VC Domain Group, the Add VC Domain/Remove VC Domain operation will need to be reissued to synchronize the state of the VC Domain with the restored VCEM database. To avoid the manual steps required to perform this resynchronization, HP strongly recommends backing up the environment whenever a VC Domain is added or removed from a VC Domain Group. Appendix F contains a description of the steps required to reissue these operations. 2. Verify that the startup type for the HP Insight software services is set to Automatic. Table 9 provides a list of the HP Insight software names. A sample script to change the startup type is provided in Appendix D. 3. After all mxsync synchronization phases have completed successfully, you may now start the services described in Table 9 in the reverse order defined by the Stop Order column. Depending on your configuration, several of these services may have been started during the previous step. A sample script to start the services is provided in Appendix D. 23

24 4. You may need to perform the following manual steps to resynchronize the VM-based logical servers that were modified after backup. a. If a logical server was activated, as a post-backup activity, the state of the logical server in the restored ID-VSE database (i.e., deactivated) will be out-of-sync with the state of the managed VM (i.e., activated). Until the states are synchronized, the logical server will not be fully manageable by ID-VSE. To synchronize the state, without changing the state of the underlying VM, you will need to first unmanage and then re-import the logical server. To unmanage the logical server, select VSE Management from the Tools menu in HP SIM. From the Logical Servers perspective select the logical server to be resynchronized. Select Unmanage Logical Server from the Delete menu. To import the VM, select VSE Management from the Tools menu in HP SIM. Select the VM to be imported and select Logical Servers->Import from the Tools menu to perform the import. b. If a logical server was deactivated, as a post-backup activity, the state of the logical server in the restored ID-VSE database (i.e., activated) will be out-of-sync with the state of the managed VM (i.e., deactivated). Until the states are synchronized, the logical server will not be fully manageable by ID-VSE. To synchronize the states, you will need to reissue the ID-VSE deactivate request. However, before performing this operation you will need to temporarily activate the VM using the underlying VM management infrastructure. For details, please refer to the procedures described in Appendix E. Note: Depending on the state of the HP SIM database, it may be necessary to re-identify the VM host at this point. Once the VM has been successfully activated, select VSE Management from the Tools menu in HP SIM. Select Logical Servers->Refresh from the Tools menu in VSE Management. Select Refresh of Virtual Machine Resources. Click Refresh. You can now use ID-VSE to deactivate the logical server by selecting VSE Management from the Tools menu in HP SIM. From the Logical Servers perspective select the logical server to be activated. Select Logical Servers -> Activate from the Tools menu. 5. You may need to perform the following manual steps to resynchronize logical servers that were modified after backup. a. If a logical server was deleted, as a post-backup activity, you may need to manually delete the logical server definition from the restored ID-VSE database. Select VSE Management from the Tools menu in HP SIM. From the Logical Servers perspective select the logical server to be deleted. Select Delete Logical Server from the Delete menu. If the underlying resource has already been deleted, the delete operation will fail and you will need to perform an Unmanage Logical Server. For VM-based logical servers, if the logical server was active at the time the backup was taken, you will need to perform an Unmanage Logical Server. b. If a logical server was created, as a post-backup activity, you may need to manually import the logical server to add the definition to the restore ID-VSE database. You can 24

25 import the logical server by selecting VSE Management from the Tools menu in HP SIM. Select Logical Servers->Import from the Tools menu in Virtualization Manager. c. If a logical server definition was modified after backup (e.g., Identity, Compute, Storage or Network), these changes will need to be manually re-entered after resynchronization has successfully completed. Note: If a VC-based logical server, created after backup, was deactivate when the failure occurred, the VC profile definition will be lost and the logical server will need to be recreated. 6. You may need to perform the following manual steps to resynchronize the HP IO services. Note: The HP Insight Orchestration Windows service will initiate a refresh on startup. Once running, a refresh will be run once an hour. Therefore, it may take up to an hour for HP IO to detect changes. If there are no HP Insight Orchestration jobs running, you can restart the HP Insight Orchestration service to accelerate detection of the changes. a. If an HP IO service was resumed, as a post-backup activity, the state of the HP IO service in the restored HP IO database (i.e., Standby) will be out-of-sync with the state of the associated logical servers (i.e., activated). To resynchronize the state in the HP IO database, the resume request will need to be reissued to update the state of the service in the restored HP IO database. Select Insight Orchestration from the Tools menu in HP SIM. From the Services tab select the service to be placed on Standby. Click Resume servers. After the resume operation has completed, the HP IO service will be manageable, however, under the Resource Details tab, the Resource ID may be incorrect. During the next HP IO refresh, the Resource ID will be updated to the correct value. b. If an HP IO service was placed on Standby, as a post-backup activity, the state of the HP IO service in the restored HP IO database (i.e., Active) will be out-of-sync with the state of the associated logical servers (i.e., deactivated). To resynchronize the state in the HP IO database, the Standby request will need to be reissued to update the state of the service in the restored HP IO database. Select Insight Orchestration from the Tools menu in HP SIM. From the Services tab select the service to be placed on Standby. Click Standby servers. c. If an HP IO service was deleted, as a post-backup activity, you will be able to delete the HP IO service from the restored HP IO database during the execution of mxsync (phase 2 in the HPIO section). However, you may need to manually delete the associated logical server definition by selecting VSE Management from the Tools menu in HP SIM. From the Logical Servers perspective select the logical server to be deleted. Select Delete Logical Server from the Delete menu. For VM-based logical servers, if the logical server was active at the time the backup was taken, you will need to perform an Unmanage Logical Server. 7. For HP Insight Recovery configurations, if you have restored and resynchronized the primary site CMS, you must ensure the recovery site is updated to reflect the changes made to the primary site configuration. Please refer to the HP Insight Recovery 1.0 Online Help for details on how to export and import HP IR configuration data. 25

26 Note: After resynchronization is complete, HP recommends performing a planned failover/failback to verify the HP IR configuration. 8. Most changes made after the backup was taken will need to be manually re-entered. Examples include, but are not limited to, adding or deleting a logical server or HP IO services, network configuration changes, storage configuration changes, adding or removing VC Domain or VC Domain Groups, license changes, monitored workload definition changes, and user preference changes. Note: HP recommends backing up the environment after making configuration changes to the Insight software environment. 26

27 References Refer to the following for more information regarding the topics discussed in this paper. 1. For HP Insight Software product documentation, click Technical Documentation under the Tools and Resources section at 2. For HP Insight Dynamics VSE product documentation, click Product Documentation under the Quick Links section at 3. For HP Insight Orchestration product documentation, click Support & Documents under the Support section at 4. For HP Insight Recovery product documentation, click Support & Documents under the Support section at 27

28 Appendix A HP Insight mxsync Utility man page The HP Insight mxsync Utility man page is included in this Appendix. Appendix B contains the output from a sample run of mxsync. A user running mxsync must have administrator privileges on both the CMS and database servers, have administrative privileges to the Insight Dynamics VSE database, and be a valid HP SIM user. NAME mxsync - eases the task of synchronizing an Insight Software CMS environment that has been restored from backup with the state of the managed resources. This utility can be used to identify potential inconsistencies, recommend corrective actions, and perform most recommended actions. SYNOPSIS mxsync [--help] [--version] [-e] [-c] DESCRIPTION If the HP Insight software Central Management Server (CMS) environment becomes lost or corrupted, it may be necessary to restore the environment from a backup copy. The mxsync utility guides the Administrator through the process of synchronizing an Insight software CMS environment that has been restored from backup with the state of the managed resources. The mxsync tool is divided into five sections that are executed sequentially in the following order: HP Systems Insight Manager (HP SIM), Virtual Connect Enterprise Manager (VCEM), HP Insight Dynamics Logical Server (ID-VSE), HP Insight Orchestration (HP IO) and HP Insight Recovery (HP IR). Each section is responsible for identifying inconsistencies between its representation of a resource and that of the underlying layer. A section consists of one or more phases. A phase defines an analysis and execution step for a specific synchronization activity. 28

29 Section Phase Synchronization Activity HP SIM 1 Discover any system changes that may have occurred after backup was performed. VCEM 1 Ensure WWN and MAC address assigned to a VCEM profile have not been reassigned to a different VC profile. Ensure WWN and MAC addresses allocated to VC profiles, after backup, have been correctly marked as External in the VCEM database. 2 Resynchronize each VC domain managed by VCEM. ID-VSE Ensure the ID- VSE logical server network, storage and bay assignment definitions are consistent with the underlying VCEM profile definitions. Ensure WWN and MAC addresses assigned to an ID-VSE logical server are still owned by ID-VSE and have not been reassigned to a different VCEM profile. Ensure WWN addresses assigned to ID-VSE storage pool entries (SPE) are still allocated to ID- VSE and associated with VCEM profiles that are managed by IS-VSE. 1 HP IO 2 3 HP IR 1 HP IR 2 Terminate HP Insight Orchestration requests that were in-progress when the backup was taken or requests that were scheduled to run after the backup but before the restore. This is necessary to prevent the HP Insight Orchestration Windows service from attempting to restart a request that may have already completed. Update the HP IO database to remove the definition of HP IO services that were deleted post-backup. Update the HP IO database to mark, as allocated, any IP address that was allocated after backup. Start the HP Logical Server Automation service since it is needed for HP IR resynchronization Use an up-to-date HP IR configuration export file to install any postbackup HP IR configuration changes. If any protected Logical Servers have been deleted, delete the corresponding IR recovery groups as necessary. The mxsync tool supports two modes of operation, an analysis-only mode and an execution mode. By default, mxsync runs in an analysis-only mode. When started in an analysis-only mode, all execution steps will be skipped and no updates will be made to the restored database or files. The analysis-only mode can be used to preview the recommended changes for the next synchronization phase. Warning: When run in analysis-only mode, an unresolved issue in one phase can cause subsequent analysis phases to report incomplete or incorrect results. Warning: When run in execution mode, all issues identified in a phase must be resolved before proceeding to the next phase. mxsync Initialization As a general rule, the Insight software Windows services should not be running during the execution of mxsync. However, the mxsync tool does depend on several HP Insight software Windows services to perform the analysis and execute steps. Prior to starting the synchronization activity, the mxsync tool will check the status of the Insight software services and recommend actions for stopping and starting services. 29

30 When run in execution mode, if you respond y to the prompt to make the recommended changes, the required services will be started and the non-required Insight software services will be stopped. When run in analysis-only mode, the mxsync tool does not make any changes to the running state of the Windows services. If the required services are not started, the mxsync tool will exit with an error. You will need to manually start the services to run mxsync. If non-required services are running, mxsync will allow the analysis to proceed. However, a warning will be displayed stating that in-progress operations may cause mxsync to report non-existent issues. If issues are identified, when the system is quiesced, the services should be stopped and the analysis rerun. Service required by mxsync Service Display Name Analysis-only Requirement by section Execution HP Logical Server Automation HP IR HP IR HP Virtual Machine Management Service HP SIM HP SIM HP Systems Insight Manager All All OpenSSH Server HP SIM HP SIM Note: If, in your environment, you have implemented a policy to automatically restart stopped services, this policy will need to be temporarily disabled to avoid restarting the services during the recovery process. Warning: Although selected services are started by the mxsync tool, executing operations that change the state of managed resources should be avoided until resynchronization is complete. mxsync HP SIM Synchronization Section To discover system changes that may have occurred after the backup was created, during the execution step, you will be asked to connect to HP SIM, login as Administrator and run HP SIM tasks to discover Onboard Administrators, hypervisor host operating systems, and operating systems deployed on blades, VM guests and stand-alone servers. The mxsync tool will pause while you complete discovery. Once discovery is complete, you can return to mxsync to continue the synchronization process. For more information about performing an HP SIM discovery, see the HP Systems Insight Manager 5.3 Technical Reference Guide. If, as a post-backup activity, managed resources were added to the discovery tasks, or credential or licenses were added or modified, these changes should be made before running discovery. mxsync HP Virtual Connect Enterprise Manager Synchronization Section During this section the restored VCEM database will be resynchronized with the actual state of the VC Domains managed by VCEM. There are two phases in this section. During the first phase, the mxsync tool will analyze the WWN and MAC addresses assigned to profiles in the VC domains managed by VCEM to identify addresses that may have been allocated or reassigned after backup. 30

31 An address could have been allocated if, for example, as a post-backup activity, a VC-based logical server or HP IO service was created. To prevent a post-backup allocated address from incorrectly being reassigned to a second server, during the execution step of this phase, the mxsync tool will mark these addresses as External in the VCEM database. An address could have been reassigned if a VC-based logical server was deleted, the address freed and then reassigned to a new logical server. To eliminate the conflict caused by reassigning an address to a different VC profile, during the execution step of this phase, the mxsync tool will delete the definition of the VCEM profile that contains the reassigned address. If, as a post-backup activity, a VC Domain was added to or removed from a VC Domain Group, the VC Domain will no longer be accessible to VCEM after the VCEM database has been restored from backup. During phase 1, you will be given the opportunity to provide the credentials for the added or removed VC Domain. This will allow mxsync to access the VC Domain to identify any WWN or MAC addresses that were assigned to the VC Domain after backup. If no VC Domains were added or removed after backup, you can respond n to the prompt to provide credentials. During the second phase, the mxsync tool will issue requests to VCEM to resynchronize each VC domain managed by VCEM. Note: During the analysis step of phase 2, mxsync will display an ISSUE for each VC Domain managed by VCEM stating that the VC Domain MAY need to be synchronized with VCEM. This is only a potential issue. The test to check if the VC Domain is out of sync will temporarily place the VC Domain in maintenance mode and therefore can only be performed during the execution step. If, as a post-backup activity, a VC Domain was added to or removed from a VC Domain Group, you will need to manually re-issue the Add to VC Domain or Remove From VC Domain operation to successfully complete phase 2. mxsync HP Insight Dynamics Logical Server Synchronization Section During this section, the restored HP Insight Dynamics database will be updated to reflect the changes made to the VCEM database during the previous section. There are three phases in this section. During the first phase, the mxsync tool will analyze each logical server definition to ensure the underlying VCEM profile exists and the network, storage and bay assignment configuration data stored in the ID-VSE database are consistent. The restored logical server definition can be out-of-date if, for example, as a post-backup activity, the logical server was deactivated. Note: If a logical server definition was modified after backup, these changes will need to be manually re-entered after resynchronization has successfully completed. During the second phase, the mxsync tool will analyze each logical server definition to ensure the assigned WWN and MAC addresses are still allocated to Insight Dynamics - VSE (ID-VSE) and have not been reassigned to another VCEM profile. The latter can happen if, for example, as a post-backup activity, the logical server was deleted, the addresses freed and then reassigned to a new server. To prevent two servers from using the same address, during the execution step of this phase, the mxsync tool will delete reassigned MAC addresses from the logical server definition and, in the case of duplicate WWN address, delete the logical server definition. 31