HPOM 8.1 High Availability Utilizing Microsoft SQL Server Log Shipping

Transcription

1 HPOM 8.1 High Availability Utilizing Microsoft SQL Server Log Shipping White Paper version 2.4 HPOM 8.1 High Availability Utilizing Microsoft SQL Server Log Shipping... 1 Change record... 3 Architectural Overview... 4 Conceptual Overview... 5 Supported HPOM for Windows versions... 5 Advantages and Disadvantages of SQL Server Log Shipping Replication... 5 Package Contents... 5 SHIP_OVDB Script... 5 SQL Server Synchronization Policies... 6 Re-Synch Database Tool... 6 Failoverstart Script... 7 Disable HPOM Services Tool... 7 Enable HPOM Services Tool... 7 Prerequisites... 8 Installation Process... 8 Installation Checklist... 8 Install the Primary and Backup HPOM Management Servers... 9 Install the HPOM Database Replication Package onto Both Servers... 9 Configure Trusted Certificates for Multiple Management Servers Configure the MultipleActionManagers Policy, and Distribute it to Managed Nodes Manage the Backup server from the Primary Server Deploy the VP_SM Instrumentation to the Primary and Backup Management Servers Stop HPOM for Windows on the Backup Server Set cscript as the Default Script Host Enable Remote DB Access for SQL Server 2005/SQL Express on the Backup Server Configure the SQL Server Synchronization Tool Initialize the Replication to the Backup Server Configure the SQL Server Synchronization Policies Install the Scheduled Task Policies to Start Automated Log Shipping Replicate the Instrumentation Folder after Modifications Uninstallation Process... 21

2 HP Operations Manager Fail-over to Backup Procedure Using the Switch Primary Manager Tool What to Do After Fixing the Old Primary Server Continue Using the Backup Server as the Primary Server Establish the new Primary server as a permanent Primary Configure the old Primary HPOM server as a standby backup server Initialize the new Backup servers database and start replication Fail back to the old Primary once it has been fixed Appendix A: Upgrade and Patch Installation Appendix B: Error Codes from VBScripts ship_ovdb.vbs and failoverstart.vbs Error Error Error Error Error Error Error Error Error Error Error Error Error Error Error Appendix C: Customized Configurations For more information

3 Change record Version Date Changes 1.0 July 2003 First version for Operations for Windows July 2003 Added comments about OVO backup job see changes marked with 1.2 August 2003 Added note about instrumentation replication 1.3 October 2003 Removed Preliminary 1.4 July 2004 Added support for Reporter database replication. Replaced Switch management server tool by opcragt. 1.5 February 2005 Update location of Instrumentation folder. Changed 7.20 to 7.x in header/footer. 1.6 August 2007 Updated for HPOM October 2007 Added uninstallation procedure 2.0 July 2008 Adapt to HPOM 8.1 (including HTTPS) 2.1 October 2008 Added information that two additional policies should be disabled on the backup server 2.2 November 2008 Added some more information about stopping the management server on the backup system 2.3 January 2009 Add some further improvements (additional registry key for OvStoreProv, additional tools,..) 2.4 April 2009 Extend the ship_ovdb.vbs by the optional parameter <DB path location> 3

4 Architectural Overview HPOM 8.0 Primary Manager HPOM 8.0 Backup Manager SQL Server DB SQL Server DB Backup Copy Restore Transaction Log Transaction Log 4

5 Conceptual Overview The warm standby backup management server maintains a periodically updated copy of the SQL Server database of the active management server. The standby database on the backup management server is in Recovery (Read- Only) mode. A scheduled task policy on the primary HPOM server periodically backs up the transaction log of the active database, copies it to the standby server and restores it to the standby database. As a result, the two databases are kept in sync. In case of a failure of the primary management server, the database on the backup server is switched from Recovery to Online mode and the HPOM services are started on the standby management server. Next, the Switch Management Server Tool is executed on every managed node. This switches the agents to report only to the standby management server. Then the administrator merely has to restart the management server console in order to connect to the backup management server. Supported HPOM for Windows versions The SQL Server log shipping replication is currently supported for OVO for Windows 7.2 and higher. Note This document refers to HPOM 8.x only. Use the corresponding Whitepapers for older OVO versions. Advantages and Disadvantages of SQL Server Log Shipping Replication Advantages: 1. It is a cheaper and simpler solution than a full-blown HA-Cluster such as Microsoft Cluster Server. 2. Log shipping does not require much processor overhead on the active management server. This is because performing a transaction log backup on the active management server requires only a fraction of the processing power needed to restore the transaction log backup on the standby management server. Disadvantages: 1. There is no HA-software to perform a sanity check on the primary management server. Therefore, it requires human interaction to start the backup management server, and to execute the provided tool to reconfigure the agents to report to the backup management server. 2. Some data may be lost. How much data is lost depends on how often the transaction log is backed up and shipped to the standby management server. The default interval for database replication is 15 minutes. Package Contents This section describes the package contents that are installed. The package includes an installation tool (MS Installer based) that simplifies the initial setup of the resilient HPOM server configuration. Note that you must perform certain configuration steps manually, as described later in this document. SHIP_OVDB Script The Ship_OVDB.vbs script provides the main engine for: backing up the primary database (using SQL-DMO objects provided with SQL Server) 5

6 shipping the database to the backup management server (using the FileSystemObject) restoring the database on the backup management server (using SQL-DMO objects provided with SQL Server) The original Ship_OVDB.vbs script should NOT be edited, but the script may be copied and then modified if required (this is an advanced option that is neither recommended nor supported). The SHIP_OVDB script utilizes the credentials of the executing user to start the backup, copy the backup, and restore the backup. Several prerequisites must be met for the script to work properly: 1) The user running the script must be a System Administrator (sa) of both SQL servers, as well as a local (windows) administrator of both systems. By default, members of the local administrators group are members of the SQL Server System Administrator role. 2) Administrative shares must be enabled on both management servers (C$, D$, E$, ADMIN$, etc), as they are used for the file copy operations between the management servers. 3) Remote SQL Server access must be enabled on the backup management server. This script is called by the Re-Synch Database tool and the two SQL Server synchronization policies. The account under which this tool and the policies run needs to meet the above conditions. The usage of this script is: ship_ovdb.vbs <action> <backupsvr> [<DB path location>] where: <action> is DB or LOG Use DB to backup the full primary database and restore it to the backup database server Use LOG to backup the primary database transaction log and restore it to the backup database server <backupsvr> is the name of the HPOM backup management server <DB path location> (optional) is the physical location of DB files on the Microsoft SQL Server system used by the backup management server. Use this parameter only if the automatic mechanism to get this pathname fails. SQL Server Synchronization Policies The policy group HPOM-HA is created on the two management servers by the installer utility. This group contains the two scheduled task policies: 1) Backup-Ship-Restore HPOM Database This policy is used to backup the local HPOM database, ship it to the backup management server, and restore the log in warm standby on the backup management server. By default, this policy is configured to run every Sunday at 23 p.m. 2) Backup-Ship-Restore HPOM Logs (Scheduled Task) This policy is used to backup local trans log differences, ship them to the backup management server, and restore the logs in warm standby on the backup management server. By default, this policy is configured to run every 15 minutes. For more details, see Configure the SQL Server Synchronization Policies. Re-Synch Database Tool The replication solution implements a tool called Re-Synch HPOM Database. You can find this tool in a new Tools folder named Database Replication in the Tools section of the management console. The Re-Synch HPOM Database tool calls the SHIP_OVDB script, and is responsible for performing a complete database backup on the 6

7 primary management server, which is then copied and restored on the backup mangagement server. This tool is used in the following two situations: 1) When you first start replication between the primary management server and the backup management server. The Re-Synch Database tool performs the initial step to initialize the backup database with a copy of the primary database. 2) When replication breaks. This may happen when network outages occur and log files are not restored in the order in which they were generated. You can simply and easily re-establish the backup server state by executing the Re-Synch Database tool, which will move a full database backup to the backup management server, and restore it to the state of the primary management server Failoverstart Script When the primary management server fails, the administrator starts the failoverstart.vbs script to bring the backup management server online. The script performs the following actions: - Sets the registry key value HKLM\SOFTWARE\Hewlett-Packard\OVEnterprise\Management Server\OvStoreProv\Disabled to 0 thus the OvStoreProv.exe can be started again - Recovers the database, applying all transaction log changes supplied through the log shipping process - Sets the database to multi-user, read-write access - Starts critical HPOM processes in preparation for consoles to connect Disable HPOM Services Tool The new version of the replication solution implements a new tool called Disable HPOM Services. You can find this tool in the Tools folder named Database Replication in the Tools section of the management console. The Disable HPOM Services tool calls the disable_hpom_services.vbs script. Important: This tool should be started from the primary management system and executed on the backup management system. It prepares the system for the backup role. It - sets the startup type of the HPOM services on the backup server to Disabled - stops the HPOM services - sets the registry key value HKLM\SOFTWARE\Hewlett-Packard\OVEnterprise\Management Server\OvStoreProv\Disabled to 1 Important: This stops the OvStoreProv.exe if it is running and prevents this process of being restarted. These settings prevent that the HPOM services are restarted by another process. This avoids that the database of the backup management server is accessed through these processes. While this management server is in backup mode his database should be changed only as a backup of the primary management server. Note: This tool should only be launched on a remote management server system. If you would start it on a nonmanagement server system it will fail but it will have no effect on the system. Enable HPOM Services Tool The new version of the replication solution implements a new tool called Enable HPOM Services. You can find this tool in the Tools folder named Database Replication in the Tools section of the management console. The Enable HPOM Services tool calls the enable_hpom_services.vbs script. It does the reverse steps from the Disable HPOM Services tool. It: 7

8 - sets the registry key value HKLM\SOFTWARE\Hewlett-Packard\OVEnterprise\Management Server\OvStoreProv\Disabled to 0 This enables the OvStoreProv.exe-process to be restarted. - sets the startup type of the HPOM services on the backup server to Automatic - starts the HPOM services The actions carried out by this tool are usually executed by the failoverstart.vbs script. Note: If you would launch this tool on a non-management server system it will fail but it will have no effect on the system. Prerequisites 1) Two HPOM management servers, configured identically, and in a single Windows domain (NT4/Ads), are required. It is assumed that the two servers have unique hostnames and share the same DNS domain. The machines do NOT need to be co-located. However, if replication traffic is large, you must make sure that adequate network bandwidth is available for the normal log ships and policy copies to succeed within the log-shipping interval. The default interval is 15 minutes. This may need to be adjusted in high-latency situations. 2) If you are using a remote database server, you need to make sure that SQL-DMO is installed on your management server. For example, you can install the "Microsoft SQL Server 2005 Backward Compatibility Components" on these systems. 3) A single login account that is an administrator of both HPOM management servers used for scheduled task policies and tools. 4) Servers have the same patches applied. SQL Server / SQL Express at same service pack level. 5) Administrative shares must be enabled on the backup management server for the file transfers to occur from the primary to the backup server. 6) Remote SQL Server access must be enabled on the Backup Server. Otherwise the database restore on the backup management server cannot be initiated from the primary management server by the script ship_ovdb.vbs. Installation Process Installation Checklist To enable the replication process, perform the following tasks: - Install the primary and backup HPOM management servers. - Install the HPOM Database Replication package onto both servers. - Configure trusted certificates for multiple management servers. - Configure the MultipleActionManagers policy, and distribute it to managed nodes. - Manage the backup management server from the primary management server. - Deploy the VP_SM Instrumentation to the primary and backup management servers. - Stop the HPOM for Windows services on the backup management server. - Set cscript.exe as the default script host on both servers. 8

9 - Configure the SQL Server synchronization tool. - Enable remote DB access for SQL Server 2005/SQL Express on the backup server. - Initialize the replication to the backup management server. - Configure the SQL Server synchronization policies. - Install the Scheduled Task Policy to start automated log shipping. - Prepare the backup management server to be able to take over in case of a failure. These tasks are described in detail in the following sections. Work through each step carefully, following all of the procedures that are described. Install the Primary and Backup HPOM Management Servers For each HPOM host system: Install an HP Operations Manager 8.x management server (and the latest patches) on two separate computers. Install the same Smart Plug-Ins on both management servers. This is important, as some Smart Plug-Ins extend the WMI namespace and service prototypes. Each server should have a unique DNS name, and must be registered correctly in your corporate DNS. Additionally, both management servers should be in the same Windows domain to aid with SQL Server authentication. Both management servers should use the same HP-OVE-User account (or whatever account you configure when installing the management server). The assumption of this setup is that only one management server is actively running, and the second management server is just a warm standby. If the primary management server fails, HPOM for Windows can be started on the backup management server, and the agents on the managed nodes are subsequently reconfigured to report to the backup management server. Install the HPOM Database Replication Package onto Both Servers To install the HPOM Database Replication Package onto both primary and backup management servers, follow these steps: 1) On both HPOM management servers, unzip all of the contents of the %OvInstallDir%\misc\OvOW\HPOM_HA_with_SQL_Log_Shipping.zip file into a temporary folder. The resulting folder should contain the following files. 9

10 2) Run the Setup.exe installer utility on both of the HPOM management servers. Note that the HPOM server must be running for the installation to complete successfully, as Policies and Tools will be uploaded into the HPOM servers configuration. 10

11 The installer performs the following steps on each system: Copies the ship_ovdb.vbs, disables_hpom_services.vbs and enables_hpom_services.vbs scripts to the VP_SM Instrumentation folder. Uploads two Scheduled Task policies to a Policy Group called HPOM-HA. The policies are called Backup- Ship-Restore HPOM Database and Backup-Ship-Restore HPOM Logs. Uploads the Re-sync Database, Switch Management Server, Enable HPOM Services and Disable HPOM Services Tools to a new tool folder called Database Replication. Copies the script failoverstart.vbs to the folder %OvInstallDir%\support. Creates a shortcut on the desktop for the failoverstart.vbs script. The shortcut is called Recover Database and Start HPOM Processes. Configure Trusted Certificates for Multiple Management Servers In case you need to switch the management server from the primary to the backup server during a failover, you need to exchange the certificates between these two servers. - On every management server, export the trusted certificates to files using the following commands: ovcert -exporttrusted -file c:\temp\certificates.general.export ovcert -exporttrusted -ovrg server -file c:\temp\certificates.server.export (You can specify different filenames and locations. Note that the directory you specify must already exist.) - Copy these files to every other management server, and import each trusted certificate using the following commands: ovcert -importtrusted -file c:\temp\certificates.general.export ovcert -importtrusted -ovrg server -file c:\temp\certificates.server.export - For every management server, update the trusted certificates on existing nodes: In the console tree, click Tools > HP Operations Manager Tools > Certificate Management. 11

12 In the details pane, double-click Update trusted certificates. A dialog listing nodes and services appears. Select Nodes and click Launch... The Tool Status dialog appears and shows progress. For more information, see Configure trusted certificates for multiple management servers in the online help. Configure the MultipleActionManagers Policy, and Distribute it to Managed Nodes You need to run the Switch Manager tool (described later in this document) from the backup HPOM for Windows management server. Therefore, it is necessary to utilize the Flexible Management concepts of the HPOM agent to permit the HPOM for Windows backup management server to issue commands to the agent. This is configured using a Flexible Management policy to define the two HPOM for Windows management servers and to specify that both servers are pemitted to perform actions on the HPOM agent. A policy template ( MultipleActionManagers ) is provided to simplify this configuration. You can find this policy template either in the Samples > en > Agent-based Flexible Management policy folder or the Agent policies grouped by type > Flexible Management policy folder that was installed with the base HPOM management server. To configure the MultipleActionManagers policy, do the following: On the primary management server, locate the Agent policies grouped by type > Flexible Management policy group and edit the MultipleActionManagers policy. Modify the Flexible Management Syntax as described in the comments. Replace the hptest.bbn.hp.com strings with the primary management server name, and hpsystem.bbn.hp.com strings with the name of the backup management server. It is recommended to use fully qualified domain names here. You can leave the IP address at so that DNS performs the name resolution. After the names of the management servers, add the keyword ID followed by the respective management server's core ID. To get the management server s core ID, open a command prompt on the respective system, and then type the following command: ovconfget -ovrg server sec.core CORE_ID Copy the core ID that the command returns. 12

13 For example, if your backup management server is called hpom.example.com, and its core ID is 89aea662- b9e d-8a612e083f23 replace both instances of the following line: NODE IP "hpsystem.bbn.hp.com" with the following line: NODE IP "hpom.example.com" ID "89aea662-b9e d-8a612e083f23" Also change the Description fields to indicate the Primary and Backup servers as appropriate. Save the modified policy and deploy it to all HTTPS managed systems. Notes Deploy this policy to ALL MANAGED SYSTEMS of the primary server (not only the management server). If you still have DCE managed nodes you need to create a copy of that policy, remove the ID parts and deploy it to all DCE nodes. Also keep in mind that you have to deploy this policy to every new node that you add to the management server. If you forget to deploy this policy, the backup server will not be able to manage the node in case of a failure of the primary management server. Manage the Backup server from the Primary Server To manage the backup management server from the primary management server, do the following: 13

14 1. Configure the MultipleActionManager policy on the backup management server, in a similar way to that described above. 2. Deploy the policy to the backup management server and all its HTTPS managed systems. 3. Export nodes from the backup management server. From a command line on the backup server execute the following command: ovpmutil cfg nds dnl c:\temp\nodes.mof (you can specify a different name and location for the mof-file. 4. Copy the file containing the node information to the primary management server. Copy the file from the previous step to the primary management server. 5. Import nodes on the primary management server. From a command line on the primary management server, execute ovpmutil cfg nds upl <mof-file copied from the backup server> 6. Switch the management server for the backup server by launching the tool Change the HPOM primary manager : a. Open a management console on the primary server. b. In the console tree, click Tools > HP Operations Manager Tools > Agent Administration. c. In the details pane, double-click Change the HPOM primary manager. A dialog listing nodes and services appears. d. Select the backup server and click Launch. The Tool Status dialog appears and shows progress. 7. Synchronize inventory of the backup server. a. In the HPOM GUI of the primary management server select the backup management server in the node pane. b. From the context sensitive menu select All Tasks > Synchronize inventory > Policies and packages. 8. Switch certificate server for the backup management server by lauunching the tool Switch Agent's Certificate Server (under HPOM Operations Manager Tools > Certificate Management ). From this point forward, the backup server is now a Managed Node of the primary server. If something goes wrong with the backup management server, the primary server will be notified. Deploy the VP_SM Instrumentation to the Primary and Backup Management Servers The VBScripts that are used to perform the replication have been copied to the source Instrumentation folders of the primary HPOM management server. You need to deploy these scripts to the agents Instrumentation folders. To deploy the VP_SM instrumentation, do the following: 1. Right-click on the primary HPOM server node, and select All Tasks > Deploy Instrumentation 14

15 2. Select the VP_SM instrumentation group and select OK to deploy it. Check that the instrumentation deployed successfully by reviewing the Policy Management\Deployment Jobs snap-in. 3. Repeat this step for the backup HPOM management server. Stop HPOM for Windows on the Backup Server Now you must stop HPOM for Windows services on the backup management server before you proceed with the following instructions. 15

16 1. Disable all the server policies from the backup management server to make sure that the services will not be restarted by a policy. Execute this step from a HPOM GUI of the primary management server to make sure that the primary management server database stores the current state of these policies. These policies are: - VP_SM_OVOWServices - VP_SM-Server_SynchAgentServices - VP_SM-Server_EventLogEntries - OvSvcDiscServerLog - VP_SM-WMI-Restart - VP_SM_CHK_OVODB - VP_SM_DB_Backup - ServiceLog_Maint_Job Update_HierarchicalNodes Note: the policies VP_SM_CHK_OVODB amd VP_SM_DB_Backup are installed automatically on the backup management server only if you are using a local database. If you have added your own policies, that use the HPOM management server services, you must disable them, too. To avoid possible concurrency issues on the primary management server database, disable the VP_SM_DB_Backup-policy also on the primary management server, or make sure that it does not run concurrently to the high availability policies Backup-Ship-Restore HPOM Database and Backup-Ship-Restore HPOM Logs. Note: this policy is installed automatically on the primary management server only if you are using a local database. 2. Disable and stop the HPOM services of the backup management server From a HPOM GUI of the primary management server start the Disable HPOM Services tool on the backup management server. This following services will be stopped: - OvAutoDiscovery Server - OvDnsDiscoveryService - OvEpMessageActionServer - OvServiceLogger - OvEpStatusEngine - OvOWReqCheckSrv - OvPmad - OvSecurityServer - OvMsmAccessManager - OvowWmiPlatProv - OvServerMonitor This tool will also set the registry key value HKLM\SOFTWARE\Hewlett- Packard\OVEnterprise\Management Server\OvStoreProv\Disabled to 1 to prevent OvStoreProv.exe from restarting on the backup management server. 16

17 Important To clear all user connections on the Backup SQL Server database, stop and restart the SQL Server (<your database instance name>) service, and make sure it is still set to Automatic start. Set cscript as the Default Script Host On both the primary and backup management servers, make the default script host cscript.exe. You need to do this to ensure that the provided scripts use console output rather than popup Message Box windows for progress indications the scripts generate a number of status and progress messages which you can view as Tool output or console message annotations. From a command prompt, run cscript /h:cscript to make cscript.exe the default script host for the server. This will make the script host utilize console output, which is friendlier for batch jobs. Enable Remote DB Access for SQL Server 2005/SQL Express on the Backup Server With SQL Server 2005 / SQL Express, remote database access is disabled by default. The script ship_ovdb.vbs requires remote database access on the backup management server. Note If your database runs on SQL Server 2000 DB, remote access is enabled by default, and you can skip the following steps. To enable remote DB access, execute the following steps: a) On the backup database server, launch the SQL Server Surface Area Configuration Manager b) Click Surface Area Configuration for Services and Connections. c) In the tree selector, click Remote Connections and make sure that the option Local and remote connections is selected. d) Click SQL Server Browser and make sure that the SQL Server Browser service is started automatically. e) Restart the SQL Server instance after SQL Server Browser service has been started. 17

18 Configure the SQL Server Synchronization Tool The installer will have created a new Tool folder called Database Replication Open the Tools\Database Replication folder, right-click on the Re-Synch HPOM Database tool, select Properties, and click the Details tab. Replace BACKUPSERVER with the name of your backup HPOM management server. For more details about the script executed by this tool, see the section "SHIP_OVDB Script". Note that this Tool is configured to Allow the Operator to Change Parameters. This allows an operator to specify a different target server for database synchronization this is important when the backup server has been brought online and the (failed) primary server needs to be brought back into commission. See the discussion of What to do after a failover later in this document. Click the Target tab. 18

19 Replace DOMAIN\USER with an account name that is an administrator of BOTH the PRIMARY and BACKUP management servers. Enter a valid password in the Password and Verify Password fields, and click OK. Initialize the Replication to the Backup Server Execute the Database Replication\Re-Synch HPOM Database tool to start the replication. When you see this job complete successfully you can proceed to the next step. It may take several minutes or more for the job to complete the backup of the primary database, copy the backup file, and restore the database to the backup management server. Configure the SQL Server Synchronization Policies The policy group HPOM-HA is created on the two management servers by the installer utility. The policies in this Policy Group need to be modified before you deploy them. The policies must only be enabled on the primary management server. Policies to modify: Backup-Ship-Restore HPOM Database (Scheduled Task) Backup-Ship-Restore HPOM Logs (Scheduled Task) You need to configure each of the scheduled task policies appropriately. For both policies, you MUST: 1. Enter a valid domain user that is an administrator of both the PRIMARY and BACKUP operations servers 2. Enter a valid password for the domain user 3. Modify the command line, replacing BACKUPSERVER with the name of your backup server. 19

20 Sample modification: The name of the backup management server here is db_backup_server. This server name has been used to replace the default string (which you will see in the policies) of BACKUPSERVER. You should enter your backup management server name in the place of the highlighted text. The first item on the command line should be the keyword DB (with the preceding space) or LOG. Additionally, the Execute as user field needs to be an administrator of both the primary and backup management servers. In this case, we are using the administrator account, but this is not a requirement. You can use any account that has administrator privileges on both the primary and backup management servers. For more details about the script executed by this policy, see the section "SHIP_OVDB Script". You must check the Specify password field, and enter the password for the user entered there. Execute these steps for the Backup-Ship-Restore HPOM Database and Backup-Ship-Restore HPOM Logs policies. The default schedule for the Backup-Ship-Restore HPOM Database policy is EVERY SUNDAY at 23:00 hours. You can change this to whatever works for you. Install the Scheduled Task Policies to Start Automated Log Shipping When the initial database synchronization is complete, you should then deploy the scheduled task policies to the primary management server: HPOM-HA\Backup-Ship-Restore HPOM Database HPOM-HA\Backup-Ship-Restore HPOM Logs 20

21 The Backup-Ship-Restore HPOM Database policy does not only ship and restore a full database backup from the primary to the backup management server. It also generates a full database backup file named OVBackup_Full.bak in the backup -subdirectory of the Database File Location (on the database server) specified during the installation of HP Operations Manager. We recommend that you save this full database backup file on tape because it will be overwritten every time the Backup-Ship-Restore HPOM Database policy is executed. The default schedule for the Backup-Ship-Restore HPOM Logs policy is every 15 MINUTES. If the policy Backup-Ship-Restore HPOM Logs works correctly, it will send a message to the Acknowledged Messages log. You can view this message and its annotation for greater detail. If either policy fails, it logs a MAJOR severity event to the ACTIVE message browser for the primary management server. View this message and its annotation for greater detail. Check also Appendix B for possible issues and fixes. After fixing the issue re-execute the Database Replication\Re-Synch HPOM Database tool to synchronize the databases again. Replicate the Instrumentation Folder after Modifications The installation is now complete. This note is just to make you aware that the instrumentation folder is not replicated from the primary to the backup management server by the replication tools. Thus you should not forget to copy modifications of this folder from the primary to the backup management server manually whenever you change or add something there. Location of the instrumentation folder: HPOM 8.0 %OvShareDir%\Instrumentation Uninstallation Process 1. Uninstall the HPOM high availability policies from the managed nodes: In the console tree, right-click the policy group HPOM-HA and select All Tasks Uninstall from. 2. Delete the policy group HPOM-HA from the management server: In the console tree, right-click the policy group HPOM-HA and select Delete. 3. Delete the HPOM high availability policies from the management server: a. In the console tree, under Agent policies grouped by type, right-click the scheduled task policies Backup-Ship-Restore HPOM Database and Backup-Ship-Restore HPOM Logs. b. Select All Tasks Delete from server. Note that this menu item will only be available if the policy is not deployed on any managed node. 4. Delete the tool group Database Replication: a. In the console tree, right-click Tools and select Configure Tools. b. In the Configure Tools dialog box, right-click Database Replication and select Delete. 5. Delete the registry key [HKEY_LOCAL_MACHINE\SOFTWARE\Hewlett- Packard\OVEnterprise\Management Server\DBReplication]. 6. Delete the folder %OvInstallDir%\Support\OVOW-HA\ including all subfolders. 21

22 7. Delete the file %OvInstallDir%\Support\failoverstart.vbs. 8. Delete the files ship_ovdb.vbs, disable_hpom_services.vbs and enable_hpom_services.vbs from %OvShareDir%\Instrumentation\Categories\VP_SM\Windows. HP Operations Manager Fail-over to Backup Procedure If the primary manager fails for any reason (hardware/network/disaster), a failover to the backup management server will need to be initiated. Because the SQL database on the failover system is standing by, initialization of the backup system takes only a few simple steps. 1) On the backup HPOM management server, call the failoverstart.vbs to bring the backup database online, and start the critical HPOM services. This can be simply done by executing the shortcut called Recover Databases and Start HPOM Processes that has been added to the desktop of the backup management server by the installer. 2) Start a HPOM for Windows MMC Console and connect to the backup management server. 3) Execute the Switch Primary Manager Tool on each managed node (see the next section). 4) Switch certificate server for all nodes To do that launch the tool Switch Agent's Certificate Server (under HPOM Operations Manager Tools > Certificate Management ) 5) Enable the policies disabled in the step Stop HPOM for Windows on the Backup Server. At this point, the backup management server is at the state of the primary manager when the last transaction log backup was shipped. Using the Switch Primary Manager Tool The Switch Manager tool provides a mechanism to instruct managed systems that their agents should switch their primary manager to a different HPOM for Windows server. Typically this tool will be used when it is necessary to switch to using the backup HPOM for Windows management server (with the replicated database) as the primary management server. The tool is located in the Tools folder called Database Replication and is called Switch Management Server tool. Before you can use the Switch Management Server Tool from the Backup HPOM server you MUST configure and deploy the MultipleActionManagers policy correctly, as described earlier in this document. The Switch Management Server tool utilizes the opcragt tool that is part of the HPOM for Windows distribution. The Switch Management Server Tool should be used from the backup management server to instruct HPOM for Windows agents that they should switch their primary manager from the current primary management server to the 22

23 backup (which is issuing the command). The Switch Management Server iterates through all managed nodes and sends their agents a command to switch to the new management server. The old primary management server is also a managed node of the backup management server. If the old primary management server is unavailable when this tool is executed, the command will fail for this node. Thus the completion state of the Switch Management Server tool will be failed although it did its job successfully. What to Do After Fixing the Old Primary Server Once the old primary management server has been fixed there are a number of possible options for continued operations. Two options are described here: 1) Continue using the backup server as the new primary management server, and make the old primary server (which failed) into the backup management server. 23

24 2) Fail back to the old primary management server once it has been fixed. Continue Using the Backup Server as the Primary Server. If you plan to continue using the backup server as your primary HPOM management server (for example, if it has the same hardware resources as your original Primary server and you wish to maintain continuous operations), then you will need to: 1) Establish the new primary server as a permanent primary and prepare it to replicate to the new backup server when it is available. 2) Configure the old primary HPOM management server as a standby backup server. 3) Initialize the database of the new backup management server and start replication. Each of these steps is described below in more detail. In the context of these discussions, old primary server refers to the machine that was originally defined as the primary manager, and which failed. The term new primary server refers to the machine that was originally designated as the backup and which is now acting as the Primary. 1. Establish the new Primary server as a permanent Primary The following steps need to be undertaken on the backup (new primary) server immediately. a) Modify the SQL Server synchronization policy so that the BACKUPSERVER definition refers to the old primary server (which will now become a backup server). See Configure the SQL Server synchronization policies for more details. Note that you only need to update the BACKUPSERVER definition the login credentials for the policies will be correct. b) Modify the Database Replication Tools to change the default target system from the old backup server (now primary) to the old primary server. See Configure the SQL Server synchronization tool for more details. As has been noted, this tool is configured to allow an operator to specify the Target server so this step is optional but desirable as it prevents any risk that the wrong server may be identified when running the tools. 2. Configure the old Primary HPOM server as a standby backup server. Once the old primary server has been fixed, you need to prepare it as a backup server. a) Use the Switch Management Server Tool on the new primary management server to instruct the agent on the old primary management server to switch its primary manager. (NOTE, you already instructed the agent on the new Primary manager and all the other nodes to switch to the new primary management server when you switched the primary manager configurations of the managed systems!). b) DISABLE the database replication policies Backup-Ship-Restore Database and Backup-Ship-Restore Logs from the old primary server. Until you do this, the server will continue to attempt to replicate the database and Policies folder to the existing primary server. The database replication will fail (access violations). c) On the old primary management server (now the backup system), disable the server policies, change the startup type of the HPOM services from Automatic to Disabled and STOP them. See Stop HPOM for Windows on the Backup Server for more information. d) Enable remote DB access for SQL Server 2005/SQL Express on the primary database server See Enable remote DB access for SQL Server 2005/SQL Express on the Backup Server for more details. 24

25 e) Restart the SQL Server instance The old primary server is now prepared as a backup server. 3. Initialize the new Backup servers database and start replication. The final stage is to initialize the database on the new backup server and deploy the replication policies. a) On the new primary management server, run the Resync HPOM Database tool. Reference Initialize the replication to the backup server b) Deploy the replication policies to the new primary management server. Reference Install the Scheduled Task Policies to start automated log shipping That completes the process of establishing the old backup server as a permanent primary server. Fail back to the old Primary once it has been fixed. If you need to failover to the original primary management server (referred to in this section as the old primary server) from the existing new primary management server then the steps to undertake this are described below. 1. The first task, when the old primary management server is restarted, is to DISABLE the database replication policies. Until this is done the old primary management server will continue to attempt to replicate the database to the new primary server. The database replication will fail (access violations). Run the HPOM MMC console on the new primary management server, view the Policy Inventory for the server, select the Backup-Ship-Restore HPOM Database and Backup-Ship-Restore HPOM Logs policies, right click and DISABLE them. 2. Stop the HPOM services on the old primary management server. You should also stop and restart the SQL Server (<your database instance name>) service. See Stop HPOM for Windows on the Backup Server for more information. This step is necessary to allow the database from the current primary management server to be replicated to the old primary management server. 3. Using the HPOM MMC console on the new primary management server (the previous backup server), execute the following steps: a. disable the Backup-Ship-Restore Database and Backup-Ship-Restore Logs policies if they have been deployed and enabled on the new primary management server. b. run the Resync Databases Tool specifying the old primary server as the target server. This requires that you overwrite the old backup server name (in the parameters) with the old primary server name. This synchronizes the database state from the current primary server to the newly recovered old primary server. 4. On the old primary server execute the shortcut called Recover Databases and Start HPOM Processes that has been added to the desktop of the backup management server by the installer. Once this completes, you can start an HPOM MMC console and connect to this machine, as it will, once again, be the primary server. 5. From the HPOM MMC console on the old primary server launch the Switch Management Server Tool to switch the Primary manager of ALL managed nodes (including the current Primary) to the newly recovered old primary server. 25

26 The only steps remaining are to establish the backup system (the machine that has been acting as primary server) and restart replication. 1. First disable the server policies and stop the HPOM services on the backup server (which was referred to as the new primary server until this point). For details, see Stop HPOM for Windows on the Backup Server. 2. Finally, re-enable the Replication Policies on the primary server. a. Run the HPOM MMC console on the old primary server. b. Change the Resync Database Tool and readd the old backup management server (the new primary management server) as target server. c. Run the Resync Database Tool. For details, see Initialize the replication to the backup server. d. View the Policy Inventory for the server. e. Select the Backup-Ship-Restore Database and Backup-Ship-Restore Logs policies, right click and ENABLE them. This completes the failover back to the original primary management sever. 26

27 Appendix A: Upgrade and Patch Installation This is the procedure to upgrade an HPOM for Windows HA-system consisting of primary and a backup server with a database replication in place or to install a patch. Just follow these steps: 1) Disable the Scheduled Task policy Backup-Ship-Restore HPOM Database on the primary management server. 2) Disable the Scheduled Task policy Backup-Ship-Restore HPOM Logs on the primary management server. 3) Upgrade HPOM for Windows or install the patch on the primary server, as described in the HPOM for Windows installation/upgrade guide. 4) If the package includes also a new version of the High Availability package, reinstall this package on the primary server. In this case, you need to re-deploy the instrumentation and the new versions of the policies. 5) Execute the script failoverstart.vbs on the backup server. This brings the database online and starts the HPOM for Windows services. 6) Upgrade HPOM for Windows or install the patch on the backup server. 7) If the package also includes a new version of the High Availability package, reinstall this package on the backup server. In this case, you need to re-deploy the instrumentation and the new versions of the policies. 8) Stop HPOM for Windows on the backup server (see page 13). 9) Run the tool Database Replication\Re-Synch HPOM Database on the primary manager. This will ship the primary database to the backup. Depending on the database size, this could take some time. 10) Wait until the job has completed. Now the database on the primary and the backup server are in sync again. 11) Enable the Scheduled Task policies Backup-Ship-Restore Database, and Backup-Ship-Restore Logs on the primary management server to resume the periodic log shipping and Policy replication to the backup management server. 27

28 Appendix B: Error Codes from VBScripts ship_ovdb.vbs and failoverstart.vbs Error Returns (from the scheduled task policies, from the Re-Sync Database tool or from the failoverstart script) Error 1 Script cannot connect to the primary database server to initiate the backup operation. Verify that SQL Server (<your database instance name>) is running, can be connected from the primary management server, and that you are running the script on the PRIMARY management server. Error 2 Cannot find database 'openview' on the specified primary SQL Server Instance. Is the primary HPOM management server really using this SQL Server Instance? Error 3 Error in getting the path where the datafiles are stored on the primary database server. Error 4 Backup of the database failed. This indicates that the script was able to connect to the local database correctly, but could not complete the backup job for some reason. Examine the output of the tool/command for additional details. This may indicate that the primary backup location is out of disk space. Error 5 Could not connect to the remote (target) server. This indicates that the script cannot connect to the backup server to initiate the restore of the database/log file. The most common cause of this failure is that the user running the scheduled task or tool is not a member of the SQL system administrator role on the backup server. The scheduled task or tool user must be an administrator of both the primary management server and the backup management server. By default, members of the local Administrator group (win2k/2k3) are system administrator (sa) in the SQL instance. Error 6 Cannot find database 'openview' on the specified backup SQL Server Instance. Is the backup HPOM management server really using this SQL Server Instance? Error 7 Error in getting the path where the datafiles are stored on the backup database server. Error 8 Copy of backup file failed! This may indicate one of several possibilities. a. The backup server is not available because of a network/system outage. b. The user that is running the tool/scheduled task does NOT have the correct permissions to the file share on the backup server. (You will see error 5, Access Denied, in this case.) c. The backup server does not have sufficient disk space to accommodate the copy transaction. Error 9 Cannot restore backup on target server. This indicates that the restore procedure failed for some reason. The most common causes for this are: a. A network outage caused a previous log shipment to fail. This causes the replication to go out of phase. To correct this, run the Re-Synch Database tool. 28

29 b. The HPOM processes are NOT shut down on the backup server. Restore operations cannot take place when users are connected to the backup server s HPOM database. Disconnect the users (forcibly, if required) on the backup server. c. There are open transactions when the preceding log backup is performed. In this case, you will see something like this in the script output: Error 4305: [Microsoft][ODBC SQL Server Driver][SQL Server]The log in this backup set begins at LSN , which is too late to apply to the database. To solve this issue you must restore the full backup from the primary management server to the secondary management server, for example, by launching the Re-Synch HPOM Database tool on the primary management server. Error 100 Invalid command line passed into the script. The correct command lines are as follows: Backup-Ship-Restore HPOM Database policy: cscript %ovagentdir%\bin\instrumentation\ship_ovdb.vbs DB <BACKUPSERVER> Backup-Ship-Restore HPOM Log policy: cscript %ovagentdir%\bin\instrumentation\ship_ovdb.vbs LOG <BACKUPSERVER> Re-Synch HPOM Database tool: cscript %ovagentdir%\bin\instrumentation\ship_ovdb.vbs DB <BACKUPSERVER> In all cases, the <BACKUPSERVER> should be the short hostname of the Backup, warm-standby, management server. Error 101 Invalid Action specified on the command line. The only valid actions are DB (used by the Backup-Ship-Restore Database policy and the Re-Synch Database tool), and LOG (used by Backup-Ship-Restore Logs policy) Error 102 The BACKUPSERVER parameter that was supplied is the LOCAL HPOM management server. Cannot replicate the database or copy the Policies folder to the Local machine. Check that the BACKUPSERVER parameter specified in the call to ship_ovdb.vbs script refers to a valid remote backup server. Error 104 Cannot connect to registry. This means that the script cannot access the registry to get the required information. Make sure that the policy is running as a user with administrative privileges on both the primary and backup management servers. Error 105 Can t retrieve SQL Instance Name from the registry. The scripts read HKLM\Software\ Hewlett- Packard\OVEnterprise\Management Server\DBAccess\OvOWInstance from the system registry. If this value does not exist, the script cannot complete. This value should be created during the installation of HPOM. Error 110 SQL-DMO not installed on this system! This might happen if you are using a remote database server or SQL Server To solve this issue, you can install the "Microsoft SQL Server 2005 Backward Compatibility Components" on your HPOM management server systems. 29

30 Appendix C: Customized Configurations Setup log shipping replication combined with a remote database If you want to setup log-shipping replication combined with the HPOM for Windows remote database setup, then do the following: Install an HP Operations 8.0 management server on two separate computers as described below. Follow the steps of this guideline to setup the log shipping replication. But bear in mind that you must install the tools and policies on the management servers and setup log shipping between the two database servers which are on separate machines now. 30

31 For more information Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Itanium is a trademark or registered trademark of Intel Corporation in the U.S. and other countries and is used under license. April 2009