Altiris Consulting Disaster Recovery Checklist NS and SQL Feburary 2006
Disaster Recovery Checklist Page 2 Notice The content in this document represents the current view of Altiris as of the date of publication. Because Altiris responds continually to changing market conditions, this document should not be interpreted as a commitment on the part of Altiris. Altiris cannot guarantee the accuracy of any information presented after the date of publication. While this document is based upon Altiris' consulting experience, its practices may not apply in every situation or environment, and Altiris makes no warranty, express or implied, with respect to any methodology set forth herein, Such methodologies merely constitute suggested methods or processes, which may or may not be optimal given Customer's situation and environment. Copyright 2005, Altiris, Inc. All rights reserved. Altiris, Inc. 588 West 400 South Lindon, UT 84042 Phone: (801) 226-8500 Fax: (801) 226-8506 BootWorks U.S. Patent No. 5,764,593. RapiDeploy U.S. Patent No. 6,144,992. Altiris, BootWorks, Inventory Solution, PC Transplant, RapiDeploy, and RapidInstall are registered trademarks of Altiris, Inc. in the United States. Carbon Copy is a registered trademark licensed to Altiris, Inc. in the United States and a registered trademark of Altiris, Inc. in other countries. Microsoft, Windows, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries. Other brands and names are the property of their respective owners. Information in this document is subject to change without notice. For the latest documentation, visit www.altiris.com.
Disaster Recovery Checklist Page 3 Contents Introduction... 4 Assumptions... 4 Disaster Recovery Scenarios... 4 Standard Build Document... 4 Pre-requisites for Disaster Recovery... 5 Sample Notification Server backup plan... 5 Sample SQL Server backup plan... 5 Recommended installation checklist for Altiris software... 6 Recovery of NS Server with Same Hardware... 7 Recovery of NS SQL Server... 8 Recovery of NS Server with Dissimilar Hardware... 9 Validating the Notification Server Recovery... 16 Appendix A: SQL Server DB - Disaster Recovery... 23 Assumptions... 23 Procedure... 23
Disaster Recovery Checklist Page 4 Introduction Assumptions The checklists in this document contain the steps to recover from the failure of an Altiris Notification Server host and/or the loss of the SQL server used by the Notification Server. This document does not address disaster recovery process for the Altiris Deployment Server. These procedures assume the following: The Notification Server does not reside on the same host server as the SQL server An existing backup solution is in place for both the SQL server and the Notification Server Disaster Recovery Scenarios This document covers three disaster recovery scenarios: In the first scenario, the Notification Server will be recovered from a failure using the same hardware as the failed NS Server. In the second scenario, the supporting SQL server will be recovered from a failure. In the third scenario, the Notification Server (NS) will be recovered from a failure, but the NS server hardware being used for the recovery does not exactly match the original NS server hardware. This third scenario is also applicable in the event of hardware or OS upgrade of the Notification Server. Standard Build Document A best practice, implemented by many of Altiris customers, is to generate a detailed checklist for the proper procedures to install the Operating System and to configure the production servers (including security settings.) Because this checklist is bound to be unique for each organization, it is referenced in this document as the standard build document.
Disaster Recovery Checklist Page 5 Pre-requisites for Disaster Recovery A successful recovery depends upon proper backup procedures put into place and performed on a regularly scheduled basis. Use of enterprise level backup software is strongly recommended. Some example screenshots in this document are from Veritas NetBackup, but similar settings will exist in other enterprise backup solutions. The following list is a sample backup plan for the relevant servers: Sample Notification Server backup plan Implement a full (all files and registry) weekly backup, including system state data If feasible, implement a daily incremental backup Sample SQL Server backup plan The goal is to use a SQL maintenance plan to create dumps of the online NS databases so that the backup solution can capture them. Restoration of the database typically involves rebuilding the SQL server, restoring the dumped database files, and then using SQL Server s tools to convert the dumped files from a backup copy to the live database. Using SQL Enterprise Manager, implement a database maintenance plan to perform: A daily backup job for the database files for the Altiris and Altiris_Incidents databases A 4-hour interval backup job for transaction logs on the Altiris and Altiris_Incidents databases Using the backup solution At a minimum, ensure that all the database backup files are backed up on a daily basis. The default location for those files is \SQL\SQLSERVER\BACKUP\* Implement a full weekly backup (all files and registry), including system state data.
Disaster Recovery Checklist Page 6 Recommended installation checklist for Altiris software One of our recommended best practices is for customers to document the steps performed to install and configure the selected Altiris solutions. This checklist can then be used to recreate the production implementation in a lab environment, recover from a combined server and backup failure, and can be referred to during DR procedures requiring new server hardware for the NS. Recommended items for the installation checklist include: Version(s) of Altiris software currently in production Accounts used for the Altiris service and database accounts Database connection settings Modifications to Altiris configuration files such as coresettings.cfg Registry changes to modify Altiris software such as disabling Alert Manager Customized inventory collection files Imported custom NS objects Configuration settings made within the NS console (if not already documented within a separate NS design document) This checklist should constantly be updated to reflect current settings in production.
Disaster Recovery Checklist Page 7 Recovery of NS Server with Same Hardware In this scenario, the NS server has suffered a critical failure and needs to be recovered using the same hardware as the original NS Server. Examples of critical failure include: The NS server has suffered an unrecoverable software crash. The RAID array suffers the failure of too many hard drives. The IIS service is no longer functional. Important: This procedure is dependent upon the implementation of the recommended backup schedule: a weekly full backup with daily incremental backups. Step Task 1 Follow the steps outlined in your organization s standard build document to Reinstall the OS Install the appropriate backup software client Restore the server from the backup media, and Validate the server s basic health. 2 As part of the backup restoration process, all NS related files, registry entries, and services will be restored. Reboot the server after the restore is complete, and proceed to the section entitled Validating the Notification Server after Recovery in this document.
Disaster Recovery Checklist Page 8 Recovery of NS SQL Server This procedure should be used when the remote SQL server, which is accessed by the NS, has suffered a critical failure. Examples of critical SQL failures include: The server has experienced an unrecoverable software crash The RAID array suffers the failure of too many hard drives, or The Altiris / Altiris_Incidents database becomes corrupt. Important: This procedure is dependent upon the prior implementation of a regular SQL maintenance & backup plan. The database should be backed up daily and the transaction log every 4 hours. Step Task 1 If the SQL services and/or the remote SQL server s OS have failed, then follow the steps outlined in the customer s standard build document to Reinstall the OS Install the appropriate backup software client Restore the server from the backup media, and Validate the server s basic health. 2 Follow the steps outlined in Appendix A: SQL Server Disaster Recovery Checklist, to restore the NS databases - Altiris and Altiris_Incidents. Because DB files are always considered open, this step is necessary as a live backup of the SQL data and transaction logs is not possible using most backup solutions. 3 Reboot the Notification Server to force a clean and immediate reconnection to the database. 4 Proceed to the section entitled Validating the Notification Server recovery in this document.
Disaster Recovery Checklist Page 9 Recovery of NS Server with Dissimilar Hardware This procedure should be used when the NS server has suffered a critical failure and the original hardware configuration is no longer available. Appropriate scenarios include: The original NS server is destroyed or unusable and different hardware must be used as a replacement. A significant hardware upgrade is to be performed on the NS server. The procedure boils down to installing a standalone copy of the NS with all the solution components, swapping the temporary database for the legacy DB, and performing some cleanup tasks. Important: This procedure is dependent on the prior implementation of the recommended backup schedule: a weekly full backup with daily incremental backups. Step Task 1 Ensure that the remote SQL server for the NS is functional, and both the supporting databases (Altiris & Altiris_Incidents) are healthy. If necessary, follow the steps outlined in Appendix A, SQL Server DB - Disaster Recovery. 2 Follow the steps outlined in the customer s standard build document to reinstall the OS. Important: Do not perform a full system restore to the server, since the hardware configuration data in the backup set will not be compatible with the new server. Reconfigure the IIS service to block all non-local IP addresses until the restoration process is complete. This will prevent the existing agents from connecting to the server and modifying their settings. 3 Recreate the NS Server Configuration. Install the core Notification Server on the target host, when prompted to create a new DB named Altiris, instead set the name of the new DB to Altiris_DR_Temp.
Disaster Recovery Checklist Page 10 Step Task License installation at this step is not necessary. Licenses will be reinstalled when the pointer to the legacy NS database is reconfigured. Perform any additional configuration tasks that require changes to the NS server s file system,.cfg file changes, and registry changes. Refer to the Altiris software installation checklist for completeness. 4 Verify that the same versions of Altiris solutions (and hotfixes) are installed as part of the recovery process. To verify, use the following SQL query via SQL Query Analyzer to retrieve the list of installed products on the original Altiris database. SELECT Name, Installed, ModifiedDate FROM SolutionCenterUpdate WHERE (Installed = 1) Repeat the query against the temporary database (Altiris_DR_Temp) and compare the results of the two queries. Install any missing components and/or hotfixes before proceeding. 5 Execute a restore job. From the backup software server s console, select the target NS server and configure it with the following settings: Selectively restore any available directories that contained custom packages such as E:\packages\* Restore any customized files (such as inventory scan templates) located under the \\nsserver\nscap share (%AltirisRoot%\Notification Server\nscap) If available, restore the hotfix updates located at: D:\%AltirisRoot%\Patch Management\Packages\Updates\* Do not restore permissions, as the SIDs will no longer match the users and groups on the new server. Select Overwrite existing files. The following screenshot illustrates the proper settings for use in the restore job if Veritas NetBackup was used:
Disaster Recovery Checklist Page 11 Step Task 6 Verify that the same security roles exist in both the temporary and legacy databases. a. Use the following SQL query via SQL Query Analyzer to retrieve the list of Altiris Roles defined in the original Altiris database: SELECT FROM Name SecurityRole b. Repeat the query against the temporary database (Altiris_DR_Temp) and compare the results of the two queries. 7 Recreate any missing roles. a. From the NS web console, authenticate as a member of the Altiris Administrators role. b. Select Configuration Tab > Configuration >Server Settings > Security Role Management. d. Click the Blue cross icon to Add a new role, a new window will
Disaster Recovery Checklist Page 12 Step Task open. e. Type the name of the missing role in the Name textbox. f. Click the Add button: a new window will open. g. To ensure that the domain and group radio buttons are selected, enter the name of the domain (it defaults to the NS s domain). h. Enter the name of domain group that correlates to the Altiris Role. i. Click Find. j. Select the appropriate group from the Results box. k. Click OK, the Select Users or Groups window will close after a few seconds. l. If additional domain groups belong in a given role, click the Add button again, and repeat as necessary. m. Click Save. The Role Properties window will close after a few seconds. n. Click Apply. Repeat the above steps for each Role that needs to be recreated Note: It is not necessary to recreate the object and scope security settings for each of the roles, as that data is stored in the database.
Disaster Recovery Checklist Page 13 Step Task 8 Update the original Altiris database with the User and Group SIDs that correspond to the Altiris Roles. This is necessary because the new NS server has a different SID. Use the following SQL script to complete the task: DECLARE @DBSecurityworking varchar(50), @DBSecuritybroken varchar(50), @strsql nvarchar(4000) SET @DBSecurityworking = 'Altiris_DR_Temp.dbo' SET @DBSecuritybroken = 'Altiris.dbo' SET @strsql = 'UPDATE ' + @DBSecuritybroken + '.SecurityTrustee SET ' + @DBSecuritybroken + '.SecurityTrustee.Trustee = secfix.[good Trustee] FROM ' + @DBSecuritybroken + '.SecurityTrustee, (SELECT a2.trusteeid, a1.[name], a1.trustee AS [Good Trustee], a2.trustee FROM (SELECT st.trusteeid, ISNULL(sr.[Name], ''N/A'') AS [Name], st.[trustee] FROM ' + @DBSecurityworking + '.SecurityTrustee AS st LEFT OUTER JOIN ' + @DBSecurityworking + '.SecurityRole AS sr ON st.guid = sr.trusteeguid) a1 INNER JOIN (SELECT st.trusteeid, ISNULL(sr.[Name], ''N/A'') AS [Name], st.[trustee] FROM ' + @DBSecuritybroken + '.SecurityTrustee AS st LEFT OUTER JOIN ' + @DBSecuritybroken + '.SecurityRole AS sr ON st.guid = sr.trusteeguid) a2 ON a1.[name] = a2.[name] WHERE (a1.[name] <> ''N/A'') --Security Roles OR ((a2.[trusteeid] = 1) AND (a1.[trusteeid] = 1)) --Admin account ) secfix WHERE secfix.trusteeid = ' + @DBSecuritybroken + '.SecurityTrustee.TrusteeId AND secfix.[good Trustee] <> ' + @DBSecuritybroken + 'SecurityTrustee.[Trustee]'
Disaster Recovery Checklist Page 14 Step Task 9 If an error is returned, check to ensure that the following database names were correctly defined in the script: SET @DBSecurityworking = 'Altiris_DR_Temp.dbo' SET @DBSecuritybroken = 'Altiris.dbo' 10 Change the database pointer on the new NS server to point to the legacy Altiris database. a. Select Configuration Tab > Configuration > Server Settings > Database Settings b. Select the list box next to the Use existing database, and change from Altiris_DR_Temp to Altiris c. Click the apply button d. Acknowledge the warning message about leaving the system in a paused state A window titled Reset database will come up, it may take several minutes for the switchover to complete. Once complete, the browser will refresh, and the NS console should now display all the legacy NS configuration data. 11 For patch 6.0 Recreate the MBSA vulnerability scan package. a. Select the Configuration Tab > Configuration > Solution Settings > Software Management > Patch Management > Server Settings > Background Actions. b. Right-click on the Create Microsoft Baseline Security Analyzer Distribution Package and select Start Task. Note: This task has a dependency upon the prior execution of the MBSASetup-en.msi file that was completed during the NS server installation checklist as covered in step 3. 12 If non-default web ports were used for the Altiris NS: a. Reconfigure the IIS listening ports b. Reconfigure the Altiris Agent settings to point to the nondefault port.
Disaster Recovery Checklist Page 15 Step Task 13 Resume the Notification Server client processing. a. Disable IP address filtering on the IIS Service (non-local addresses were blocked in an earlier step) b. Using the NS Console, select Configuration Tab > Configuration > Server Settings > Notification Server Settings > Advanced Settings c. Check the box marked Enable Notification Server Processing. d. Click the Apply button. 14 Install the Altiris licenses using the NS console 15 Proceed to the next section, Validating the Notification Server Recovery.
Disaster Recovery Checklist Page 16 Validating the Notification Server Recovery Follow these steps to validate the recovery of the Notification Server and/or the remote SQL server for the functional NS server. Step Task 1 Verify the health of all the packages (NS core packages, software delivery, and patch management). a. Bring up the NS Log viewer page in a new browser window http://localhost/altiris/ns/logview.asp Note: This URL assumes that the browser is located on the NS server. b. From the NS server s desktop, open the scheduled tasks Start Menu > Programs > Accessories > System Tools > Scheduled Tasks c. Right-click on the NS.Package.Refresh task and select Run. Note: Part of the Package Refresh process is recreating the SMB and Web shares for each package. This process does NOT manage all the necessary NTFS permissions. If download problems persist after fixing any missing file problems, and re-executing the Package Refresh task, then double check the NTFS permission inheritance.
Disaster Recovery Checklist Page 17 Step Task Specific NTFS settings for each package are beyond the scope of this document, but, by default, the group Everyone should have Read and Execute permissions to the package folder and its contents. 2 On the NS Log viewer web page, click the Refresh button If any Software Delivery packages are broken due to missing files, a message similar to the following will be found in the log: Nov 19 09:37:47 Access to the directory:[ E:\packages\WSH \ ] was denied and returned the following 'system error code':[ 3 ] LogFileFin derror AltirisNativeHelper.dll aexsvc.exe (2412) Nov 19 09:37:47 Nov 19 09:37:47 Unable to create the distribution shares for package "Windows Scripting Host" (Guid: 2fb23292- ed6f-45e6-b18d- 8ad1582d6992). Reason: The system cannot find the file specified The package with guid 2fb23292- ed6f-45e6-b18d- 8ad1582d6992 references a location on the Notification Server that does not exist (E:\packages\WS H) Altiris.NS. StandardIte ms. SoftwareD elivery.sw D Support.M anagedistri bu tionpoints AltirisNativeHelper.dll aexsvc.exe (2412) 3284 3284 3 Restore the specified missing directory along with the missing package source files. If the missing files are not available from the backup job, they may also be found on the package servers that were previously assigned to the package. 4 Repeat step 1 to determine if any Patch Management packages are broken. a. On the NS Log viewer web page, click the Refresh button.
Disaster Recovery Checklist Page 18 Step Task The following sample error message indicates a problem with a Patch Management bulletin package: Access to the directory:[ D:\Altiris\Patch Management\Packages\Updates\MS04-014\{1870361d- 3db0-4c77-b48d-8811cefb8db2}\ ] was denied and returned the following 'system error code':[ 3 ] The package with guid 93854908-d326-49d6-bada-55d565f6528e references a location on the Notification Server that does not exist (D:\Altiris\Patch Management\Packages\Updates\MS04-024\{71869aa3-9c6f-448c-89e5-654d31a9d868}) Unable to create the distribution shares for package "WindowsServer2003-KB839645-x86-enu.exe" (Guid: 93854908-d326-49d6-bada-55d565f6528e). Reason: The system cannot find the path specified b. To correct the error, go to Task Tab > Software Management > Patch Management > Software Bulletins. Then ensure that the target bulletin is still enabled (i.e., set to TRUE). c. To rebuild ALL broken/missing PM packages, right-click on any enabled bulletin, and select Recreate All Packages. The status of the PM package recreation may be monitored by refreshing the NS Log viewer. Starting log message will be: Beginning Patch Management Software Update Packages download task In-process message will look similar to : Beginning Patch Management Software Update Package download {77c5832e-ca6e- 4576-8129-f6e86421779d} Ending message will be: Finished Patch Management Software Update Packages download task 5 Repeat step 1 and view the log a final time to determine if all packages are now healthy. 6 Check the status of the package servers to determine if they require any maintenance work. a. Go to Configuration Tab > Configuration > Server Settings > Notification Server Infrastructure > Package Servers. b. For each Package Server, compare the number of Available Packages vs. Pending Packages. Make a note of which package servers still have pending packages. Wait 3 to 6 hours (dependant upon the PS client s configuration interval) and repeat this step to check on the progress of the pending packages.
Disaster Recovery Checklist Page 19 Step Task 7 (Optional) Monitor a faulty Package Server via the Package Server s local Windows console a. From the faulty Package Server s console, double click the Altiris Agent icon. b. Select the Package Server tab c. Click the Package Server update button d. Monitor the status of each package in the window. For any failed packages, verify that the source of the package is valid (communications, permissions, DNS resolution, etc) 8 (Optional) Remotely fix (reinstall) a faulty Package Server from the NS Console a. Select the faulty package server from the Package Servers configuration tool listed above. b. Click the Red X icon to delete the Package Server agent from the target server. c. Click the Apply button. d. Wait a sufficient amount of time for the Package Server plug-in to uninstall on the remote server (5-10 minutes). e. Click the blue cross to Add a new package server. f. Enter the original package server s name. g. Click Find. h. Select the package server. i. Click OK. j. Click Apply. k. Monitor the package server status to determine if any package problems persist. If so revisit steps 1-3 to resolve. The most common fault at this stage is insufficient NTFS permissions. 9 Verify the status of the installed licenses. a. From the NS console, go to Configuration Tab > Configuration > Licensing.
Disaster Recovery Checklist Page 20 Step Task b. Ensure that no product entries exist under the Trial Licenses section. c. Also verify that for each entry under Installed Licenses, the number of In Use licenses is less than the number of Total licenses. 10 Verify proper communication between the Altiris Agents and the rebuilt server. The basic communication activity of the existing Altiris Agents can be viewed by running a report. Perform this step after a significant portion of the client s configuration interval time has passed (at least 2 hours). a. From the NS console, go to Reports Tab > Reports> Notification Server Infrastructure > Configuration Request > Last N Configuration requests. b. Check the Request Date for recent activity. The amount of recent activity will be impacted by how long the NS server has been in a functional state. 11 (Recommended) Spot check a few Altiris Agents from their local server s console. a. First, double-click the Altiris agent icon in the icon tray. If the icon is not visible, launch the executable AeXAgentActivate.exe which is located in the program directory Altiris Agent. b. Next, click the about button found on the Altiris Agent window as illustrated below. c. On the About Altiris Agent window that opens, select the Altiris Agent Details tab (shown below).
Disaster Recovery Checklist Page 21 Step Task d. Click the Update Configuration button. The Altiris Agent will attempt to connect to the NS server, and request any configuration changes that have been assigned to this specific agent since it last checked in. A successful check-in is indicated by an update to the Configuration Last Changed timestamp. e. If the agent is unable to obtain an updated configuration file, then review the agent.log found in the Altiris Agent directory. Note: Repeatedly pressing the Update Configuration button may not result in any change to the timestamp, as the NS will not have identified any new tasks to be assigned to the agent. Wait a few minutes, and click the button again to retest the communication. 12 Validate the successful restoration of Altiris Roles and their security assignments. Instruct a member of each Altiris role to test their access to the NS console, and to their previously assigned collections, tasks, and reports. 13 Validate the proper functionality of inventory forwarding between the NS server and the primary NS reporting server. a. Determine the current Inventory Forwarding schedule, by going to Configuration Tab > Configuration > Server Settings > Notification Server Infrastructure > Inventory Forwarding tool. Consult the Rule Description table to determine the last time an inventory data-forwarding job was
Disaster Recovery Checklist Page 22 Step Task scheduled to run. b. From the regional NS server s web console, select the Reports Tab > Reports > Notification Server Infrastructure > Server > Inventory Forwarding > Inventory forwarding status. c. Compare the last successful forwarding event with the expected last job run. d. If the last successful time does not match the last scheduled time, then return to the Inventory Forwarding tool. Click the resynchronize icon to force an immediate inventory-forwarding job. e. Resolve any errors reported after clicking the icon.
Disaster Recovery Checklist Page 23 Appendix A: SQL Server DB - Disaster Recovery Assumptions Procedure Note that the following sample step-by-step instructions apply to the SQL Server 2000 user databases. Thes instructions assume that The server s OS has already been rebuilt or repaired. The SQL services have been installed/re-installed as necessary. Recovery of the master, msdb, and model databases is not covered. Step Task 1. Log on to the SQL server with an administrative account (such as the SQL SA account or a local administrator on the SQL server.) 2. Using SQL Server Enterprise Manager, check all the databases to verify that the system databases are online and the user databases are in suspect mode (or non-existent). 3. Close all open instances of SQL Server Enterprise Manager connected to the SQL server (including the one used to perform prior steps). 4. Stop the SQLServerAgent service. 5. Stop the MSSQLSERVER service. 6. Place the SQL server into single user mode: Go to the command prompt and run "sqlservr s <instance name> -m" from the \MSSQL\Binn folder to start MSSQLSERVER or MSSQL$<instancename> 7. List all the databases shown as (Suspect) in the SQL Enterprise Manager (or are known to be missing). Altiris Altiris_Incidents
Disaster Recovery Checklist Page 24 Step Task 8. Use SQL Enterprise Manager to restore the above databases one by one from the full backup stored in \MSSQL\Backup\<databasename>\<databasename>_db_*.bak. a. Make sure that the location of the MDF and LDF files are correct. b. If you are going to restore the transaction logs / differential backups then make sure you choose Leave database nonoperational, but be able to restore additional transaction logs. c. After the restore is done, and if you want to restore transaction logs / differential backups, make sure that the database is in (Loading) state. 9. Restore these (Loading) databases one by one from the differential / transaction log backups Stored in \MSSQL\Backup\<databasename>\<databasename>_tlog_*.trn. If more than one transaction logs are to be restored, then choose Leave database nonoperational, but be able to restore additional transaction logs. Do not choose this option for the last transaction logs / differential backups. Remember to choose Database differential or Transaction Log before the restore. 10. Make sure that after the databases are restored, none of them are in (Suspect) or (Loading) state. 11. Check the SQL Server error logs. Go to SQL Enterprise Mgr local DB Management SQL Server Logs Current Timestamp to verify that all the databases were recovered successfully. 12. Close SQL Server Enterprise manager. 13. Restart MSSQLSERVER service. 14. Start SQLServerAgent service.
Disaster Recovery Checklist Page 25 Step Task 15. Check the application log, found in the NT event viewer, to verify that MSSQLServer and SQLServer Agent started without any errors. 16. Use SQL Server Enterprise Manager to recreate any lost maintenance jobs for integrity checking, database optimization and backup jobs.