MCAPS 3000 DISASTER RECOVERY GUIDE

MCAPS 3000 DISASTER RECOVERY GUIDE Manual Part Number 99875294-1 FEBRUARY 2004 REGISTERED TO ISO 9001:2000 1710 Apollo Court Seal Beach, CA 90740 Phone: (562) 546-6400 FAX: (562) 546-6301 Technical Support: (651) 415-6800 www.magtek.com

Copyright 2004 MagTek, Inc. Printed in the United States of America Information in this document is subject to change without notice. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without the express written permission of MagTek, Inc. MagTek and MCAPS 3000 are registered trademarks of MagTek, Inc. Microsoft is a registered trademark of Microsoft Corporation. REVISIONS Rev Number Date Description 1 12 Feb 04 Initial Release ii

TABLE OF CONTENTS SECTION 1. FOREWORD... 1 SECTION 2. OVERVIEW... 3 RELATED DOCUMENTS... 4 SECTION 3. DISASTER RECOVERY IMPLEMENTATION... 5 PRIMARY AND BACKUP MCAPS 3000 HOST... 5 SAVING THE MCAPS 3000 DATABASE... 5 HOST SWITCHOVER/FAILOVER... 5 SECTION 4. HOST FAILOVER... 7 BACKGROUND, CAPS TERMINAL OPERATION... 7 UNIQUE HOST IP ADDRESS... 7 DIAL-UP TERMINALS... 7 AUTOMATIC FAILOVER... 8 MANUAL FAILOVER... 8 SECTION 5. MCAPS 3000 DATABASE: SAVING TO REMOTE SQL SERVER... 9 DATA TRANSFORMATION SERVICES (DTS) - MICROSOFT SQL DATABASE... 9 TESTING THE DTS PACKAGE... 21 VERIFYING THE REMOTE DATABASE... 24 MODIFYING THE DTS SCHEDULE...25 SECTION 6. MCAPS 3000 DATABASE: SAVING TO NETWORKED STORAGE... 27 CONFIGURING DTS ON THE BACKUP HOST... 35 SECTION 7. DISASTER RECOVERY DRILLS... 37 DESIGNATED DISASTER RECOVERY TERMINALS... 37 MCAPS 3000 CONSOLE OPERATIONS... 37 SECTION 8. LOCAL DATABASE BACK UP AND RECOVER... 39 BACK UP/RECOVER MICROSOFT SQL DATABASE... 39 SECTION 9. HARDWARE SECURITY MODULE... 51 HSM REDUNDANCY... 51 DES KEY MANAGEMENT... 51 APPENDIX A. MAINTAINING TWO PRIMARY HOSTS... 53 PRIMARY HOST - PRIMARY HOST CONFIGURATION... 53 USER DATABASE UPDATES... 53 TRANSACTION DATABASE... 53 CAPS TERMINAL INSTALLATION AND TSN... 53 INDEX... 55 TABLE OF FIGURES Figure 5-1. Accessing the Data Transformation Services (1 Database)... 9 Figure 5-2. Accessing the Data Transformation Services (2 MCAPS 3000)... 10 Figure 5-3. Accessing the Data Transformation Services (3 Export Data)... 11 Figure 5-4. DTS Wizard Welcome Screen... 12 Figure 5-5. Select Data Source... 13 Figure 5-6. Select Data Destination... 14 Figure 5-7. Specify Table Copy... 15 Figure 5-8. Select Objects to Copy... 16 Figure 5.9. Schedule DTS... 17 Figure 5-10. Select DTS Name and Description... 18 Figure 5-11. Save DTS Package... 18 Figure 5-12. Completing the DTS Wizard (1)... 19 Figure 5-13. Completing the DTS Wizard (2)... 20 Figure 5-14. Completing the DTS Wizard (3)... 20 Figure 5-15. Data Transformation Services... 21 Figure 5-16. DTS Local Packages... 22 iii

Figure 5-17. DTS Local Package Execution... 22 Figure 5-18. DTS Package Execution... 23 Figure 5-19. Successful Completion of DTS Package Execution... 23 Figure 5-20. Verifying Database... 24 Figure 5-21. Modifying DTS Schedule... 25 Figure 6-1. Select Microsoft Access Sequence... 28 Figure 6-2. DTS Package Execution (1 Destination)... 29 Figure 6-3. DTS Package Execution (2 Destination)... 29 Figure 6-4. DTS Package Execution (3 Destination)... 30 Figure 6-5. DTS Package Execution (Select File)... 30 Figure 6-6. DTS Package Execution (Save File 1)... 31 Figure 6.7. DTS Package Execution (Save File 2)... 31 Figure 6.8. Select Table Copy or Query... 32 Figure 6-9. Select Source Tables and Views... 33 Figure 6-10. Select Source Tables and Views (After Select All)... 33 Figure 6-11. Save DTS Package... 34 Figure 6-12. Execution Complete... 34 Figure 6-13. MCAPS 3000 Database in Microsoft Access DB Format... 35 Figure 8-1. Accessing the Database Maintenance Planner... 39 Figure 8-2. Database Maintenance Plan Wizard Welcome Screen... 40 Figure 8-3. Select Databases... 41 Figure 8-4. Update Data Optimization Information... 42 Figure 8-5. Database Integrity Check... 43 Figure 8.6. Specify Database Backup Plan... 44 Figure 8.7. Specify Backup Disk Directory... 45 Figure 8-8. Specify Transaction Log Backup Plan... 46 Figure 8-9. Specify Transaction Log Backup Disk Directory... 47 Figure 8-10. Reports To Generate... 48 Figure 8-11. Maintenance Plan History... 49 Figure 8-12. Completing the Database Maintenance Plan Wizard... 50 iv

SECTION 1. FOREWORD To minimize disruptions caused by total or partial loss of an institution s Primary data processing center, a Backup DP center is frequently set up and maintained. The Backup DP center s data are regularly updated from the Primary center. The term Disaster Recovery is commonly used to identify the Backup system and the process of regularly updating data at the Backup center from the Primary site. There are many ways of accomplishing Disaster Recovery, including a variety of methods using tools provided by Microsoft s SQL Server DBMS 1, the native DBMS of MCAPS 3000. This document details a few specific methods of accomplishing Disaster Recovery for MCAPS 3000. However, given that each institution s needs and resources are different, it is recommended that this document be used only as a general guide: configuration of the Backup system and database update and frequency should be tailored for the specific requirements of the institution. 1 Microsoft SQL Server 2000 Enterprise Version has a rich set of tools for saving and restoring a user database. The three main tools are: - Database backup and recovery - Data Transformation Services - Replication. Detailed steps of saving/restoring the MCAPS 3000 database using Database Backup/Recovery and Data Transformation Services are listed in this document. Replication is not covered at present. 1

MCAPS 3000. Disaster Recovery Guide 2

SECTION 2. OVERVIEW At the foundation of successful operation of MagTek Card Activation and PIN Selection (MCAPS) system is the MCAPS 3000 database. This database stores items necessary for the operation of MCAPS 3000 such as the users of the system (e.g. Security Officers), CAPS terminal transaction data, and which terminals are installed on the system. Clearly, the loss of this database would be catastrophic. When MCAPS 3000 is installed and running on a Clustered hardware host, there are hardware redundancies such as RAID 1 hard disks that greatly minimize the possibility of the loss of database due to failure of some hardware component local to the MCAPS 3000 Host 2. In addition, a policy of regularly backing up the database to media, outside of and apart from, the MCAPS 3000 Host such as a tape or network drive - adds another layer of database protection. Assuming the hardware redundancy described above is present and the database is regularly backed up, recovery of data is relatively easy, provided the Host hardware and its environment as a whole is intact. However, in case of some disaster that causes irreparable damage to the server hardware or to the site, recovery of the Host may take such a long time that it will have a negative impact on the customer base of the institution. To prepare for such contingencies, institutions setup a Backup Host, in addition to the Primary Host. This is a common method for providing relatively seamless switchover of operations for On-Line Application/Transaction Processing systems and is generally called Disaster Recover. This document describes Disaster Recovery as it applies to MCAPS 3000 and various methods of implementation. 2 An MCAPS 3000 application can run either on a single Server or a Cluster, which consists of two Servers. We will use the generic term Host if the context does not require to make a distinction between a Single-Server or Cluster hardware setup. Also, unless noted otherwise, Host will designate MCAPS 3000 Host. 3

MCAPS 3000. Disaster Recovery Guide RELATED DOCUMENTS MagTek Part Number 99875245, MCAPS 3000. Operation Manual Other For information on MS SQL visit www.microsoft.com. 4

SECTION 3. DISASTER RECOVERY IMPLEMENTATION Implementation of a successful Disaster Recovery plan consists of the following elements: 1. Setup and configuration of a Backup MCAPS 3000 Host. 2. Periodic, automatic saving of the MCAPS 3000 Database to a site that is geographically removed 3 from the Primary MCAPS 3000 Host 3. Host switchover in case of Disaster. PRIMARY AND BACKUP MCAPS 3000 HOST The Backup MCAPS 3000 host will be a configured setup similar to the Primary host. Preferably it would use an identical hardware configuration as the Primary. However, this is not strictly necessary. For example, the Primary can be a full High Availability Cluster: 2 Servers, RAID 1 drives for Servers and Shared Storage, redundant power supplies, 2 MCAPS 3000 HSMs, etc. The Backup can either be a single Server, or a Cluster with reduced hardware redundancy. SAVING THE MCAPS 3000 DATABASE The database from the Primary host needs to be saved periodically to a remote site. The periodicity is determined by the Customer. Microsoft SQL Server Database Management System s Data Transformation Services (DTS) Utility allows automatic saving of the MCAPS 3000 database to a remote site. DTS update granularity is 1 minute: updates can technically occur every 1-minute, although clearly this would not be very realistic. Depending on system size and how often the MCAPS 3000 database is changed, updates can be scheduled to occur every 1 hour, every 6 hours, every 1 day, etc. DTS setup details will be covered below. To what particular site and in what form the database is saved depends largely on the corporate network setup of the given institution. For example, if the Primary host s SQL Server can detect the Remote host s SQL Server on the corporate net, then the database can be copied directly. If, however due to firewalls or other security restrictions the remote host is not accessible on a SQL Server 4 peer-to-peer basis, then intermediate save-and-forward steps will be necessary. These will be covered in detail below. HOST SWITCHOVER/FAILOVER There are two main methods of switchover from Primary Host to Backup Host: Manual and Automatic. In PrimaryHost-BackupHost topology, the Backup is on standby i.e., it takes over when the Primary is unable to process transactions. Each host has its own independent MCAPS 3000 database, therefore, CAPS terminals can connect to one and only one Host at one time. The issues associated with each will be covered in more detail below. 3 Note that this is an important distinction: Disaster Recovery assumes that all hardware in the immediate vicinity of the Primary MCAPS 3000 Host may be destroyed or damaged, including any tapes saved locally. 4 Unless noted otherwise, the term SQL Server refers to Microsoft SQL Server DBMS. 5

MCAPS 3000. Disaster Recovery Guide 6

SECTION 4. HOST FAILOVER BACKGROUND, CAPS TERMINAL OPERATION In the CAPS system, the terminals always initiate a transaction. As such, the Host IP address must be static and known to the IP interface attached to the terminal, although the IP address of the terminal itself can be dynamically assigned. The IP interface hardware or software allows for programming more than one Host IP address. The host IP addresses are stored in a list. When a transaction is initiated, the first host IP address in the list is contacted. If there is no response, the next host IP in the list is contacted, and so on. In most MCAPS 3000 applications only one host IP address will be stored, in which case the same host is retried until the number of retries is exhausted. UNIQUE HOST IP ADDRESS In this configuration the Primary MCAPS 3000 Host and Backup MCAPS 3000 Host have unique addresses, e.g. 172.21.10.100 and 172.21.10.200. Both addresses are pre-programmed into the CAPS terminal IP interface. The first IP address on the list is the Primary MCAPS 3000 Host. The second on the list is the Backup MCAPS 3000 Host. With this configuration, and the Backup Host always UP, a fully automatic failover from Primary to Backup is possible. However, this method is not recommended. The possibility of temporary non-connect to Primary Host causes the CAPS terminals IP interface to try the second host on the list and frequently gets a connect to the Backup 5. Duplicate Host IP Address In this configuration the Primary MCAPS 3000 Host and Backup MCAPS 3000 Host have the same address, e.g. 172.21.10.100 and 172.21.10.100. DIAL-UP TERMINALS The Host phone number for dial-up terminals is stored in the CAPS4. There is no provision to store multiple phone numbers in the standard CAPS terminal application. The failover to the remote Host must be done at the phone switch. 5 See additional discussion on this subject in Appendix A. 7

MCAPS 3000. Disaster Recovery Guide AUTOMATIC FAILOVER Automatic switchover is based on the use of health-poll. A mechanism is setup whereby the CAPS terminal transactions are routed either to the Primary or Backup Host based on their UP or DOWN status, which is determined by periodically polling each Host. The danger here is that if the Primary is improperly marked DOWN, then CAPS transactions will be improperly routed to the Backup Host. As a result, some transactions that should have been sent to the Primary Host will be routed to the Backup Host. Though requiring intervention by skilled personnel, MANUAL FAILOVER is preferred, since the possibility of unintended failover is greatly reduced. MANUAL FAILOVER Manual failover can be accomplished by a variety of methods. Some of the more common ones: Start/Stop MCAPS 3000Service via MCAPS 3000Start. The Backup Host is UP and running 6, but MCAPS 3000Service is Stopped. When needed, MCAPS 3000Service is started (in both Servers, if the Host is a Cluster) and the CAPS terminal transactions will be processed by the Backup Host. Backup Host is UP and MCAPS 3000Service is Running. However, the Public 7 IP Ethernet connection to corporate WAN/LAN is disconnected. When needed, the connection is made, and CAPS terminal transactions will be processed by the Backup Host. 6 Although the Backup hardware does not need to be powered up and running, it is strongly recommended that it be so. This will facilitate DR drills, and minimize the possibility that the system will not come up when it s most needed. 7 Ensure the Private network cables between the Servers are not disconnected. 8

SECTION 5. MCAPS 3000 DATABASE: SAVING TO REMOTE SQL SERVER This section describes the method of saving the MCAPS 3000 database from Primary host to Backup host directly. It assumes that the Backup host is accessible from the Primary host on a SQL Server peer-to-peer basis. Microsoft SQL Server DBMS provides various tools for saving and restoring a user s database. This section describes MCAPS 3000 database saving and recovery using the Data Transformation Services utility. DATA TRANSFORMATION SERVICES (DTS) - MICROSOFT SQL DATABASE On the Server workstation, access the Data Transformation Services as follows: Invoke SQL Server Enterprise Manager, then click Databases, as indicated in Figure 5-1. Figure 5-1. Accessing the Data Transformation Services (1 Database) 9

MCAPS 3000. Disaster Recovery Guide Highlight MCAPS 3000 8 database, then Click Tools, select Data Transformation Services, and then Click Export Data, as indicated in Figures 5-2 and 5-3. Figure 5-2. Accessing the Data Transformation Services (2 MCAPS 3000) 8 The name MCAPS 3000 may also appear as MCAPS 3000-01 MCAPS 30001, etc., depending on the name assigned to the MCAPS 3000 SQL Database during MCAPS 3000 Installation. 10

Section 5. MCAPS 3000 Database: Saving to Remote SQL Server Figure 5-3. Accessing the Data Transformation Services (3 Export Data) 11

MCAPS 3000. Disaster Recovery Guide The welcome page will pop up, as shown in Figure 5-4. Click Next to begin using the wizard. Figure 5-4. DTS Wizard Welcome Screen 12

Section 5. MCAPS 3000 Database: Saving to Remote SQL Server First, select the database to be saved. Generally, only the MCAPS 3000 9 database file needs backup as indicated in Figure 5-5. Figure 5-5. Select Data Source For the Server select MCAPS 3000SQL 10. Note: if there are other installed SQL Servers on the Domain, ensure that the correct one is selected for the Data Source, i.e. the Primary MCAPS 3000 Server/Cluster. Click Next. 9 The name MCAPS 3000 may also appear as MCAPS 3000-01 MCAPS 30001, etc., depending on the name assigned to the MCAPS 3000 SQL Database during MCAPS 3000 Installation. 10 The name MCAPS 3000SQL may be different in your system: it is the name that was assigned during the installation of Microsoft SQL Server on your computer. 13

MCAPS 3000. Disaster Recovery Guide The destination selection screen will appear as shown in Figure 5-6. For the Destination select Microsoft OLE DB Provider for SQL Server. For Server select the correct name for the Backup 11 server 12. Then for the Database select the correct database name. Figure 5-6. Select Data Destination Click Next. 11 Although MCAPS 3000SQL is shown here, the actual names of Primary and Backup SQL Servers will be different, since no two SQL Servers on the same net that visible to each other can have the same name. 12 If no other SQL Servers are found on the pull down menu for Server, then the Backup SQL Server cannot be seen by the Primary SQL Server: contact your organizations MIS for assistance. 14

Section 5. MCAPS 3000 Database: Saving to Remote SQL Server As shown in Figure 5-7, select Copy objects and data between SQL Server databases. Click Next Figure 5-7. Specify Table Copy 15

MCAPS 3000. Disaster Recovery Guide Select the objects as shown in Figure 5-8 and Click Next. Figure 5-8. Select Objects to Copy This next menu is used to schedule the DTS, as shown in Figure 5-9. 16

Section 5. MCAPS 3000 Database: Saving to Remote SQL Server Figure 5.9. Schedule DTS Select the fields as shown above and Click Next. The DTS schedule can be changed after it has been configured For Name and Description fields enter appropriate text as shown in Figures 5-10 and 5-11 and click Next. 17

MCAPS 3000. Disaster Recovery Guide Figure 5-10. Select DTS Name and Description Figure 5-11. Save DTS Package 18

Section 5. MCAPS 3000 Database: Saving to Remote SQL Server In the Summary window, Figure 5-12, verify that all selections are correct and click Finish. Figure 5-12. Completing the DTS Wizard (1) 19

MCAPS 3000. Disaster Recovery Guide After you click Finish, you should see the following two screens, Figures 5-13 and 5-14, shown below. Note that Executing Package here indicates not the copying of the database itself, but the process of initializing and saving the DTS Package. If the process was successful, click Done. Figure 5-13. Completing the DTS Wizard (2) 20 Figure 5-14. Completing the DTS Wizard (3)

Section 5. MCAPS 3000 Database: Saving to Remote SQL Server TESTING THE DTS PACKAGE Next, the DTS package will be verified to ensure that upon a scheduled time, the package will copy the correct database from the correct source to the correct destination, with all the MCAPS 3000 tables and fields copied. Invoke SQL Server Enterprise Manager, then click Data Transformation Services, as indicated in Figure 5-15. Figure 5-15. Data Transformation Services 21

MCAPS 3000. Disaster Recovery Guide Expand subfiles (Figure 5-16) by clicking on the + left of Data Transformation Services folder and select Local Packages (Figure 5-17). Right-Click on the package listed under Name. A pull down menu will appear. Select Execute Package and Left-Click. Figure 5-16. DTS Local Packages 22 Figure 5-17. DTS Local Package Execution

Section 5. MCAPS 3000 Database: Saving to Remote SQL Server When the database transfer begins you should see the screen shown in Figure 5-18. The transfer may take several minutes depending on the traffic on the corporate WAN/LAN and the MCAPS 3000 database size. Upon completion, the screen shown in Figure 5-19 should appear. Click OK then Done. Figure 5-18. DTS Package Execution Figure 5-19. Successful Completion of DTS Package Execution 23

MCAPS 3000. Disaster Recovery Guide VERIFYING THE REMOTE DATABASE To verify the correct database with all the MCAPS 3000 database tables were copied, invoke SQL Enterprise Manager on the Backup MCAPS 3000 Host. Select the correct database, e.g., MCAPS 3000, and then click on Tables. The names and number of listed tables should be exactly the same as on the Primary MCAPS 3000 Host, particularly those listed User under the column Type. See Figure 5-20. Figure 5-20. Verifying Database 24

Section 5. MCAPS 3000 Database: Saving to Remote SQL Server MODIFYING THE DTS SCHEDULE To modify the DTS schedule, invoke SQL Enterprise Manager and traverse down to the Data Transformation Services Local Packages. Right-Click on the package listed under Name. A pull down menu will appear. Select Schedule Package and Left-Click. A schedule screen will appear, as shown in Figure 5-21. Make the necessary changes and click OK. Figure 5-21. Modifying DTS Schedule 25

MCAPS 3000. Disaster Recovery Guide 26

SECTION 6. MCAPS 3000 DATABASE: SAVING TO NETWORKED STORAGE This section describes the method of saving the MCAPS 3000 database from Primary host to Backup host via one or more intermediate steps. It assumes that the Backup host is not accessible from the Primary host on a SQL Server peer-to-peer basis. Given that the database cannot be saved directly to another SQL Server, it needs to be translated to another format for storage. The translated database can then be forwarded to a site that is accessible by the Remote host, which will import the database into the SQL Server. The DTS utility is used for both ends of the operation: conversion and export of Data Source in the format of Microsoft OLE DB Provider for SQL Server to some other format e.g., Microsoft Access Database, and conversion and import to the Backup SQL Server. In between, there may be one or more intermediate transfer steps via other file transmission tools, such as FTP. This process is schematically shown on the next page. To setup DTS for this method, again, as described in Section 5 Invoke SQL Server Enterprise Manager and continue to the point where DTS Wizard requests information about Destination. At this step instead of choosing the default Microsoft OLE DB Provider for SQL Server, click on the pull down arrow to display the available database formats and select Microsoft Access 13 as shown in Figures 6-1, 6-2, and 6-3. 13 Any other intermediate database format can be used. However, Microsoft formats are preferable to minimize any possibility of conversion errors or data corruption. 27

MCAPS 3000. Disaster Recovery Guide Primary MCAPS 3000 Host Microsoft OLE DB Provider for SQL Server SQL Server DTS Microsoft Access Database (Copy to Networked storage, FTP ) Microsoft Access Database SQL Server DTS Backup MCAPS 3000 Host Microsoft OLE DB Provider for SQL Server Figure 6-1. Select Microsoft Access Sequence 28

Section 6. MCAPS 3000 Database: Saving to Networked Storage Figure 6-2. DTS Package Execution (1 Destination) Figure 6-3. DTS Package Execution (2 Destination) 29

MCAPS 3000. Disaster Recovery Guide Click Next. A screen will appear for selecting the Access Database 14 on the destination networked storage device. Click on [ ] to the right of the File name and navigate to the site where the previously created Access Database file is stored. See Figures 6-4 and 6-5. Figure 6-4. DTS Package Execution (3 Destination) Figure 6-5. DTS Package Execution (Select File) 14 The Access Database should have been previously created. The database will be empty, but DTS requires a file name. 30

Section 6. MCAPS 3000 Database: Saving to Networked Storage Find the file, highlight it, click Save, then click Next. See Figures 6-6 and 6-7. Figure 6-6. DTS Package Execution (Save File 1) Figure 6.7. DTS Package Execution (Save File 2) 31

MCAPS 3000. Disaster Recovery Guide Select Copy table(s) and view(s) then click Next. See Figure 6-8. Figure 6.8. Select Table Copy or Query 32

Section 6. MCAPS 3000 Database: Saving to Networked Storage Upon clicking Next the screen for selecting source tables and views will appear. See Figure 6-9. Click Select All (see Figure 6-10), then click Next. Figure 6-9. Select Source Tables and Views Figure 6-10. Select Source Tables and Views (After Select All) 33

MCAPS 3000. Disaster Recovery Guide After you press Next, the DTS schedule screen will appear. Repeat the steps as shown in the previous section, and then press Next. After this step, the screen in Figure 6-11 will appear. Figure 6-11. Save DTS Package Press Done. The DTS package is saved. Next execute the package to verify its operation as detailed in the previous section. When the package execution completes, the screen in Figure 6-12 will appear. 34 Figure 6-12. Execution Complete

Section 6. MCAPS 3000 Database: Saving to Networked Storage As the final step, verify that there was no data corruption during the translation and transfer process. Open the saved Access Database and ensure that all the tables from MCAPS 3000 database were copied, as shown in Figure 6-13. Figure 6-13. MCAPS 3000 Database in Microsoft Access DB Format CONFIGURING DTS ON THE BACKUP HOST To complete the link of automatically updating the MCAPS 3000 database on the Backup host, the DTS there should be configured as well, except the data translation is from Microsoft Access format to Microsoft OLE DB Provider for SQL Server, i.e., Data Source Microsoft Access Destination Microsoft OLE DB Provider for SQL Server 35

MCAPS 3000. Disaster Recovery Guide After the DTS package in the Backup host is configured 15, execute the package and ensure the MCAPS 3000 database in the Remote SQL Server contains all the tables as they appear in the Primary SQL Server. 15 It is strongly recommended that the Remote DTS package execution start time be several hours after the execution start time of the Primary DTS package. 36

SECTION 7. DISASTER RECOVERY DRILLS An important component of setting up and maintaining a successful DR program is scheduling and conducting periodic drills to test and verify the Backup Host s operation. The following are recommended steps: all or parts of the list can be implemented as appropriate. The details of the drills will necessarily vary depending on the state of the Backup Host: is it always UP, is its IP address the same as the Primary Host s, etc. DESIGNATED DISASTER RECOVERY TERMINALS At least one - preferably several - CAPS terminal(s) should be designated as DR terminal(s). The Host IP address of the IP interface of this terminal(s) should be set to the address of Backup MCAPS 3000 host. Regularly, on-line transactions should be initiated at the designated terminal(s) to ensure the Backup host is up and able to perform CAPS transactions. MCAPS 3000 CONSOLE OPERATIONS At regular intervals, an MCAPS 3000 Console should be connected to the Backup Host and MCAPS 3000 Console operations performed. Database inquiry operations are benign and can be performed freely. Any database change operations should be done with care: One or more Authorized Agents can be added and then immediately deleted. 37

MCAPS 3000. Disaster Recovery Guide 38

SECTION 8. LOCAL DATABASE BACK UP AND RECOVER MCAPS 3000 does not provide explicit disaster recovery 16 ; therefore, it is important to schedule regular Backups and Maintenance of data. This section describes how to perform backups using the MS SQL Database Maintenance Plan Wizard. BACK UP/RECOVER MICROSOFT SQL DATABASE On the PC workstation, access the Database Maintenance Planner as follows: Click SQL Server Enterprise Manager, then click Tools, and then click Database Maintenance Planner as indicated in Figure 8-1. Figure 8-1. Accessing the Database Maintenance Planner 16 This section is reproduced from MCAPS 3000 OPERATION MANUAL PN 99875245. Although not strictly applicable to Disaster Recovery since the Database Maintenance Planner saves to and restores from local storage, it can augment the remote save/restore: saved DB tapes can be store at remote sites. 39

MCAPS 3000. Disaster Recovery Guide The welcome page will pop up, as shown in Figure 8-2. Click Next to begin using the wizard. Figure 8-2. Database Maintenance Plan Wizard Welcome Screen 40

Section 8. Local Database Backup and Recover First, select the database from which backups are to be performed. Generally, only the MCAPS 3000-01 17 database file needs backup as indicated in Figure 8-3. Figure 8-3. Select Databases Choose the These databases option and select the database(s) from which the backups will be performed, then click Next. If necessary, you can use the wizard to optimize/reorganize your archived data. Here we chose not to do anything. Click Next. 17 The name MCAPS 3000-01 may also appear as MCAPS 3000, MCAPS 30001, etc., depending on the name assigned to the MCAPS 3000 SQL Database during MCAPS 3000 Installation. 41

MCAPS 3000. Disaster Recovery Guide The options for improving the database are shown in Figure 8-4. Figure 8-4. Update Data Optimization Information 42

Section 8. Local Database Backup and Recover An integrity check on the database prior to archiving is recommended. The Database Integrity Check Screen is shown in Figure 8-5. Figure 8-5. Database Integrity Check Select the Check database integrity and choose Include indexes to verify index fields as well. (Chose Attempt to repair minor problems only if you are aware of what changes will be made.) Then choose Perform these checks before doing backups. Set the schedule for your backup by clicking on Change; then click Next to continue. 43

MCAPS 3000. Disaster Recovery Guide This next menu is used to schedule backup(s), as shown in Figure 8-6. Figure 8.6. Specify Database Backup Plan Select Back up the database as part of the maintenance plan to enable this menu. Optionally, select Verify the integrity of the backup when complete. (If you have scheduled to check the integrity of your data from the previous menu, then it is unlikely that you would need to perform this action again). Then select the media that you would like to store your data in (Tape or Disk). Set the schedule for your backup by clicking on Change; then click Next to continue. 44

Section 8. Local Database Backup and Recover Since disk was chosen as our backup media, it is necessary to specify where the data will be stored in the disk, as indicated in Figure 8-7. Figure 8.7. Specify Backup Disk Directory Chose a directory, and select Create a subdirectory for each database if the subdirectory does not exist. Input a three-character extension for your backup file in the field box for Backup file extension. (It would be best to choose an extension that has no association with any existing application; i.e., TXT or EXE ). Click Next to continue. 45

MCAPS 3000. Disaster Recovery Guide The Transaction Log contains activities performed to the database, including scheduled backups. It is a good idea to maintain a history of the transaction log for future reference. As indicated in Figure 8-8, check the box for Backup transaction log if you would like to back up your transactions log. Figure 8-8. Specify Transaction Log Backup Plan Optionally, select Verify the integrity of the backup when complete. Then select the media that you would like to store your data in (Tape or Disk). Set the schedule for your backup by clicking on Change; then click Next to continue. 46

Section 8. Local Database Backup and Recover Since we chose disk as our backup media, you will need to specify where the data will be stored in the disk. Figure 8-9 is the screen for Transaction Log Backup Disk Directory. Figure 8-9. Specify Transaction Log Backup Disk Directory Chose a directory and select Create a subdirectory for each database if the subdirectory does not exist. Input a three-character extension for your backup file in the field box for Backup file extension. (It would be best to choose an extension that has no association with any existing application; i.e., TXT or EXE ). Click Next to continue. 47

MCAPS 3000. Disaster Recovery Guide The Reports to Generate Screen, shown in Figure 8-10, stores reports generated by the maintenance plan. Figure 8-10. Reports To Generate These reports can be useful to maintain organization of backup data. To enable this function, select Write report to a text file in directory and chose a directory to store the file. Optionally the report can be sent as an email to a System Administrator or other responsible individual. Click Next to continue. 48

Section 8. Local Database Backup and Recover The Maintenance Plan History Screen, Figure 8-11, creates a database table of backup history. Figure 8-11. Maintenance Plan History This can allow a manual integrity check on the back-up files or as a reference to the backup history. To enable this function, select Write history to the msdb.dbo.sysdbmainplan_history table on this server. Then select Limit rows in the table to: and chose a number. Choose a number of significant size but not large enough to effect system performance. Click Next to continue. 49

MCAPS 3000. Disaster Recovery Guide As indicated in Figure 8-12, the database maintenance plan wizard configurations are done. Give your plan a name and verify your settings. If everything looks good, click Finish and your plan will become immediately effective. Reference Documents: Figure 8-12. Completing the Database Maintenance Plan Wizard For additional information on MS SQL visit www.microsoft.com. 50

HSM REDUNDANCY SECTION 9. HARDWARE SECURITY MODULE For a High Availability MCAPS 3000 Cluster, MagTek recommends at least 2 MCAPS 3000 HSMs for the Primary Host. It is also recommended that these 2 HSMs be physically situated in the same area as the Cluster, so they can be connected to the servers via a local hub. This ensures that any network problems will not delay or bring down the entire MCAPS 3000 system. If the HSMs are connected to Servers via WAN/LAN then any problems on those nets can severely affect the operation of the MCAPS 3000. For the Backup host, institutions may choose a reduced redundancy hardware configuration: 1 HSM may be sufficient. DES KEY MANAGEMENT Due to security requirements, HSM key loading is a manual process. Therefore, when new keys are loaded into the Primary HSMs, the process must be precisely duplicated at the Backup HSMs. Key check values must be verified in all HSMs to ensure they are the same for the given set of keys. 51

MCAPS 3000. Disaster Recovery Guide 52

APPENDIX A. MAINTAINING TWO PRIMARY HOSTS This section discusses a possible method of maintaining two Primary Hosts: a fully automatic failover can be envisioned. Since MagTek has not tested this configuration and has not fully analyzed all the technical issues associated with it, MagTek does not recommend this topology at this time. Although conceptually feasible, there are a multitude of practical technical issues that need to be considered and resolved. Nevertheless, this section is included in this document to initiate discussion and development of new ideas on this important subject. PRIMARY HOST - PRIMARY HOST CONFIGURATION A Cluster of Primary-Primary Hosts can be setup. In this configuration, both Hosts are Primary and process CAPS terminal transactions independent of each other. The databases are kept in synch via regular updates under DTS 18. CAPS Terminal s IP interface has the Host address of both listed, as described above. USER DATABASE UPDATES For simplicity, user database updates can be forced to only one of the Primary Hosts. Generally the population of Security Officers and System Managers is fairly static. Authorized Agents (AA) change more frequently. However, with scheduled synchronization of databases, AA record changes will be reflected in both databases in a within at most a few hours. TRANSACTION DATABASE Since a CAPS terminal can randomly connect to either one of the Hosts, there will transactions from the same terminal in both databases. PIN offsets uploaded to the Customer Account File (CAF) Host need to be passed through an intermediary processing stage to collate transaction data. CAPS TERMINAL INSTALLATION AND TSN After a CAPS terminal is loaded with application and institutions keys, it needs to be installed on the CAPS2000/MCAPS 3000 before it can be used. During the process of Installation, the CAPS Host assigns a unique Terminal Serial Number (TSN) to the unit: this number is kept in both the CAPS/MCAPS 3000 database and terminal internal non-volatile memory. 18 Future releases of Microsoft SQL Server DBMS may provide tools to facilitate MCAPS 3000 database synchronization further. SQL Server Yukon new feature of Database Mirroring will greatly facilitate automatic failover. 53

MCAPS 3000. Disaster Recovery Guide When two Primary Hosts are active, it is possible that two different terminals being installed during the time-window when the databases have not been synchronized, can be assigned the same TSN 19 : Terminal-1 connects to Host-1, and Terminal-2 connects to Host-2. This situation is quite undesirable and should be prevented. One approach might be to designate the Hosts as Even Host and Odd Host : terminals connecting to Even-Host for install are assigned even TSN numbers, and those connecting to Odd-Host are assigned odd numbers. 19 This is unlikely, given that the Terminal Identification Number (TIN) is one of the parameters used to derive TSN. TIN is assigned at the CAPS-TKIU loading station, and is generally unique, but is not guaranteed to be. 54

INDEX A Accessing the Data Transformation Services 9, 10, 11 Automatic Failover... 8 B Back up/recover Microsoft SQL Database. 39 Backup MCAPS Host... 7 C CAPS Terminal Installation And TSN... 53 CAPS Terminal Operation, Background... 7 Cluster... 3 Clustered hardware host... 3 Completing the DTS Wizard (1)... 19 Completing the DTS Wizard (2)... 20 Completing the DTS Wizard (3)... 20 Configuring DTS On The Backup Host... 35 D Data Transformation Services... 21 Database Maintenance Completing the Database Maintenance Plan Wizard... 50 Database Maintenance Database Integrity Check... 43 Database Maintenance Plan History... 49 Database Maintenance Plan Wizard Welcome Screen... 40 Database Maintenance Planner, Accessing the... 39 Database Maintenance Reports To Generate48 Database Maintenance Select Databases... 41 Database Maintenance Specify Backup Disk Directory... 45 Database Maintenance Specify Database Backup Plan... 44 Database Maintenance Specify Transaction Log Backup Disk Directory... 47 Database Maintenance Specify Transaction Log Backup Plan... 46 Database Maintenance Update Data Optimization Information... 42 DES Key Management... 51 Designated Disaster Recovery Terminals... 37 Dial-Up Terminals... 7 Disaster Recovery... 1 Disaster Recovery Drills... 37 Disaster Recovery Implementation... 5 Disaster Recovery, description... 3 DP center... 1 DTS Local Package Execution... 22 DTS Local Packages... 22 DTS Package Execution... 23, 30 DTS Package Execution (1 Destination).. 29 DTS Package Execution (2 Destination).. 29 DTS Package Execution (3 Destination).. 30 DTS Package Execution (Save File 1)... 31 DTS Package Execution (Save File 2)... 31 DTS Select Source Tables and Views... 33 DTS Select Table Copy or Query... 32 DTS Wizard Welcome Screen... 12 E Execution Complete... 34 F Foreword... 1 H Hardware redundancy... 3 Hardware Security Module... 51 Host IP Address, Unique... 7 Host Switchover/Failover... 5 HSM Redundancy... 51 L Local Database Back Up And Recover... 39 M Maintaining Two Primary Hosts... 53 Manual Failover... 8 MCAPS Console Operations... 37 MCAPS database... 3 MCAPS Database Saving to Networked Storage... 27 Saving To Remote SQL Server... 9 55

MCAPS 3000. Disaster Recovery Guide MCAPS Database in Microsoft Access DB Format... 35 MCAPS Database, Saving The... 5 MCAPS Host... 3 MCAPS Host, Primary And Backup... 5 MCAPS, description... 3 Microsoft s SQL Server DBMS... 1 Modifying The DTS Schedule... 25 O Overview... 3 P Primary Host... 3 Primary Host - Primary Host Configuration 53 Primary MCAPS Host... 7 R RAID 1 hard disks... 3 Related Documents... 4 S Save DTS Package... 34 Save DTS Package, Wizard... 18 Schedule DTS, Wizard... 17 Select Data Destination, Wizard... 14 Select Data Source,Wizard... 13 Select DTS Name and Description, Wizard 18 Select Microsoft Access Sequence... 28 Select Objects to Copy, Wizard... 16 Single Server... 3 Specify Table Copy, Wizard... 15 Successful Completion of DTS Package Execution... 23 T Testing The DTS Package... 21 Transaction Database... 53 U User Database Updates... 53 V Verifying The Remote Database... 24 56