Data Replication User s Manual (Disaster Recovery System Installation and Operation Guide)

Transcription

1 NEC Storage Software Data Replication User s Manual (Disaster Recovery System Installation and Operation Guide) IS027-18E

2 NEC Corporation No part of the contents of this book may be reproduced or transmitted in any form without permission of NEC Corporation. The contents of this book may be modified without notice in the future.

3 Preface This manual is bundled with NEC Storage RemoteDataReplication/DisasterRecovery to explain, through descriptions and examples, how Disaster Recovery functions are used for data replication and how disaster recovery systems are introduced and operated. This manual is intended for readers who already possess the basic knowledge about data replication that is essential for using Disaster Recovery functions to introduce and operate a disaster recovery system. For description of data replication, refer to the NEC Storage Software Data Replication User s Manual (Function Guide) (IS015), NEC Storage Software Data Replication User s Manual (Installation and Operation Guide) (IS016 or IS020) in accordance with your OS in use, and NEC Storage Software ControlCommand Command Reference (IS041). Refer to the NEC Storage Software Manual Guide (IS901) for a general description of NEC Storage and related manuals. Remarks 1. This manual explains functions implemented by the following program products: NEC Storage Manager and NEC Storage BaseProduct NEC Storage ControlCommand (Note) NEC Storage DynamicDataReplication NEC Storage RemoteDataReplication NEC Storage RemoteDataReplication/DisasterRecovery Note: NEC Storage ControlCommand is a program product that has integrated the following five functions. ReplicationControl SnapControl ReplicationControl/DisasterRecovery ProtectControl PowerControl 2. This manual is applicable to the program products of the following versions: NEC Storage Manager Ver6.2 NEC Storage BaseProduct Ver6.2 NEC Storage ControlCommand Ver6.2

4 3. The NEC Storage Manager is referred to as ism or Storage Manager in this manual unless clearly specified. Also, the following terms refer to the corresponding NEC Storage software products. Term NEC Storage Software Product AccessControl BaseProduct ControlCommand DynamicDataReplication or DDR PathManager PerformanceMonitor PerformanceNavigator RemoteDataReplication or RDR RemoteDataReplication/DisasterRecovery NEC Storage AccessControl NEC Storage BaseProduct NEC Storage ControlCommand NEC Storage DynamicDataReplication NEC Storage PathManager NEC Storage PerformanceMonitor NEC Storage PerformanceNavigator NEC Storage RemoteDataReplication NEC Storage RemoteDataReplication/DisasterRecovery 4. The NEC Storage series disk array subsystem is referred to as a disk array in this manual unless clearly specified. Also, the following terms refer to the corresponding NEC Storage hardware products. Term NEC Storage Hardware Product D3 series D4 series D8 series xxx series or xxxx series Sxxx or Sxxxx * xxx and xxxx represent the model number. NEC Storage D3 series NEC Storage D4 series NEC Storage D8 series NEC Storage xxx series or NEC Storage xxxx series NEC Storage Sxxx or NEC Storage Sxxxx 5. The following terms in this manual refer to the NEC Storage software manuals. Term NEC Storage Software Manual User s Manual (UNIX) User s Manual Configuration Setting Tool User s Manual (GUI) Messages Handbook Data Replication User s Manual (Function Guide) Data Replication User s Manual (Installation and Operation Guide) ControlCommand Command Reference Partitioning User s Manual PathManager User s Manual (Linux) 6. Trademarks and registered trademarks NEC Storage Software NEC Storage Manager User s Manual (UNIX) (IS001) NEC Storage Software NEC Storage Manager User s Manual (IS004) NEC Storage Software Configuration Setting Tool User s Manual (GUI) (IS007) NEC Storage Software Messages Handbook (IS010) NEC Storage Software Data Replication User s Manual (Function Guide) (IS015) NEC Storage Software Data Replication User s Manual (Installation and Operation Guide for Windows) (IS016) NEC Storage Software Data Replication User s Manual (Installation and Operation Guide for Linux) (IS020) NEC Storage Software ControlCommand Command Reference (IS041) NEC Storage Software Partitioning User s Manual (IS043) NEC Storage Software PathManager User s Manual (Linux) (IS202) Microsoft, Windows, Windows Server, and Windows Vista are trademarks or registered trademarks of Microsoft Corporation in the United States and other countries. HP-UX is a registered trademark of Hewlett-Packard Co. in the United States.

5 UNIX is a registered trademark of The Open Group in the United States and other countries. VERITAS, VxVM, VxFS, NetBackup, VERITAS Volume Manager, VERITAS File System, and VERITAS NetBackup are trademarks or registered trademarks of VERITAS Software Corporation in the United States and other countries. Solaris is a trademark or a registered trademark of Sun Microsystems, Inc. in the United States and other countries. Linux is a trademark or registered trademark of Mr. Linus Torvalds in the United States and other countries. Other product names and company names, etc. are trademarks or registered trademarks of the associated companies. 7. This product includes the OSSs below. For details on the licenses, refer to Appendix A in this manual. Apache log4j ( flex ( OpenSSL ( 8. In this document, the capacity is calculated based on units of 1024 (for example 1 KB = 1024 bytes) unless otherwise specified. 9. In this document, matters to which careful attention needs to be paid will be described as follows: Be sure to observe the instructions. If the indications are ignored and the system is improperly operated, settings which have been already made might be affected. Type of Indication Type Description Describes contents which require users to pay special attention for operation. 1st Edition in November th Edition in January 2010

6 Contents Chapter 1 Overview of Disaster Recovery What is Disaster Recovery? Disaster Recovery Use Format Copy Mode Options for Disaster Recovery Systems Freshness (Up to Date) of Backup Data and Use Format Layout and Configuration of Disaster Recovery System Placement of Sites Selection of Configuration and Lines...10 Chapter 2 Disaster Recovery Functions Atomic Group (ATgroup) ATgroup Configuration Display and Setting Functions ATgroup State Display Function ATgroup Replication Control Functions Copy of Order Guarantee Function Atomic-break Function Semi-Synchronous Copy Mode of Order Guarantee and Atomic Break Simultaneous Replication of Serially Configured Pair Remote Pair Operations RDR Quick Sync Volume Update Prevention Volume Comparing/Identity Certified Swap Function for RDR Pair Operation of Partitioning Function...31 Chapter 3 Install Software and Build Environment System Configuration Hardware Software Placement of Management Servers Placement of Control Volume Software Installation Installation of RemoteDataReplication/DisasterRecovery ReplicationControl/DisasterRecovery Installation and Environment Setting Building an ATgroup Pairs Registered to an ATgroup Steps for Building an ATgroup...43 Chapter 4 Software Operation Methods Storage Manager s ATgroup Function Operation Method (GUI) ATgroup Information Screen Displaying the ATgroup Configuration and State Creating an ATgroup Deleting an ATgroup Creating/Deleting an ATgroup Volume Renaming an ATgroup Changing the Allowed Response Time of MV Replicating an ATgroup Separating an ATgroup Restoring an ATgroup Forced Separation of an ATgroup Recovery from Fault Separation Forced Deletion of an ATgroup Outputting the ATgroup Information List in CSV format Saving an ATgroup and Registered Pair Information ReplicationControl/DisasterRecovery Command Operation Methods (CLI) Replicating an ATgroup (ismrc_replicate) i

7 4.2.2 Separating an ATgroup (ismrc_separate) Restoring an ATgroup (ismrc_restore) Recovery from Fault Separation (ismrc_change) Waiting for the ATgroup Sync State to Change (ismrc_wait) Displaying the ATgroup Copy State (ismrc_query) Displaying the ATgroup List (ismrc_sense) Building an ATgroup (ismrc_atg) Remote Operation Commands Chapter 5 Operation Examples Backup Line Fault Recovery Use of RV when Disaster Occurs Copy Back Mode from RV to MV Restoration for Copy Back Mode Copy Back Mode by Pair Configuration Substitutions Copy Back Mode through Swap for Single RDR Pair Notes on Copy Back Mode Procedure through Swap for Single RDR Pair Recovery Procedure at Pair Swap Failure Copy Back Mode through Pair Swap in RDR/DR Configuration Copy Back Mode Procedure through Swap for RDR Pair Recovery Procedure at Pair Swap Failure Chapter 6 Notes Specifications of ATgroup Notes on Building an Environment Notes for Operations Appendix A License A.1 Apache log4j License A.2 Flex License A.3 OpenSSL License Index ii

8 This page is intentionally left blank. iii

9 Chapter 1 Overview of Disaster Recovery Chapter 1 Overview of Disaster Recovery Given the increasingly important role of information systems in the context of business activities, disaster recovery support for such information systems are needed more than ever. RemoteDataReplication and DisasterRecovery provide functions that lay the foundation for disaster recovery. This chapter gives a general description of disaster recovery systems that use RemoteDataReplication and DisasterRecovery software What iis Diisaster Recovery? Disaster recovery is a solution that enables businesses to recover operations after a system fault has occurred due to a disaster, accident, etc. The types of disasters that disaster recovery is designed to anticipate include natural disasters such as earthquakes and floods as well as fires and other man-made disasters caused by human error. Main site Backup site Earthquake, flood, fire, etc. Recovery Figure 1-1 Disaster Recovery When constructing a disaster recovery system, the anticipated disasters and the system repair level differ according to the system s environment and requirements. When the scope of effects from anticipated disasters goes beyond a particular building, data must be stored at remote locations to enable configuration of a system that will take over the operations of the affected system. Typically, disaster recovery systems feature copying 1

10 Chapter 1 Overview of Disaster Recovery of data to remote sites as an essential component in anticipation of disasters whose effects transcend one building. RemoteDataReplication and DisasterRecovery provide various methods for copying data to remote sites. Table 1-1 Anticipated Disasters Scope of Disaster Wide area (in or between states, provinces, etc.) Specified districts (within a city) Specified building Specified system Cause of Disaster Earthquake, etc. Flood, power outage, etc. Fire, water damage, terrorist attacks, etc. Hardware fault Software fault Human error Virus, etc. Two indicators are used to determine the repair level of a system: the recovery point objective (RPO) and the recovery time objective (RTO). The RPO indicates from which point in time data will be recovered. In smaller systems, the RPO can recover data from a point just before the disaster occurred. A smaller RPO value is used when creating a disaster recovery system in which disaster-related data loss must be minimized. The RTO indicates the amount of time required to recover and resume operations. When this RTO value is low, recovery of operations following a disaster occurs more quickly, thus reducing the down time. A smaller RTO value is used when creating a disaster recovery system in which down time must be minimized. Generally, setting smaller RPO and RTO values entails higher costs and makes both the creation and the administration of the system more difficult. Consequently, both the RPO and RTO are determined based on the required repair level, then the type of disaster recovery system to create is selected in order to suit these RPO and RTO values. RemoteDataReplication and DisasterRecovery are software that supports the configuration of these various types of disaster recovery systems. 2

11 Chapter 1 Overview of Disaster Recovery RPO Now Several hours ago System in which data loss must be minimized Mission critical business system Yesterday System in which down time is allowed High repair level, high degree of difficulty, high costs Last month Client Next week Tomorrow Several hours later Mail server No down time RTO Recovery Point Objective: This indicates point in time from which system can be recovered Recovery Time Objective: This indicates amount of time until system operations can be resumed Figure 1-2 Relations among Recovery Point Objective, Recovery Time Objective, and Repair Level 3

12 Chapter 1 Overview of Disaster Recovery 1..2 Diisaster Recovery Use Format In disaster recovery systems, the use format is set by selecting options such as the repair point (the point in time from which data will be recovered), the maximum recovery time before data can be used again, the method used to copy data to remote sites (copy mode), and the type of synchronization (replication or separation) used while the system is operating. Copy mode options that can be selected for a disaster recovery system are described below, along with the freshness (up to date) of backup data and use formats Copy Mode Opttiions ffor Diisastter Recovery Systtems The disaster recovery system s copy mode options are synchronous mode, semi synchronous copy mode of order guarantee, and background copy mode. Different use formats can be implemented depending on the copy mode. The features of each copy mode are described below. 4

13 Chapter 1 Overview of Disaster Recovery Synchronous mode In this mode, business I/O operations are fully synchronized and copied to a remote site. Because copying is always synchronized with business I/O operations, the data being copied is always consistent and up to date. Completion of business I/O operations is held pending until the copy operation is completed, so the response to business I/O operations depends on the line speed. Main site Backup site Application server (iv) (i) (ii) (iii) Synchronization Master Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request to replication volume Main site s disk array Backup site s disk array (iii) Write request completion response to replication volume Backup site s disk array Main site s disk array (iv) Write request completion response to master volume Main site s disk array Application server Figure 1-3 Overview of Operations in Synchronous Mode 5

14 Chapter 1 Overview of Disaster Recovery Semi-synchronous copy mode of order guarantee Data is copied to a remote site without being synchronized with business I/O operations. Since data is copied to business I/O operations via the guaranteed sequence method, the data being copied is consistent. When a high-speed line is used, this has almost no impact on business I/O responses. Main site Backup site Application server (ii) (i) (iii) (1) (2) (iv) (1) (2) (3) Master Semi-synchronous guaranteed sequence (3) Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request completion response to master volume Main site s disk array Application server (iii) Write request to replication volume according to sequence of writing to master volume (Sequence is (1) (1), (2) (2), (3) (3) ) Main site s disk array Backup site s disk array (iv) Write request completion response according to sequence of writing to replication volume (Sequence is (1) (1), (2) (2), (3) (3)) Backup site s disk array Main site s disk array Figure 1-4 Overview of Operations in Semi-synchronous Copy Mode of Order Guarantee 6

15 Chapter 1 Overview of Disaster Recovery Background copy mode Copying is asynchronous in relation to business I/O. Since copying to a remote site is done in batches, there is no consistency in the data being copied. This has no impact on business I/O responses. Main site Backup site Application server (ii) (i) (iii) (iv) Background Master Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request completion response to master volume Main site s disk array Application server (iii) Write request to replication volume Main site s disk array Backup site s disk array (iv) Write request completion response to replication volume Backup site s disk array Main site s disk array Figure 1-5 Overview of Operations in Background Copy Mode Summary of copy mode features The following is a summary of each copy mode s features. Copy Mode Table1-2 Summary of Copy Mode Features Consistency of Data Freshness of Data Being Copied Business I/O Response Synchronous mode Consistent Same as master Depends on line speed Semi-synchronous copy mode of order guarantee Consistent Almost same as master Does not depend on line speed if high-speed line is used Background copy mode Not consistent Not the latest data Does not depend on line speed 7

16 Chapter 1 Overview of Disaster Recovery Freshness ((Up tto Datte)) off Backup Datta and Use Formatt The copy mode and sync state are selected based on how fresh (up to date) the backup data needs to be when used for recovery following a disaster, and this also helps determine the use format for operations. For example, if it is essential to have the latest data for recovery after a disaster, a high-speed line should be used to continually replicate in either synchronous mode or [semi synchronous copy mode of order guarantee], and the use format that copies consistent data with a guaranteed write order to a remote site should be selected. For cases where it is acceptable to recover using backup data after a disaster, the background copy mode that does not affect business I/O operations should be used and the use format that periodically copies data to a remote site via pause point backup should be selected. For cases where it is acceptable to simply store backup data to a remote site, the use format that periodically executes pause point backup to the remote site without placing a server at the backup site should be selected. Freshness of Backup Data (Up to Date) Recovery from replication during replicating Table 1-3 Freshness (Up to Date) of Backup Data and Use Format Copy Mode Synchronous mode Semi synchronous copy mode of order guarantee Use Format Synchronous Mode Continual replication using high-speed line Continual replication using high-speed line Repair Point Time Until Data Recovery Same as master Short (*1) Almost same as master Short (*1) Recovery from backup data Background copy mode Data is backed up at periodic pause points Backup point Long (*2) *1: Perform rollfoward, if necessary, when database has been configured. *2: Restoration from backup media is required. Rollback from log is required when database has been configured. 8

17 Chapter 1 Overview of Disaster Recovery 1..3 Layout and Confiiguratiion of Diisaster Recovery System Points to consider when placing and configuring a disaster recovery system are described below Pllacementt off Siittes In a disaster recovery system, the sites are placed at physically dispersed locations so that faults will not occur simultaneously at both the main site and backup site in the wake of any anticipated disaster. Anticipated Scope of Disaster Cause of Disaster Wide area (in or between states, provinces, etc.) Earthquake, etc. Table 1-4 Estimation of Distances between Sites (Examples) Specified districts (within a city) Flood, power outage, etc. Specified building Fire, water damage, terrorist attacks, etc. Scope of Disaster s Effects Halts economic activities in entire area Halts economic activities in specified area Halts economic activities in specified companies Distance between Sites Several hundred kilometers Several dozen kilometers Several hundred meters to several kilometers 9

18 Chapter 1 Overview of Disaster Recovery Sellecttiion off Conffiigurattiion and Liines To make sure that recovery is always possible in a disaster recovery system, hardware must be configured so that there are no single fault points whereby a single fault can stop the entire system s operations. This is why an independent, redundant configuration of servers, access paths, and power supplies are recommended. In a network that has a disaster recovery system, maintaining the quality of communication lines and redundancy is required. In particular, an independent and redundant configuration is recommended for the lines between the main site and backup site so that data can always be transferred to the backup site, even when a line fault has occurred. In addition, the line speed that is required varies according to the disaster recovery configuration and the amount of data updating that accompanies business tasks. Consequently, the required line speed must be estimated in advance. Terminal Terminal Terminal WAN Network equipment Network equipment Server Server Network equipment Network equipment Disk array Main site Disk array Backup site Figure 1-6 Network Configuration Example 10

19 Chapter 2 Disaster Recovery Functions Chapter 2 Disaster Recovery Functions This chapter describes the disaster recovery functions provided by RemoteDataReplication and RemoteDataReplication/DisasterRecovery Atomiic Group (ATgroup) Disaster recovery functions are able to create a group of volume pairs which require consistency. A group that has such consistency is called an Atomic Group (hereafter abbreviated as ATgroup). When several RDR pairs are registered to an ATgroup, data can be copied from the master volume (Master Volume: MV) to a replication volume (Replication Volume: RV) in the order in which each volume is updated. Preserving the order in which each volume is updated enables the consistency of data to be maintained in the RV to be used for recovery. For example, when using a database for business tasks, consistency must be maintained among the volumes that comprise the database. Accordingly, all pairs in the volumes that comprise a database are registered to ATgroups. When an ATgroup is created for each business task and each pair is registered, data consistency can be maintained in the RV for each business task. In addition since ATgroups can be used to manage the state of individual groups, it enables state displays and operations on a per-group basis, which means that the usability of RV data after a disaster can be determined on a per-group basis. Main site Backup site MV1 RV1 MV2 RV2 MV3 RV3 Disk array Disk array ATgroup * Consistency can be maintained among RV1, RV2, and RV3 that are included in an ATgroup. Figure 2-1 ATgroup 11

20 Chapter 2 Disaster Recovery Functions The following describes, in relation to ATgroups, (1) the ATgroup configuration display and setting functions, (2) the state display function, (3) replication control functions, (4) the copy of order guarantee function, and (5) the atomic-break function. The number of ATgroups that can be created, the number of pairs that can be registered to ATgroups, and the number of ATgroups to which replication can be executed in semi synchronous copy mode of order guarantee are limited. For further description of ATgroups, refer to 6.1 Specifications of ATgroup. 12

21 Chapter 2 Disaster Recovery Functions ATgroup Conffiigurattiion Diispllay and Settttiing Functtiions The elements that configure the ATgroup can be displayed and set. Any ATgroup name can be set for an ATgroup, and these names can be used to distinguish among the ATgroups. The MV side of ATgroups in a database group is called a concentrator, and the RV side is called the distributor. The disk array on the concentrator side (MV side) is used to create an ATgroup, which is built by registering RDR pairs. Main site Backup site DB_Group1 MV1 RV1 MV2 RV2 Concentrator Distributor DB_Group2 MV3 RV3 MV4 RV4 Concentrator Distributor Disk array Disk array MV1-RV1, MV2-RV2: ATgroup name of pair included in DB_Group1 s ATgroup MV3-RV3, MV4-RV4: ATgroup name of pair included in DB_Group2 s ATgroup Figure 2-2 Building of ATgroup (Example) 13

22 Chapter 2 Disaster Recovery Functions Any pair that is registered to an ATgroup must meet the following conditions. DDR pairs cannot be registered to an ATgroup. Only RDR pairs can be registered to an ATgroup. A pair that is not the top-level pair cannot be registered to an ATgroup. Only the top-level pair can be registered. If the MV of the pair to be registered includes several RDR pairs, it cannot be registered to an ATgroup. Each RDR pair must be set as a single pair registered to an ATgroup. A registered pair in an ATgroup and the pair to be registered to an ATgroup must be separated. An RDR pair that exists among different disk arrays cannot be registered to a single ATgroup. The RDR pair must be in the same disk array. If the RV is also used as a snapshot, it cannot be registered. It is required that the RV is not used as a snapshot. Pairs that can be registered to ATgroup are illustrated below. Main site Backup site (i) MV RV (ii) MV RV/MV RV RV Disk array Disk array [Description of pairs that can be registered to ATgroup] (i) Single RDR pairs are registered. (ii) A combination of the top-level single RDR pair and DDR pair can be registered as an RDR pair. Figure 2-3 Pairs that can be Registered to ATgroup 14

23 Chapter 2 Disaster Recovery Functions Pairs that cannot be registered to an ATgroup are illustrated below. Main site Backup site (1) Backup site (2) (i) MV RV MV (ii) RV/MV RV (iii) MV RV RV (iv) MV Replicate RV MV RV (v) MV RV (vi) MV RV/BV SV Disk array Disk array Disk array [Description of pairs that cannot be registered to an ATgroup] (i) Pair configured as a DDR pair. (ii) Pairs that is not the top-level pair. (iii) Several RDR pairs that are configured from a single MV. (iv) Pairs in an ATgroup that are not separate from pairs registered to an ATgroup. (v) Pairs in different disk arrays that are in a single ATgroup. (Each of these pairs can be registered to a separate ATgroup) (vi) RV is also used as a snapshot. Figure 2-4 Pairs that cannot be Registered to ATgroup 15

24 Chapter 2 Disaster Recovery Functions ATgroup Sttatte Diispllay Functtiion The state of each pair in an ATgroup can be displayed per ATgroup. ATgroup states are displayed as ATgroup consistency states and ATgroup sync states. ATgroup consistency states The ATgroup consistency state indicates whether or not the data is consistent in all RVs included in the ATgroup. When the ATgroup consistency state is Atomic the data in the ATgroup s RVs is consistent and can be used to recover business operations. When the ATgroup consistency state is Non-atomic the RV data included in that ATgroup is not consistent data except when separation has been completed at a pause point in business operations. Thus, it is not normally used for recovery of business operations. ATgroup Consistency State Atomic Non-atomic Invalid Table 2-1 ATgroup Consistency States Description RV data is consistent as an ATgroup. RV can be accessed when ATgroup s sync state is separated. RV data is not consistent as an ATgroup. This state indicates that the ATgroup does not contain any pairs or has just been created. 16

25 Chapter 2 Disaster Recovery Functions ATgroup sync states These indicate the sync state of pairs in an ATgroup. The sync state of an ATgroup changes according to the sync state of each pair in the ATgroup. ATgroup Sync State Separated Separating (Sep/exec) Separate start (Sep/start) Fault separation (Fault) Fault recovering (Fault/recovering) Replicate synchronous (Rpl/sync) Replicating (Rpl/exec) Replicate start (Rpl/start) Replicate suspend (Rpl/suspend) Restoring (Rst/exec) Restore suspend (Rst/suspend) Invalid Table 2-2 ATgroup Sync States Description This state indicates that all pairs in the ATgroup have been separated. The state of pairs in the ATgroup is either separating or separated. In this state, separation has started for the ATgroup. This state indicates that the ATgroup contains a pair with a fault or a separated state caused by a forced operation. When the ATgroup consistency state is Atomic, RVs cannot be accessed. An operation that recovers from fault separation changes the ATgroup sync state to separated and enables RV access. This state indicates that the ATgroup is changing from fault separation to separated. This state indicates that all pairs in the ATgroup are synchronous due to replication. Pairs in the ATgroup are either being replicated or are synchronous. This state indicates that replication has started for an ATgroup. This state indicates that an abnormal suspend state has occurred for a pair in the ATgroup due to a fault while replicating. This state indicates that restoring will be or is being performed for an ATgroup. This state indicates that an abnormal suspend state has occurred for a pair in the ATgroup due to a fault while restoring. This state indicates that the ATgroup does not contain any pairs or has just been created. * The abbreviations have initial capital letters and are used to distinguish among the sync states of pairs. 17

26 Chapter 2 Disaster Recovery Functions ATgroup sync state when link fault occurs The ATgroup sync state is determined based on the state of the copy source, i.e., the concentrator side (MV side). If a link fault occurs, possibly due to a disk array fault or line fault, communication becomes disabled between the disk array on the concentrator side (MV side) and the disk array on the distributor side (RV side), so the state of the distributor side (RV side) immediately before the disaster occurred is maintained. Consequently, the ATgroup sync state of the concentrator side (MV side) does not match the ATgroup sync state of the distributor side (RV side). Therefore, the link state must be checked in order to confirm the ATgroup sync state. Line fault occurs during replicate synchronous state (Rpl/sync) Main site Backup site MV1 RV1 MV2 RV2 MV3 RV3 Disk array Disk array Access ATgroup sync state on concentrator side (MV side) State changes to fault separation (Fault) due to line fault Access ATgroup sync state on distributor side (RV side) Since state of concentrator side (MV side) cannot be obtained due to a line fault, the replicate synchronous state (Rpl/sync) before the fault occurred is maintained. Figure 2-5 ATgroup Sync State when Link Fault has Occurred 18

27 Chapter 2 Disaster Recovery Functions ATgroup Replliicattiion Conttroll Functtiions Replication control for pairs registered to ATgroups is performed for the entire ATgroup. Such control operations cannot be performed for specific pairs. ATgroup replication Replication is executed for all pairs within the ATgroup. Immediately after replication of the ATgroup, the ATgroup sync state becomes replicate start (Rpl/start). When replication is being executed for all pairs within the ATgroup, the ATgroup sync state is replicating (Rpl/exec). After all pairs in the ATgroup have been copied, the ATgroup sync state changes to replicate synchronous (Rpl/sync) as soon as the activity state has been replicated and the sync state is synchronous. The ATgroup s copy mode is specified when the ATgroup is being replicated. If ATgroup replication is performed during synchronous mode or semi synchronous copy mode of order guarantee, the ATgroup consistency state becomes Atomic as soon as the ATgroup sync state becomes replicate synchronous (Rpl/sync). If ATgroup replication is performed during a background copy mode, the ATgroup consistency state stays Non-atomic if the ATgroup sync state becomes replicate synchronous (Rpl/sync). During the replicate start (Rpl/start) or replicating (Rpl/exec) state, the RV data is not consistent, regardless of the copy mode. Accordingly, before replicating the ATgroup, the data should be backed up to another volume or other operations or steps should be implemented to prevent loss of data consistency. Separation of ATgroup Pairs in the ATgroup are separated. If the ATgroup consistency state is Non-atomic, separation of the ATgroup is performed specifically for each pair in the ATgroup. As soon as separation of the ATgroup is completed, the ATgroup sync state becomes separate start (Sep/start). Once separation is being executed for all pairs in the ATgroup, the ATgroup sync state becomes separating (Sep/exec). When all pairs in the ATgroup are separate, the ATgroup sync state becomes separated. At that time, the ATgroup consistency state becomes Non-atomic. If ATgroup separation is executed when the ATgroup consistency state is Atomic, the data consistency of RVs corresponding to all pairs in the ATgroup is maintained and separation is performed for all pairs at the same time. At that point, the ATgroup s consistency state is maintained as Atomic. Immediately after ATgroup separation, separate (immediate) cannot be used, which means the immediate separation of RVs. 19

28 Chapter 2 Disaster Recovery Functions Restoration of ATgroup Restoration is performed for each pair in the ATgroup. In order to maintain the RV data, this ATgroup restoration processing is only an RV store (protect) operation that does not include any updating of RVs. When an ATgroup is being restored, each pair in the ATgroup is restored and the ATgroup sync state changes to restoring (Rst/exec). When restoration of the ATgroup is completed, the ATgroup sync state becomes separated. Accordingly, if ATgroup restoration is performed when the ATgroup consistency state is Atomic, the RV data is maintained while also maintaining Atomic as the ATgroup consistency state. ATgroup replication control and state transitions Various types of ATgroup replication control and state changes are outlined below. Non-atomic RV protect restore execution state Atomic RV protect restore execution state Restoring Restoring Separated state Restoring Restored Separated while consistent Restoring Restored Separated Separated Replicating Separated due to fault. RV access disabled Recovery from fault separation Replicating Separating Fault separation When differences between MV and RV are reflected Replicating Synchronized via sync mode or semi synchronous copy mode of order guarantee Replication state Atomic break or forced separation Separating Synchronized via background copy mode Replication state Replication state during background copy mode Replication during sync mode or semi synchronous copy mode of order guarantee. Copy mode of order guarantee to RVs. Figure 2-6 Outline of Replication Control and State Transitions 20

29 Chapter 2 Disaster Recovery Functions Copy off Order Guaranttee Functtiion When the ATgroup consistency state is Atomic, the order of updating from MVs to RVs per ATgroup is guaranteed in order to maintain RV data consistency. Data is written from concentrator (MV side) to the distributor (RV side) according to the order in which data was written to the MVs that comprise the ATgroup. This guaranteeing of the update order occurs only when the copy mode is either synchronous mode or semi synchronous copy mode of order guarantee and the ATgroup consistency state is Atomic. No such guaranteeing of the update order can occur when the copy mode is background copy mode or when the ATgroup consistency state is Non-atomic. Main site Backup site Application server Standby server (i) (ii) (iii) DB_Group1 MV1 (iii) RV1 MV2 (ii) RV2 MV3 (i) RV3 ATgroup s consistency state is Atomic Disk array Disk array * (i), (ii), and (iii) indicate the write order Figure 2-7 Copy of Order Guarantee Function 21

30 Chapter 2 Disaster Recovery Functions Attomiic--break Functtiion The atomic-break function forcibly separates all pairs in the ATgroup and forcibly separates the RVs in order to maintain the consistency of RV data when the order of copying to RV cannot be guaranteed. This function is executed when the ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) but forced separation is required since the RV copy order cannot be guaranteed due to a disaster, line fault, or other fault problem. If data is written far beyond the line capacity, data copying performed in synchronization with RV is delayed. If this delay also causes substantial delay in MV writing, an atomic break is generated to stop copying to the replication volume in order to give priority to writing to MV. Execution of an atomic-break stops all copying to RVs, but the consistency of the RV data is maintained. After RVs are forcibly separated by an atomic-break, the option of enabling access to MVs can be specified as enabled or disabled during ATgroup replication. This MV Access after Atomic-break setting is described below. MV Access Continue (default) MV access can be continued even after an atomic-break has been executed. When an atomic-break is executed after a fault that does not affect business tasks (such as a line fault or RV fault), it does not affect business task access either. Since updating of MVs is enabled even after an atomic-break is executed, differences occur between the MV data and RV data after an atomic-break. For example, if a disaster causes a line fault to occur, a fault may later occur in the application server, which would result in loss of data updated after the line fault occurred. When an atomic-break is executed without a link fault such as an RV fault, the ATgroup sync state becomes fault separation (Fault). When changing from this state to RV use, an operation to recover from fault separation is executed on the distributor side (RV side) to enable RV use. Also, when an atomic-break is executed following a link fault such as a line fault, the distributor side (RV side) s ATgroup sync state remains replicate synchronous (Rpl/sync), as it was before the link fault occurred. When changing from this state to RV use, forced separation is executed on the distributor side (RV side), followed by an operation to recover from fault separation in order to enable RV use. MV Access Stop When an atomic-break is executed, MV access is stopped. In fact, execution of an atomic-break stops all business access, even when the fault that has occurred is a line fault or RV-related fault that does not affect business tasks. Since updating of MVs is not possible after an atomic-break, no differences will occur between MV data and RV data after the atomic-break is executed. For example, even if a disaster causes a line fault to occur first and an application server fault to occur later, no data loss will occur as long as the copy 22

31 Chapter 2 Disaster Recovery Functions mode is synchronous mode. To enable RV use after executing an atomic-break, a forced separation is first performed on the distributor side (RV side) to change to fault separation (Fault). Next, an operation to recover from fault separation is performed to enable RV use. When an MV access stop is specified, business tasks may be stopped without prior warning, so MV access stop should not be used ordinarily. 23

32 Chapter 2 Disaster Recovery Functions Semii--Synchronous Copy Mode off Order Guaranttee and Attomiic Break Semi-synchronous copy mode of order guarantee temporarily retains data in a buffer to copy write data to the master volume to a replication volume. However, if a large amount of data are written beyond the line capacity in a short time, free space in the buffer may become insufficient because disaster recovery cannot catch up with buffer data copying to the replication volume. In this case, even in semi-synchronous copy mode of order guarantee, data write to the subsequent master volume is delayed because disaster recovery waits for buffer vacancy. If this write delay to MV continues for a given length of time, an atomic break occurs to stop data copying to the replication volume so that priority is given to data write to the master volume. Some disk array models can adjust maintenance of replication volume copying and write priority to the master volume by setting the delay time (referred to as an allowed response time of MV) for judging this atomic break occurrence per ATgroup in semi-synchronous copy mode of order guarantee. To make this adjustment, use the performance monitoring function that is an option. Environment design 1. If MV write capacity is greater than line transfer capacity at the business peak of copying, a buffer write overflow occurs to delay the write response because the transfer capacity is exceeded. To avoid this delay, it is prerequisite to estimate the necessary line transfer capacity and prepare lines. 2. Determine the time (number of seconds, 18 seconds or less) allowed as the delay of the MV write I/O response time from the business requirements. Note, however, that if a value less than 10 seconds is specified as the allowance time, an atomic break may be determined to have occurred even if a link single-system failure is recovered via link switching. This is because disaster recovery cannot catch up with this failure recovery. Operation evaluation At the evaluation stage, perform measurement and analysis to check if estimation is correct in the same environment as the production environment. 3. Set the following statistical information and threshold using the performance monitoring function: Accumulation of maximum buffer usage on the log (accumulation by default) Accumulation of buffer wait time on the log (addition as LD information, maximum value/mean value) (accumulation by default) Setting of time (1 second) during which the threshold of the buffer wait time (maximum value) is treated as MV write delay occurrence Also set the allowed response time of MV for atomic break occurrence. 4. Check atomic break occurrence and the threshold reporting condition by performing the job simultaneously while copying data. 24

33 Chapter 2 Disaster Recovery Functions Analysis and countermeasures 5. Check if copy processing is enough for business write from the atomic break conditions, threshold excess report, and gathered performance statistical information. If no atomic break has occurred and the buffer wait time is not exceeded, the line capacity is enough. Check and store the vacancy condition of the maximum buffer usage as the original vacancy condition. The timing of atomic break operation when a copy error occurs serves as the allowed response time of MV. If no atomic break has occurred but the buffer wait time threshold is exceeded, judge whether to continue processing as is or whether to take appropriate countermeasures for the delay. If an atomic break has occurred, appropriate countermeasures are required because the line capacity is insufficient. 6. Select a method that meets the requirements from the following methods: Method 1: Enhance the line capacity to increase the transfer capacity. Method 2: If the line capacity is insufficient only in a specific time zone, consider to avoid the MV delay by performing separate operation in that time zone. Method 3: Consider memory purchase to increase the semi-synchronous buffer size. Method 4: Review the allowance time as the MV write I/O response. Production operation 7. Even in the production operation, the monitoring objects are the same as in the settings in step 3. However, the line capacity may become insufficient due to increase in the amount of business although it is sufficient at operation start. For this reason, monitor the portent and take the countermeasures on the preceding page as required. Use the threshold of the maximum buffer wait time as the threshold of the time when a write I/O delay begins to occur due to a buffer overflow. If the buffer has not still overflowed but the buffer usage is gradually increasing as compared to statistical information for the maximum buffer usage, you can reference this statistical information to determine if it is necessary to take appropriate countermeasures for the future buffer overflow. 25

34 Chapter 2 Disaster Recovery Functions 2..2 Siimulltaneous Replliicatiion of Seriialllly Confiigured Paiir Remote data replication functions can be used to simultaneously replicate serially configured pairs. When replication is performed simultaneously, replication at the second and lower levels is enabled only when the copy mode is background copy mode. Also, replication control cannot be performed when copying is performed in different directions at the same time (such as when executing replication for the former and latter serially configured pairs), when the former and latter pair s activity state is separate, or when its sync state is separating. Main site Backup site ATgroup s sync state is replicate state or pair s sync state is synchronous (synchronous, semisynchronous, or background copy) MV1 RV1/MV2 Pair s sync state is synchronous (background copy) RV2 Disk array Disk array Combination of RDR + DDR pairs Main site Backup site (1) Backup site (2) ATgroup s sync state is replicate state or pair s sync state is synchronous (synchronous, semisynchronous, or background copy) Pair s sync state is synchronous (background copy) MV1 RV1/MV2 RV2 Disk array Disk array Disk array Combination of RDR + RDR pairs Figure 2-8 Example of Replication Implemented for Serially Configured Pairs 26

35 Chapter 2 Disaster Recovery Functions 2..3 Remote Paiir Operatiions Remote data replication functions are able to control replication from the main site to a DDR pair at the backup site. They are also able to control replication from an RDR pair at the backup site to a DDR pair at the main site. Using these functions, backup operations using the backup site s DDR can be controlled from the main site. Main site Backup site Application server Standby server Operation instruction MV RV/MV Replicate, separate, restore Replication of serially configured DDR pairs can be controlled from MV side RV MV Replication of serially configured RDR and DDR pairs can be controlled from RV side Operation instruction Replicate, separate, restore RV/MV Disk array Replicate separate restore RV Disk array Figure 2-9 Example of Remote Control 27

36 Chapter 2 Disaster Recovery Functions 2..4 RDR Quiick Sync The RDR quick sync is a function to eliminate differences from the RDR pair to establish the sync state without copying data from MV to RV when the MV and RV data in the RDR pair is already the same. For RDR operation, it is necessary to reflect all data in MV to RV through the RDR link line at startup, making the data state the same (replicate synchronous). If a low-speed line is used as the RDR link line in the configuration, it takes significant time to transfer a lot of data in batch. The RDR quick sync quickly realizes the sync state without putting load on the low-speed RDR link line by other means than the RDR link line (such as transfer using tape) when MV and RV are the same data. The RDR quick sync function is offered only for ACOS-4.only for ACOS-4. This version only supports the function to display the RDR quick sync state in the ism replication management Vollume Updatte Preventtiion To perform the RDR quick sync operation, the data in the MV and RV data in the RDR pair must be the same. To make the MV and RV data the same, execute RAW backup of the MV data to tape using the backup software, transfer it from the main site to the backup site, and execute RAW restore from tape to RV. The RDR quick sync function cannot be used when MV data is updated while tape is transferred to the main site to the backup site. To prevent this situation, use the volume update prevention function and protect the target volume from the operations that causes differences. Update Prevention State Update prevention (Prevent) Table 2-3 Volume Update Prevention State Description State where the volume update prevention function protects a volume from the operations that cause differences. - Since the access right to the target volume is Read Only, write operation from the server cannot be performed. - Replication cannot be performed when the target volume is RV. - Restore cannot be performed when the target volume is MV. - Snapshot restore cannot be performed when the target volume is BV. - The logical disk cannot be deleted from the target volume. - The target volume cannot be changed to the reserve attribute. 28

37 Chapter 2 Disaster Recovery Functions Vollume Compariing//IIdenttiitty Certtiiffiied Volume comparing verifies that the MV and RV data is identical on the bit level using the volume update prevention function. If MV and RV data is identical, MV and RV can be made the same by certifying the identity. The state where volumes were compared and the identity was certified is called the identity state. Identity State Not Certified Comparing Identical Certifying Certified Different Fault(Comparing) Fault(Certifying) Table 2-4 Identity State Description MV and RV have not been compared and the identity of them has not been certified. The state is not Certified. Not Certified is also displayed when MV is in the Prevent state and RV is not in the Prevent state. MV and RV are being compared. Volume comparing found that data of MV and RV were identical. The identity between MV and RV is being certified. The identity between MV and RV has been certified and there are no differences. Volume comparing found that data of MV and RV were not identical. A fault occurred while MV and RV were being compared. A fault occurred while the identity between MV and RV was being certified. 29