Data Replication User s Manual (Disaster Recovery System Installation and Operation Guide)

Transcription

1 NEC Storage Software Data Replication User s Manual (Disaster Recovery System Installation and Operation Guide) IS027-18E

2 NEC Corporation No part of the contents of this book may be reproduced or transmitted in any form without permission of NEC Corporation. The contents of this book may be modified without notice in the future.

3 Preface This manual is bundled with NEC Storage RemoteDataReplication/DisasterRecovery to explain, through descriptions and examples, how Disaster Recovery functions are used for data replication and how disaster recovery systems are introduced and operated. This manual is intended for readers who already possess the basic knowledge about data replication that is essential for using Disaster Recovery functions to introduce and operate a disaster recovery system. For description of data replication, refer to the NEC Storage Software Data Replication User s Manual (Function Guide) (IS015), NEC Storage Software Data Replication User s Manual (Installation and Operation Guide) (IS016 or IS020) in accordance with your OS in use, and NEC Storage Software ControlCommand Command Reference (IS041). Refer to the NEC Storage Software Manual Guide (IS901) for a general description of NEC Storage and related manuals. Remarks 1. This manual explains functions implemented by the following program products: NEC Storage Manager and NEC Storage BaseProduct NEC Storage ControlCommand (Note) NEC Storage DynamicDataReplication NEC Storage RemoteDataReplication NEC Storage RemoteDataReplication/DisasterRecovery Note: NEC Storage ControlCommand is a program product that has integrated the following five functions. ReplicationControl SnapControl ReplicationControl/DisasterRecovery ProtectControl PowerControl 2. This manual is applicable to the program products of the following versions: NEC Storage Manager Ver6.2 NEC Storage BaseProduct Ver6.2 NEC Storage ControlCommand Ver6.2

4 3. The NEC Storage Manager is referred to as ism or Storage Manager in this manual unless clearly specified. Also, the following terms refer to the corresponding NEC Storage software products. Term NEC Storage Software Product AccessControl BaseProduct ControlCommand DynamicDataReplication or DDR PathManager PerformanceMonitor PerformanceNavigator RemoteDataReplication or RDR RemoteDataReplication/DisasterRecovery NEC Storage AccessControl NEC Storage BaseProduct NEC Storage ControlCommand NEC Storage DynamicDataReplication NEC Storage PathManager NEC Storage PerformanceMonitor NEC Storage PerformanceNavigator NEC Storage RemoteDataReplication NEC Storage RemoteDataReplication/DisasterRecovery 4. The NEC Storage series disk array subsystem is referred to as a disk array in this manual unless clearly specified. Also, the following terms refer to the corresponding NEC Storage hardware products. Term NEC Storage Hardware Product D3 series D4 series D8 series xxx series or xxxx series Sxxx or Sxxxx * xxx and xxxx represent the model number. NEC Storage D3 series NEC Storage D4 series NEC Storage D8 series NEC Storage xxx series or NEC Storage xxxx series NEC Storage Sxxx or NEC Storage Sxxxx 5. The following terms in this manual refer to the NEC Storage software manuals. Term NEC Storage Software Manual User s Manual (UNIX) User s Manual Configuration Setting Tool User s Manual (GUI) Messages Handbook Data Replication User s Manual (Function Guide) Data Replication User s Manual (Installation and Operation Guide) ControlCommand Command Reference Partitioning User s Manual PathManager User s Manual (Linux) 6. Trademarks and registered trademarks NEC Storage Software NEC Storage Manager User s Manual (UNIX) (IS001) NEC Storage Software NEC Storage Manager User s Manual (IS004) NEC Storage Software Configuration Setting Tool User s Manual (GUI) (IS007) NEC Storage Software Messages Handbook (IS010) NEC Storage Software Data Replication User s Manual (Function Guide) (IS015) NEC Storage Software Data Replication User s Manual (Installation and Operation Guide for Windows) (IS016) NEC Storage Software Data Replication User s Manual (Installation and Operation Guide for Linux) (IS020) NEC Storage Software ControlCommand Command Reference (IS041) NEC Storage Software Partitioning User s Manual (IS043) NEC Storage Software PathManager User s Manual (Linux) (IS202) Microsoft, Windows, Windows Server, and Windows Vista are trademarks or registered trademarks of Microsoft Corporation in the United States and other countries. HP-UX is a registered trademark of Hewlett-Packard Co. in the United States.

5 UNIX is a registered trademark of The Open Group in the United States and other countries. VERITAS, VxVM, VxFS, NetBackup, VERITAS Volume Manager, VERITAS File System, and VERITAS NetBackup are trademarks or registered trademarks of VERITAS Software Corporation in the United States and other countries. Solaris is a trademark or a registered trademark of Sun Microsystems, Inc. in the United States and other countries. Linux is a trademark or registered trademark of Mr. Linus Torvalds in the United States and other countries. Other product names and company names, etc. are trademarks or registered trademarks of the associated companies. 7. This product includes the OSSs below. For details on the licenses, refer to Appendix A in this manual. Apache log4j ( flex ( OpenSSL ( 8. In this document, the capacity is calculated based on units of 1024 (for example 1 KB = 1024 bytes) unless otherwise specified. 9. In this document, matters to which careful attention needs to be paid will be described as follows: Be sure to observe the instructions. If the indications are ignored and the system is improperly operated, settings which have been already made might be affected. Type of Indication Type Description Describes contents which require users to pay special attention for operation. 1st Edition in November th Edition in January 2010

6 Contents Chapter 1 Overview of Disaster Recovery What is Disaster Recovery? Disaster Recovery Use Format Copy Mode Options for Disaster Recovery Systems Freshness (Up to Date) of Backup Data and Use Format Layout and Configuration of Disaster Recovery System Placement of Sites Selection of Configuration and Lines...10 Chapter 2 Disaster Recovery Functions Atomic Group (ATgroup) ATgroup Configuration Display and Setting Functions ATgroup State Display Function ATgroup Replication Control Functions Copy of Order Guarantee Function Atomic-break Function Semi-Synchronous Copy Mode of Order Guarantee and Atomic Break Simultaneous Replication of Serially Configured Pair Remote Pair Operations RDR Quick Sync Volume Update Prevention Volume Comparing/Identity Certified Swap Function for RDR Pair Operation of Partitioning Function...31 Chapter 3 Install Software and Build Environment System Configuration Hardware Software Placement of Management Servers Placement of Control Volume Software Installation Installation of RemoteDataReplication/DisasterRecovery ReplicationControl/DisasterRecovery Installation and Environment Setting Building an ATgroup Pairs Registered to an ATgroup Steps for Building an ATgroup...43 Chapter 4 Software Operation Methods Storage Manager s ATgroup Function Operation Method (GUI) ATgroup Information Screen Displaying the ATgroup Configuration and State Creating an ATgroup Deleting an ATgroup Creating/Deleting an ATgroup Volume Renaming an ATgroup Changing the Allowed Response Time of MV Replicating an ATgroup Separating an ATgroup Restoring an ATgroup Forced Separation of an ATgroup Recovery from Fault Separation Forced Deletion of an ATgroup Outputting the ATgroup Information List in CSV format Saving an ATgroup and Registered Pair Information ReplicationControl/DisasterRecovery Command Operation Methods (CLI) Replicating an ATgroup (ismrc_replicate) i

7 4.2.2 Separating an ATgroup (ismrc_separate) Restoring an ATgroup (ismrc_restore) Recovery from Fault Separation (ismrc_change) Waiting for the ATgroup Sync State to Change (ismrc_wait) Displaying the ATgroup Copy State (ismrc_query) Displaying the ATgroup List (ismrc_sense) Building an ATgroup (ismrc_atg) Remote Operation Commands Chapter 5 Operation Examples Backup Line Fault Recovery Use of RV when Disaster Occurs Copy Back Mode from RV to MV Restoration for Copy Back Mode Copy Back Mode by Pair Configuration Substitutions Copy Back Mode through Swap for Single RDR Pair Notes on Copy Back Mode Procedure through Swap for Single RDR Pair Recovery Procedure at Pair Swap Failure Copy Back Mode through Pair Swap in RDR/DR Configuration Copy Back Mode Procedure through Swap for RDR Pair Recovery Procedure at Pair Swap Failure Chapter 6 Notes Specifications of ATgroup Notes on Building an Environment Notes for Operations Appendix A License A.1 Apache log4j License A.2 Flex License A.3 OpenSSL License Index ii

8 This page is intentionally left blank. iii

9 Chapter 1 Overview of Disaster Recovery Chapter 1 Overview of Disaster Recovery Given the increasingly important role of information systems in the context of business activities, disaster recovery support for such information systems are needed more than ever. RemoteDataReplication and DisasterRecovery provide functions that lay the foundation for disaster recovery. This chapter gives a general description of disaster recovery systems that use RemoteDataReplication and DisasterRecovery software What iis Diisaster Recovery? Disaster recovery is a solution that enables businesses to recover operations after a system fault has occurred due to a disaster, accident, etc. The types of disasters that disaster recovery is designed to anticipate include natural disasters such as earthquakes and floods as well as fires and other man-made disasters caused by human error. Main site Backup site Earthquake, flood, fire, etc. Recovery Figure 1-1 Disaster Recovery When constructing a disaster recovery system, the anticipated disasters and the system repair level differ according to the system s environment and requirements. When the scope of effects from anticipated disasters goes beyond a particular building, data must be stored at remote locations to enable configuration of a system that will take over the operations of the affected system. Typically, disaster recovery systems feature copying 1

10 Chapter 1 Overview of Disaster Recovery of data to remote sites as an essential component in anticipation of disasters whose effects transcend one building. RemoteDataReplication and DisasterRecovery provide various methods for copying data to remote sites. Table 1-1 Anticipated Disasters Scope of Disaster Wide area (in or between states, provinces, etc.) Specified districts (within a city) Specified building Specified system Cause of Disaster Earthquake, etc. Flood, power outage, etc. Fire, water damage, terrorist attacks, etc. Hardware fault Software fault Human error Virus, etc. Two indicators are used to determine the repair level of a system: the recovery point objective (RPO) and the recovery time objective (RTO). The RPO indicates from which point in time data will be recovered. In smaller systems, the RPO can recover data from a point just before the disaster occurred. A smaller RPO value is used when creating a disaster recovery system in which disaster-related data loss must be minimized. The RTO indicates the amount of time required to recover and resume operations. When this RTO value is low, recovery of operations following a disaster occurs more quickly, thus reducing the down time. A smaller RTO value is used when creating a disaster recovery system in which down time must be minimized. Generally, setting smaller RPO and RTO values entails higher costs and makes both the creation and the administration of the system more difficult. Consequently, both the RPO and RTO are determined based on the required repair level, then the type of disaster recovery system to create is selected in order to suit these RPO and RTO values. RemoteDataReplication and DisasterRecovery are software that supports the configuration of these various types of disaster recovery systems. 2

11 Chapter 1 Overview of Disaster Recovery RPO Now Several hours ago System in which data loss must be minimized Mission critical business system Yesterday System in which down time is allowed High repair level, high degree of difficulty, high costs Last month Client Next week Tomorrow Several hours later Mail server No down time RTO Recovery Point Objective: This indicates point in time from which system can be recovered Recovery Time Objective: This indicates amount of time until system operations can be resumed Figure 1-2 Relations among Recovery Point Objective, Recovery Time Objective, and Repair Level 3

12 Chapter 1 Overview of Disaster Recovery 1..2 Diisaster Recovery Use Format In disaster recovery systems, the use format is set by selecting options such as the repair point (the point in time from which data will be recovered), the maximum recovery time before data can be used again, the method used to copy data to remote sites (copy mode), and the type of synchronization (replication or separation) used while the system is operating. Copy mode options that can be selected for a disaster recovery system are described below, along with the freshness (up to date) of backup data and use formats Copy Mode Opttiions ffor Diisastter Recovery Systtems The disaster recovery system s copy mode options are synchronous mode, semi synchronous copy mode of order guarantee, and background copy mode. Different use formats can be implemented depending on the copy mode. The features of each copy mode are described below. 4

13 Chapter 1 Overview of Disaster Recovery Synchronous mode In this mode, business I/O operations are fully synchronized and copied to a remote site. Because copying is always synchronized with business I/O operations, the data being copied is always consistent and up to date. Completion of business I/O operations is held pending until the copy operation is completed, so the response to business I/O operations depends on the line speed. Main site Backup site Application server (iv) (i) (ii) (iii) Synchronization Master Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request to replication volume Main site s disk array Backup site s disk array (iii) Write request completion response to replication volume Backup site s disk array Main site s disk array (iv) Write request completion response to master volume Main site s disk array Application server Figure 1-3 Overview of Operations in Synchronous Mode 5

14 Chapter 1 Overview of Disaster Recovery Semi-synchronous copy mode of order guarantee Data is copied to a remote site without being synchronized with business I/O operations. Since data is copied to business I/O operations via the guaranteed sequence method, the data being copied is consistent. When a high-speed line is used, this has almost no impact on business I/O responses. Main site Backup site Application server (ii) (i) (iii) (1) (2) (iv) (1) (2) (3) Master Semi-synchronous guaranteed sequence (3) Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request completion response to master volume Main site s disk array Application server (iii) Write request to replication volume according to sequence of writing to master volume (Sequence is (1) (1), (2) (2), (3) (3) ) Main site s disk array Backup site s disk array (iv) Write request completion response according to sequence of writing to replication volume (Sequence is (1) (1), (2) (2), (3) (3)) Backup site s disk array Main site s disk array Figure 1-4 Overview of Operations in Semi-synchronous Copy Mode of Order Guarantee 6

15 Chapter 1 Overview of Disaster Recovery Background copy mode Copying is asynchronous in relation to business I/O. Since copying to a remote site is done in batches, there is no consistency in the data being copied. This has no impact on business I/O responses. Main site Backup site Application server (ii) (i) (iii) (iv) Background Master Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request completion response to master volume Main site s disk array Application server (iii) Write request to replication volume Main site s disk array Backup site s disk array (iv) Write request completion response to replication volume Backup site s disk array Main site s disk array Figure 1-5 Overview of Operations in Background Copy Mode Summary of copy mode features The following is a summary of each copy mode s features. Copy Mode Table1-2 Summary of Copy Mode Features Consistency of Data Freshness of Data Being Copied Business I/O Response Synchronous mode Consistent Same as master Depends on line speed Semi-synchronous copy mode of order guarantee Consistent Almost same as master Does not depend on line speed if high-speed line is used Background copy mode Not consistent Not the latest data Does not depend on line speed 7

16 Chapter 1 Overview of Disaster Recovery Freshness ((Up tto Datte)) off Backup Datta and Use Formatt The copy mode and sync state are selected based on how fresh (up to date) the backup data needs to be when used for recovery following a disaster, and this also helps determine the use format for operations. For example, if it is essential to have the latest data for recovery after a disaster, a high-speed line should be used to continually replicate in either synchronous mode or [semi synchronous copy mode of order guarantee], and the use format that copies consistent data with a guaranteed write order to a remote site should be selected. For cases where it is acceptable to recover using backup data after a disaster, the background copy mode that does not affect business I/O operations should be used and the use format that periodically copies data to a remote site via pause point backup should be selected. For cases where it is acceptable to simply store backup data to a remote site, the use format that periodically executes pause point backup to the remote site without placing a server at the backup site should be selected. Freshness of Backup Data (Up to Date) Recovery from replication during replicating Table 1-3 Freshness (Up to Date) of Backup Data and Use Format Copy Mode Synchronous mode Semi synchronous copy mode of order guarantee Use Format Synchronous Mode Continual replication using high-speed line Continual replication using high-speed line Repair Point Time Until Data Recovery Same as master Short (*1) Almost same as master Short (*1) Recovery from backup data Background copy mode Data is backed up at periodic pause points Backup point Long (*2) *1: Perform rollfoward, if necessary, when database has been configured. *2: Restoration from backup media is required. Rollback from log is required when database has been configured. 8

17 Chapter 1 Overview of Disaster Recovery 1..3 Layout and Confiiguratiion of Diisaster Recovery System Points to consider when placing and configuring a disaster recovery system are described below Pllacementt off Siittes In a disaster recovery system, the sites are placed at physically dispersed locations so that faults will not occur simultaneously at both the main site and backup site in the wake of any anticipated disaster. Anticipated Scope of Disaster Cause of Disaster Wide area (in or between states, provinces, etc.) Earthquake, etc. Table 1-4 Estimation of Distances between Sites (Examples) Specified districts (within a city) Flood, power outage, etc. Specified building Fire, water damage, terrorist attacks, etc. Scope of Disaster s Effects Halts economic activities in entire area Halts economic activities in specified area Halts economic activities in specified companies Distance between Sites Several hundred kilometers Several dozen kilometers Several hundred meters to several kilometers 9

18 Chapter 1 Overview of Disaster Recovery Sellecttiion off Conffiigurattiion and Liines To make sure that recovery is always possible in a disaster recovery system, hardware must be configured so that there are no single fault points whereby a single fault can stop the entire system s operations. This is why an independent, redundant configuration of servers, access paths, and power supplies are recommended. In a network that has a disaster recovery system, maintaining the quality of communication lines and redundancy is required. In particular, an independent and redundant configuration is recommended for the lines between the main site and backup site so that data can always be transferred to the backup site, even when a line fault has occurred. In addition, the line speed that is required varies according to the disaster recovery configuration and the amount of data updating that accompanies business tasks. Consequently, the required line speed must be estimated in advance. Terminal Terminal Terminal WAN Network equipment Network equipment Server Server Network equipment Network equipment Disk array Main site Disk array Backup site Figure 1-6 Network Configuration Example 10

19 Chapter 2 Disaster Recovery Functions Chapter 2 Disaster Recovery Functions This chapter describes the disaster recovery functions provided by RemoteDataReplication and RemoteDataReplication/DisasterRecovery Atomiic Group (ATgroup) Disaster recovery functions are able to create a group of volume pairs which require consistency. A group that has such consistency is called an Atomic Group (hereafter abbreviated as ATgroup). When several RDR pairs are registered to an ATgroup, data can be copied from the master volume (Master Volume: MV) to a replication volume (Replication Volume: RV) in the order in which each volume is updated. Preserving the order in which each volume is updated enables the consistency of data to be maintained in the RV to be used for recovery. For example, when using a database for business tasks, consistency must be maintained among the volumes that comprise the database. Accordingly, all pairs in the volumes that comprise a database are registered to ATgroups. When an ATgroup is created for each business task and each pair is registered, data consistency can be maintained in the RV for each business task. In addition since ATgroups can be used to manage the state of individual groups, it enables state displays and operations on a per-group basis, which means that the usability of RV data after a disaster can be determined on a per-group basis. Main site Backup site MV1 RV1 MV2 RV2 MV3 RV3 Disk array Disk array ATgroup * Consistency can be maintained among RV1, RV2, and RV3 that are included in an ATgroup. Figure 2-1 ATgroup 11

20 Chapter 2 Disaster Recovery Functions The following describes, in relation to ATgroups, (1) the ATgroup configuration display and setting functions, (2) the state display function, (3) replication control functions, (4) the copy of order guarantee function, and (5) the atomic-break function. The number of ATgroups that can be created, the number of pairs that can be registered to ATgroups, and the number of ATgroups to which replication can be executed in semi synchronous copy mode of order guarantee are limited. For further description of ATgroups, refer to 6.1 Specifications of ATgroup. 12

21 Chapter 2 Disaster Recovery Functions ATgroup Conffiigurattiion Diispllay and Settttiing Functtiions The elements that configure the ATgroup can be displayed and set. Any ATgroup name can be set for an ATgroup, and these names can be used to distinguish among the ATgroups. The MV side of ATgroups in a database group is called a concentrator, and the RV side is called the distributor. The disk array on the concentrator side (MV side) is used to create an ATgroup, which is built by registering RDR pairs. Main site Backup site DB_Group1 MV1 RV1 MV2 RV2 Concentrator Distributor DB_Group2 MV3 RV3 MV4 RV4 Concentrator Distributor Disk array Disk array MV1-RV1, MV2-RV2: ATgroup name of pair included in DB_Group1 s ATgroup MV3-RV3, MV4-RV4: ATgroup name of pair included in DB_Group2 s ATgroup Figure 2-2 Building of ATgroup (Example) 13

22 Chapter 2 Disaster Recovery Functions Any pair that is registered to an ATgroup must meet the following conditions. DDR pairs cannot be registered to an ATgroup. Only RDR pairs can be registered to an ATgroup. A pair that is not the top-level pair cannot be registered to an ATgroup. Only the top-level pair can be registered. If the MV of the pair to be registered includes several RDR pairs, it cannot be registered to an ATgroup. Each RDR pair must be set as a single pair registered to an ATgroup. A registered pair in an ATgroup and the pair to be registered to an ATgroup must be separated. An RDR pair that exists among different disk arrays cannot be registered to a single ATgroup. The RDR pair must be in the same disk array. If the RV is also used as a snapshot, it cannot be registered. It is required that the RV is not used as a snapshot. Pairs that can be registered to ATgroup are illustrated below. Main site Backup site (i) MV RV (ii) MV RV/MV RV RV Disk array Disk array [Description of pairs that can be registered to ATgroup] (i) Single RDR pairs are registered. (ii) A combination of the top-level single RDR pair and DDR pair can be registered as an RDR pair. Figure 2-3 Pairs that can be Registered to ATgroup 14

23 Chapter 2 Disaster Recovery Functions Pairs that cannot be registered to an ATgroup are illustrated below. Main site Backup site (1) Backup site (2) (i) MV RV MV (ii) RV/MV RV (iii) MV RV RV (iv) MV Replicate RV MV RV (v) MV RV (vi) MV RV/BV SV Disk array Disk array Disk array [Description of pairs that cannot be registered to an ATgroup] (i) Pair configured as a DDR pair. (ii) Pairs that is not the top-level pair. (iii) Several RDR pairs that are configured from a single MV. (iv) Pairs in an ATgroup that are not separate from pairs registered to an ATgroup. (v) Pairs in different disk arrays that are in a single ATgroup. (Each of these pairs can be registered to a separate ATgroup) (vi) RV is also used as a snapshot. Figure 2-4 Pairs that cannot be Registered to ATgroup 15

24 Chapter 2 Disaster Recovery Functions ATgroup Sttatte Diispllay Functtiion The state of each pair in an ATgroup can be displayed per ATgroup. ATgroup states are displayed as ATgroup consistency states and ATgroup sync states. ATgroup consistency states The ATgroup consistency state indicates whether or not the data is consistent in all RVs included in the ATgroup. When the ATgroup consistency state is Atomic the data in the ATgroup s RVs is consistent and can be used to recover business operations. When the ATgroup consistency state is Non-atomic the RV data included in that ATgroup is not consistent data except when separation has been completed at a pause point in business operations. Thus, it is not normally used for recovery of business operations. ATgroup Consistency State Atomic Non-atomic Invalid Table 2-1 ATgroup Consistency States Description RV data is consistent as an ATgroup. RV can be accessed when ATgroup s sync state is separated. RV data is not consistent as an ATgroup. This state indicates that the ATgroup does not contain any pairs or has just been created. 16

25 Chapter 2 Disaster Recovery Functions ATgroup sync states These indicate the sync state of pairs in an ATgroup. The sync state of an ATgroup changes according to the sync state of each pair in the ATgroup. ATgroup Sync State Separated Separating (Sep/exec) Separate start (Sep/start) Fault separation (Fault) Fault recovering (Fault/recovering) Replicate synchronous (Rpl/sync) Replicating (Rpl/exec) Replicate start (Rpl/start) Replicate suspend (Rpl/suspend) Restoring (Rst/exec) Restore suspend (Rst/suspend) Invalid Table 2-2 ATgroup Sync States Description This state indicates that all pairs in the ATgroup have been separated. The state of pairs in the ATgroup is either separating or separated. In this state, separation has started for the ATgroup. This state indicates that the ATgroup contains a pair with a fault or a separated state caused by a forced operation. When the ATgroup consistency state is Atomic, RVs cannot be accessed. An operation that recovers from fault separation changes the ATgroup sync state to separated and enables RV access. This state indicates that the ATgroup is changing from fault separation to separated. This state indicates that all pairs in the ATgroup are synchronous due to replication. Pairs in the ATgroup are either being replicated or are synchronous. This state indicates that replication has started for an ATgroup. This state indicates that an abnormal suspend state has occurred for a pair in the ATgroup due to a fault while replicating. This state indicates that restoring will be or is being performed for an ATgroup. This state indicates that an abnormal suspend state has occurred for a pair in the ATgroup due to a fault while restoring. This state indicates that the ATgroup does not contain any pairs or has just been created. * The abbreviations have initial capital letters and are used to distinguish among the sync states of pairs. 17

26 Chapter 2 Disaster Recovery Functions ATgroup sync state when link fault occurs The ATgroup sync state is determined based on the state of the copy source, i.e., the concentrator side (MV side). If a link fault occurs, possibly due to a disk array fault or line fault, communication becomes disabled between the disk array on the concentrator side (MV side) and the disk array on the distributor side (RV side), so the state of the distributor side (RV side) immediately before the disaster occurred is maintained. Consequently, the ATgroup sync state of the concentrator side (MV side) does not match the ATgroup sync state of the distributor side (RV side). Therefore, the link state must be checked in order to confirm the ATgroup sync state. Line fault occurs during replicate synchronous state (Rpl/sync) Main site Backup site MV1 RV1 MV2 RV2 MV3 RV3 Disk array Disk array Access ATgroup sync state on concentrator side (MV side) State changes to fault separation (Fault) due to line fault Access ATgroup sync state on distributor side (RV side) Since state of concentrator side (MV side) cannot be obtained due to a line fault, the replicate synchronous state (Rpl/sync) before the fault occurred is maintained. Figure 2-5 ATgroup Sync State when Link Fault has Occurred 18

27 Chapter 2 Disaster Recovery Functions ATgroup Replliicattiion Conttroll Functtiions Replication control for pairs registered to ATgroups is performed for the entire ATgroup. Such control operations cannot be performed for specific pairs. ATgroup replication Replication is executed for all pairs within the ATgroup. Immediately after replication of the ATgroup, the ATgroup sync state becomes replicate start (Rpl/start). When replication is being executed for all pairs within the ATgroup, the ATgroup sync state is replicating (Rpl/exec). After all pairs in the ATgroup have been copied, the ATgroup sync state changes to replicate synchronous (Rpl/sync) as soon as the activity state has been replicated and the sync state is synchronous. The ATgroup s copy mode is specified when the ATgroup is being replicated. If ATgroup replication is performed during synchronous mode or semi synchronous copy mode of order guarantee, the ATgroup consistency state becomes Atomic as soon as the ATgroup sync state becomes replicate synchronous (Rpl/sync). If ATgroup replication is performed during a background copy mode, the ATgroup consistency state stays Non-atomic if the ATgroup sync state becomes replicate synchronous (Rpl/sync). During the replicate start (Rpl/start) or replicating (Rpl/exec) state, the RV data is not consistent, regardless of the copy mode. Accordingly, before replicating the ATgroup, the data should be backed up to another volume or other operations or steps should be implemented to prevent loss of data consistency. Separation of ATgroup Pairs in the ATgroup are separated. If the ATgroup consistency state is Non-atomic, separation of the ATgroup is performed specifically for each pair in the ATgroup. As soon as separation of the ATgroup is completed, the ATgroup sync state becomes separate start (Sep/start). Once separation is being executed for all pairs in the ATgroup, the ATgroup sync state becomes separating (Sep/exec). When all pairs in the ATgroup are separate, the ATgroup sync state becomes separated. At that time, the ATgroup consistency state becomes Non-atomic. If ATgroup separation is executed when the ATgroup consistency state is Atomic, the data consistency of RVs corresponding to all pairs in the ATgroup is maintained and separation is performed for all pairs at the same time. At that point, the ATgroup s consistency state is maintained as Atomic. Immediately after ATgroup separation, separate (immediate) cannot be used, which means the immediate separation of RVs. 19

28 Chapter 2 Disaster Recovery Functions Restoration of ATgroup Restoration is performed for each pair in the ATgroup. In order to maintain the RV data, this ATgroup restoration processing is only an RV store (protect) operation that does not include any updating of RVs. When an ATgroup is being restored, each pair in the ATgroup is restored and the ATgroup sync state changes to restoring (Rst/exec). When restoration of the ATgroup is completed, the ATgroup sync state becomes separated. Accordingly, if ATgroup restoration is performed when the ATgroup consistency state is Atomic, the RV data is maintained while also maintaining Atomic as the ATgroup consistency state. ATgroup replication control and state transitions Various types of ATgroup replication control and state changes are outlined below. Non-atomic RV protect restore execution state Atomic RV protect restore execution state Restoring Restoring Separated state Restoring Restored Separated while consistent Restoring Restored Separated Separated Replicating Separated due to fault. RV access disabled Recovery from fault separation Replicating Separating Fault separation When differences between MV and RV are reflected Replicating Synchronized via sync mode or semi synchronous copy mode of order guarantee Replication state Atomic break or forced separation Separating Synchronized via background copy mode Replication state Replication state during background copy mode Replication during sync mode or semi synchronous copy mode of order guarantee. Copy mode of order guarantee to RVs. Figure 2-6 Outline of Replication Control and State Transitions 20

29 Chapter 2 Disaster Recovery Functions Copy off Order Guaranttee Functtiion When the ATgroup consistency state is Atomic, the order of updating from MVs to RVs per ATgroup is guaranteed in order to maintain RV data consistency. Data is written from concentrator (MV side) to the distributor (RV side) according to the order in which data was written to the MVs that comprise the ATgroup. This guaranteeing of the update order occurs only when the copy mode is either synchronous mode or semi synchronous copy mode of order guarantee and the ATgroup consistency state is Atomic. No such guaranteeing of the update order can occur when the copy mode is background copy mode or when the ATgroup consistency state is Non-atomic. Main site Backup site Application server Standby server (i) (ii) (iii) DB_Group1 MV1 (iii) RV1 MV2 (ii) RV2 MV3 (i) RV3 ATgroup s consistency state is Atomic Disk array Disk array * (i), (ii), and (iii) indicate the write order Figure 2-7 Copy of Order Guarantee Function 21

30 Chapter 2 Disaster Recovery Functions Attomiic--break Functtiion The atomic-break function forcibly separates all pairs in the ATgroup and forcibly separates the RVs in order to maintain the consistency of RV data when the order of copying to RV cannot be guaranteed. This function is executed when the ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) but forced separation is required since the RV copy order cannot be guaranteed due to a disaster, line fault, or other fault problem. If data is written far beyond the line capacity, data copying performed in synchronization with RV is delayed. If this delay also causes substantial delay in MV writing, an atomic break is generated to stop copying to the replication volume in order to give priority to writing to MV. Execution of an atomic-break stops all copying to RVs, but the consistency of the RV data is maintained. After RVs are forcibly separated by an atomic-break, the option of enabling access to MVs can be specified as enabled or disabled during ATgroup replication. This MV Access after Atomic-break setting is described below. MV Access Continue (default) MV access can be continued even after an atomic-break has been executed. When an atomic-break is executed after a fault that does not affect business tasks (such as a line fault or RV fault), it does not affect business task access either. Since updating of MVs is enabled even after an atomic-break is executed, differences occur between the MV data and RV data after an atomic-break. For example, if a disaster causes a line fault to occur, a fault may later occur in the application server, which would result in loss of data updated after the line fault occurred. When an atomic-break is executed without a link fault such as an RV fault, the ATgroup sync state becomes fault separation (Fault). When changing from this state to RV use, an operation to recover from fault separation is executed on the distributor side (RV side) to enable RV use. Also, when an atomic-break is executed following a link fault such as a line fault, the distributor side (RV side) s ATgroup sync state remains replicate synchronous (Rpl/sync), as it was before the link fault occurred. When changing from this state to RV use, forced separation is executed on the distributor side (RV side), followed by an operation to recover from fault separation in order to enable RV use. MV Access Stop When an atomic-break is executed, MV access is stopped. In fact, execution of an atomic-break stops all business access, even when the fault that has occurred is a line fault or RV-related fault that does not affect business tasks. Since updating of MVs is not possible after an atomic-break, no differences will occur between MV data and RV data after the atomic-break is executed. For example, even if a disaster causes a line fault to occur first and an application server fault to occur later, no data loss will occur as long as the copy 22

31 Chapter 2 Disaster Recovery Functions mode is synchronous mode. To enable RV use after executing an atomic-break, a forced separation is first performed on the distributor side (RV side) to change to fault separation (Fault). Next, an operation to recover from fault separation is performed to enable RV use. When an MV access stop is specified, business tasks may be stopped without prior warning, so MV access stop should not be used ordinarily. 23

32 Chapter 2 Disaster Recovery Functions Semii--Synchronous Copy Mode off Order Guaranttee and Attomiic Break Semi-synchronous copy mode of order guarantee temporarily retains data in a buffer to copy write data to the master volume to a replication volume. However, if a large amount of data are written beyond the line capacity in a short time, free space in the buffer may become insufficient because disaster recovery cannot catch up with buffer data copying to the replication volume. In this case, even in semi-synchronous copy mode of order guarantee, data write to the subsequent master volume is delayed because disaster recovery waits for buffer vacancy. If this write delay to MV continues for a given length of time, an atomic break occurs to stop data copying to the replication volume so that priority is given to data write to the master volume. Some disk array models can adjust maintenance of replication volume copying and write priority to the master volume by setting the delay time (referred to as an allowed response time of MV) for judging this atomic break occurrence per ATgroup in semi-synchronous copy mode of order guarantee. To make this adjustment, use the performance monitoring function that is an option. Environment design 1. If MV write capacity is greater than line transfer capacity at the business peak of copying, a buffer write overflow occurs to delay the write response because the transfer capacity is exceeded. To avoid this delay, it is prerequisite to estimate the necessary line transfer capacity and prepare lines. 2. Determine the time (number of seconds, 18 seconds or less) allowed as the delay of the MV write I/O response time from the business requirements. Note, however, that if a value less than 10 seconds is specified as the allowance time, an atomic break may be determined to have occurred even if a link single-system failure is recovered via link switching. This is because disaster recovery cannot catch up with this failure recovery. Operation evaluation At the evaluation stage, perform measurement and analysis to check if estimation is correct in the same environment as the production environment. 3. Set the following statistical information and threshold using the performance monitoring function: Accumulation of maximum buffer usage on the log (accumulation by default) Accumulation of buffer wait time on the log (addition as LD information, maximum value/mean value) (accumulation by default) Setting of time (1 second) during which the threshold of the buffer wait time (maximum value) is treated as MV write delay occurrence Also set the allowed response time of MV for atomic break occurrence. 4. Check atomic break occurrence and the threshold reporting condition by performing the job simultaneously while copying data. 24

33 Chapter 2 Disaster Recovery Functions Analysis and countermeasures 5. Check if copy processing is enough for business write from the atomic break conditions, threshold excess report, and gathered performance statistical information. If no atomic break has occurred and the buffer wait time is not exceeded, the line capacity is enough. Check and store the vacancy condition of the maximum buffer usage as the original vacancy condition. The timing of atomic break operation when a copy error occurs serves as the allowed response time of MV. If no atomic break has occurred but the buffer wait time threshold is exceeded, judge whether to continue processing as is or whether to take appropriate countermeasures for the delay. If an atomic break has occurred, appropriate countermeasures are required because the line capacity is insufficient. 6. Select a method that meets the requirements from the following methods: Method 1: Enhance the line capacity to increase the transfer capacity. Method 2: If the line capacity is insufficient only in a specific time zone, consider to avoid the MV delay by performing separate operation in that time zone. Method 3: Consider memory purchase to increase the semi-synchronous buffer size. Method 4: Review the allowance time as the MV write I/O response. Production operation 7. Even in the production operation, the monitoring objects are the same as in the settings in step 3. However, the line capacity may become insufficient due to increase in the amount of business although it is sufficient at operation start. For this reason, monitor the portent and take the countermeasures on the preceding page as required. Use the threshold of the maximum buffer wait time as the threshold of the time when a write I/O delay begins to occur due to a buffer overflow. If the buffer has not still overflowed but the buffer usage is gradually increasing as compared to statistical information for the maximum buffer usage, you can reference this statistical information to determine if it is necessary to take appropriate countermeasures for the future buffer overflow. 25

34 Chapter 2 Disaster Recovery Functions 2..2 Siimulltaneous Replliicatiion of Seriialllly Confiigured Paiir Remote data replication functions can be used to simultaneously replicate serially configured pairs. When replication is performed simultaneously, replication at the second and lower levels is enabled only when the copy mode is background copy mode. Also, replication control cannot be performed when copying is performed in different directions at the same time (such as when executing replication for the former and latter serially configured pairs), when the former and latter pair s activity state is separate, or when its sync state is separating. Main site Backup site ATgroup s sync state is replicate state or pair s sync state is synchronous (synchronous, semisynchronous, or background copy) MV1 RV1/MV2 Pair s sync state is synchronous (background copy) RV2 Disk array Disk array Combination of RDR + DDR pairs Main site Backup site (1) Backup site (2) ATgroup s sync state is replicate state or pair s sync state is synchronous (synchronous, semisynchronous, or background copy) Pair s sync state is synchronous (background copy) MV1 RV1/MV2 RV2 Disk array Disk array Disk array Combination of RDR + RDR pairs Figure 2-8 Example of Replication Implemented for Serially Configured Pairs 26

35 Chapter 2 Disaster Recovery Functions 2..3 Remote Paiir Operatiions Remote data replication functions are able to control replication from the main site to a DDR pair at the backup site. They are also able to control replication from an RDR pair at the backup site to a DDR pair at the main site. Using these functions, backup operations using the backup site s DDR can be controlled from the main site. Main site Backup site Application server Standby server Operation instruction MV RV/MV Replicate, separate, restore Replication of serially configured DDR pairs can be controlled from MV side RV MV Replication of serially configured RDR and DDR pairs can be controlled from RV side Operation instruction Replicate, separate, restore RV/MV Disk array Replicate separate restore RV Disk array Figure 2-9 Example of Remote Control 27

36 Chapter 2 Disaster Recovery Functions 2..4 RDR Quiick Sync The RDR quick sync is a function to eliminate differences from the RDR pair to establish the sync state without copying data from MV to RV when the MV and RV data in the RDR pair is already the same. For RDR operation, it is necessary to reflect all data in MV to RV through the RDR link line at startup, making the data state the same (replicate synchronous). If a low-speed line is used as the RDR link line in the configuration, it takes significant time to transfer a lot of data in batch. The RDR quick sync quickly realizes the sync state without putting load on the low-speed RDR link line by other means than the RDR link line (such as transfer using tape) when MV and RV are the same data. The RDR quick sync function is offered only for ACOS-4.only for ACOS-4. This version only supports the function to display the RDR quick sync state in the ism replication management Vollume Updatte Preventtiion To perform the RDR quick sync operation, the data in the MV and RV data in the RDR pair must be the same. To make the MV and RV data the same, execute RAW backup of the MV data to tape using the backup software, transfer it from the main site to the backup site, and execute RAW restore from tape to RV. The RDR quick sync function cannot be used when MV data is updated while tape is transferred to the main site to the backup site. To prevent this situation, use the volume update prevention function and protect the target volume from the operations that causes differences. Update Prevention State Update prevention (Prevent) Table 2-3 Volume Update Prevention State Description State where the volume update prevention function protects a volume from the operations that cause differences. - Since the access right to the target volume is Read Only, write operation from the server cannot be performed. - Replication cannot be performed when the target volume is RV. - Restore cannot be performed when the target volume is MV. - Snapshot restore cannot be performed when the target volume is BV. - The logical disk cannot be deleted from the target volume. - The target volume cannot be changed to the reserve attribute. 28

37 Chapter 2 Disaster Recovery Functions Vollume Compariing//IIdenttiitty Certtiiffiied Volume comparing verifies that the MV and RV data is identical on the bit level using the volume update prevention function. If MV and RV data is identical, MV and RV can be made the same by certifying the identity. The state where volumes were compared and the identity was certified is called the identity state. Identity State Not Certified Comparing Identical Certifying Certified Different Fault(Comparing) Fault(Certifying) Table 2-4 Identity State Description MV and RV have not been compared and the identity of them has not been certified. The state is not Certified. Not Certified is also displayed when MV is in the Prevent state and RV is not in the Prevent state. MV and RV are being compared. Volume comparing found that data of MV and RV were identical. The identity between MV and RV is being certified. The identity between MV and RV has been certified and there are no differences. Volume comparing found that data of MV and RV were not identical. A fault occurred while MV and RV were being compared. A fault occurred while the identity between MV and RV was being certified. 29

38 Chapter 2 Disaster Recovery Functions 2..5 Swap Functiion for RDR Paiir This function swaps the roles of MV and RV in the RDR pair. The swap function is more advantageous than unpairing and resetting the pair because it enables you to immediately switch the roles of MV and RV, avoids full copy during the replication operation after the processing, and maintains the MV data redundancy by RV. The swap function is available in the following disk arrays: - D3 series, D4 series, and D8 series The following product is required to use the swap function. - RemoteDataReplication Ver3 or later For details on the swap function, refer to 2.10 Swap Function for RDR Pair in Data Replication User s Manual (Function Guide). To perform the swap operation for the ATgroup, perform the following operations. - Unpair the RDR pair from the ATgroup and delete the empty ATgroup. - Perform the RDR pair swap operation. - After switching the sites roles, re-create the ATgroup and register the swapped RDR pair. 30

39 Chapter 2 Disaster Recovery Functions 2..6 Operatiion of Partiitiioniing Functiion The partitioning function partitions resources (disk, cache memory, and port) of a disk array on a logic basis such as business to bind and manage virtual storage (VSPAR: VirtualStorage PARtition) that has independent resources. In the partitioning function, the virtual storage described above is referred to as a partition. A user who performs operation management of a specific partition is referred to as a partition user. A storage user (user who performs operation management of all disk arrays) gives the authority to handle a specific partition to a partition user. A partition user is also classified into the following three user levels as in the case of a storage user, and can operate the functions provided by performance optimization for the partitions for which authority the partition user has and can reference the provided information. L1: Allows only reference L2: Allows operation related with ATgroup replication on the operational level (copy operation) L3: Allows all operations For details on the partitioning function, refer to Partitioning User s Manual. However, note that the application range in RemoteDataReplication/DisasterRecovery is limited when you use the partitioning function for operation. In disaster recovery operation, processing is switched to the backup site when the main site fails, and operations such as failure determination and switching range over multiple systems. Therefore, disaster recovery operation must be performed by the storage user. For RDR backup, the grouped pairs can be backed up at once by using the grouping function of the ATgroup. In this case, the operation can be performed by a parition user because it is within the business system. The partition user must have the authority for all the pairs registered in the ATgroup (the authority for the partition to which MV or RV belongs) to operate the ATgroup when the partitioning function is introduced. For details on the specific conditions on each operation, refer to the following table. 31

40 Chapter 2 Disaster Recovery Functions Table 2-5 Operation Conditions for Partition User Imposed by the Partitioning Function Displaying the ATgroup configuration Operation Description Authori- Of the pairs registered in the ATgroup, those for which the partition user has the authority are displayed. The pairs for which the partition user does not have the authorization are not displayed. Creating an ATgroup The partition user can create an ATgroup. Deleting an ATgroup If the partition user has the authority for all the pairs registered in the ATgroup, the partition user can delete the ATgroup after deleting the registered pairs. zation * Registering a pair to an ATgroup Deleting a pair registered to an ATgroup Renaming an ATgroup Change the allowed response time of MV Replicating an ATgroup Separating an ATgroup Restoring an ATgroup Forced separation of an ATgroup Recovery from fault separation Forced deletion of an ATgroup Outputting the ATgroup information list in CSV format Saving an ATgroup and registered pair information The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition use can output the ATgroup information list. Only the information on the pairs in the ATgroup for which the partition user has the authority is saved. If there are pairs for which the partition user does not have the authority, their setting information is not saved. * * * * * * * * * * * : A partition user can perform this operation without any restrictions. *: A partition user can perform this operation if certain conditions are met. 32

41 Chapter 3 Install Software and Build Environment Chapter 3 Install Software and Build Environment This chapter describes methods for software installation and environment building that are required when using disaster recovery functions System Confiiguratiion The hardware and software used to configure a disaster recovery system through disaster recovery functions are shown below. Main site Backup site Backup server Application server Standby server Backup server FC FC Tape library Tape library Management server Management server LAN LAN Management terminal Disk array Disk array Management terminal Figure 3-1 System Configuration 33

42 Chapter 3 Install Software and Build Environment Hardware Disk arrays These disk arrays are used for business tasks and backup. They are placed at the main site and backup site. Remote data replication functions (RemoteDataReplication) and disaster recovery functions (RemoteDataReplication/DisasterRecovery) must be installed. Application servers Application servers are the servers normally used for business tasks. They are placed at the main site. Standby servers Standby servers are available for business tasks. They are placed at a backup site. Backup servers Backup servers are used for ordinary backup operations. They are placed at both the main site and the backup sites. Management servers Management servers are used to perform ATgroup configuration settings and status monitoring. They are placed at the main site and backup sites, where they manage the local site s respective system. Management terminals These terminals are for management clients. Tape libraries These are tape drives that are used for backup operations. 34

43 Chapter 3 Install Software and Build Environment Soffttware Storage control software This software controls disk arrays. Storage control software is included in the BaseProduct. Storage Manager This software performs status monitoring of resources in disk arrays as well as configuration settings and replication management. The replication management functions include pair and ATgroup state display functions, setting functions, and replication control functions. Storage Manager is included in the BaseProduct. PerformanceMonitor This software provides functions to monitor performance when copying resources among disk arrays. PerformanceNavigator This software provides functions for the secure and quick performance analysis by displaying the performance information of the disk array in easy-to-understand graph and table of figures. AccessControl This software provides functions for setting access-enabled logical disks for individual application servers. DynamicDataReplication This software provides functions for data replication within disk arrays. RemoteDataReplication This software provides functions for data replication among disk arrays. RemoteDataReplication/DisasterRecovery This software provides disaster recovery functions such as ATgroup. RemoteDataReplication is required in order to install RemoteDataReplication/DisasterRecovery. However, RemoteDataReplication/DisasterRecovery does not support the usage in combination with RemoteDataReplication Asynchronous. ReplicationControl This software provides commands to control replication from application servers to pairs and to check pair status. ReplicationControl/DisasterRecovery This software provides commands to control replication from application servers to ATgroups and to check ATgroup states. 35

44 Chapter 3 Install Software and Build Environment The target hardware for installation of the above types of software is listed below. Disk array Hardware Application server and standby server Backup server Management server Management terminal Table 3-1 Target Hardware for Software Installation Software to be Installed Storage control software AccessControl DynamicDataReplication RemoteDataReplication RemoteDataReplication/DisasterRecovery ReplicationControl ReplicationControl/DisasterRecovery ReplicationControl Backup software Storage Manager PerformanceMonitor PerformanceNavigator ism client 36

45 Chapter 3 Install Software and Build Environment Pllacementt off Managementt Servers In a disaster recovery system, a management server is placed at each site so that management tasks can be continued after a disaster has occurred. Storage Manager is installed in each site s management server, where it is set up to monitor that site s disk array. Clients connect to Storage Manager at one of these sites to determine the site s disk array status. Main site Backup site Management server Storage Manager Management server is placed at each site Management server Storage Manager Application server Standby server LAN LAN MV RV Disk array Disk array Figure 3-2 Example of Storage Manager Layout 37

46 Chapter 3 Install Software and Build Environment Pllacementt off Conttroll Vollume In a disaster recovery system, a control volume (CV) must be placed so as to perform operations for RV in Not Ready state or replication operation at the remote site. A dedicated volume, which can always be accessed, is provided as a control volume for each disk array of all the servers operating ATgroups and pairs. A control volume has no limitation such as volume capacity, pool type, and RAID type. Therefore, bind a volume of minimum capacity to be created and assign it to a disk array. Application server Standby server Server 1 Server 2 Server 3 Server 4 ReplicationControl/ DisasterRecovery ReplicationControl/ DisasterRecovery ReplicationControl/ DisasterRecovery ReplicationControl/ DisasterRecovery Operation instruction Operation instruction Operation instruction Operation instruction CV1 CV2 CV3 CV4 MV1 RV/MV1 RV1 MV2 RV/MV2 RV2 MV3 RV/MV3 RV3 Disk array Disk array CV1: Control volume for Server 1 CV2: Control volume for Server 2 CV3: Control volume for Server 3 CV4: Control volume for Server 4 Figure 3-3 Example of Control Volume Layout 38

47 Chapter 3 Install Software and Build Environment 3..2 Software IInstallllatiion The following describes the installation methods used for RemoteDataReplication/DisasterRecovery and ReplicationControl/DisasterRecovery. For description of the installation methods used for other software such as ism, ReplicationControl, and RemoteDataReplication, see the installation guide for each software product IInsttallllattiion off RemotteDattaReplliicattiion//DiisastterRecovery (1) Operating environment Applicable hardware RemoteDataReplication/DisasterRecovery runs on disk array systems. Specifically, it operates on a disk array that includes remote data replication functions (RemoteDataReplication) and disaster recovery functions (RemoteDataReplication/DisasterRecovery). (2) Installation Before using RemoteDataReplication/DisasterRecovery, the RemoteDataReplication software must already be installed, the product version must support at least the physical disk capacity of the disk array, and licensing for the disk array must be cleared. License clearance is required in order to use RemoteDataReplication/DisasterRecovery functions ReplliicattiionConttroll//DiisastterRecovery IInsttallllattiion and Enviironmentt Settttiing (1) Operating Environment Operating system and linkage software For details on the OSs supporting this software and the linkage software that can be used in conjunction with this software, refer to the installation guide attached to this software. Required disk free space For details on the disk free space required to install this software, refer to the installation guide attached to this software. Required memory For details on the memory capacity required to use this software, refer to the installation guide attached to this software. 39

48 Chapter 3 Install Software and Build Environment (2) Installation For details on the installation procedure, refer to the installation guide attached to this software. (3) Uninstallation For details on the uninstallation procedure, refer to the installation guide attached to this software. (4) Update To update the ReplicationControl/DisasterRecovery software, uninstall the existing software, and then install the new software. For details on the installation and uninstallation procedure, refer to the installation guide attached to this software. (5) Software Environment Setting After the installation of the ReplicationControl/DisasterRecovery software, set the software environment. For details on the software environment setting, refer to the installation guide attached to this software. (6) Replication Operation Option Setting File Automount disabling must be set for the disaster recovery system operation. To set the automount disabling, specify NOTREADY for the UMOUNT_RVACC parameter in the [FILESYSTEMOPERATION] section of the replication operation option setting file (ini file). The following is a description example: [FILESYSTEMOPERATION] UMOUNT_RVACC=NOTREADY Also, when you use Windows Server 2003 or Windows Server 2008, execute the MOUNTVOL command with /N option, which is the OS standard for automount disabling, in addition to the setting above. MOUNTVOL /N For details on replication operation option setting file, refer to the ControlCommand Command Reference. Various operations on command execution can be set by using the replication operation option setting file. 40

49 Chapter 3 Install Software and Build Environment 3..3 Buiilldiing an ATgroup The following describes how to build an ATgroup, which is required in order to build a disaster recovery system using disaster recovery functions. Various operations are executed by using ism s replication management functions or by using commands provided by ReplicationControl and ReplicationControl/DisasterRecovery. Design items that are required by the data replication function (such as selection of logical disk, nicknames, access control, pair configurations, and platform-specific items) must be taken into consideration along with data replication items. For description of this approach and the related settings, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. 41

50 Chapter 3 Install Software and Build Environment Paiirs Regiisttered tto an ATgroup In a disaster recovery system, there are no limits on the volume size, pool types, or RAID types used for business tasks. The volumes used for recovery must be copied to a backup site while maintaining consistency in each business task unit. For example, when using a database, consistency must be maintained in the volumes of database files, log files, etc., that comprise the database. Accordingly, pairs of volumes that comprise the database include volumes that have undergone software mirroring (such as by using database functions), and all of these volumes are registered to an ATgroup. An ATgroup is created for each set of business tasks, and the volume pairs that corresponding to individual business tasks are registered to this ATgroup. Main site Backup site DB_MV1 DB_RV1 DB DB_MV2 DB_RV2 DB_MV3 DB_RV3 Log LOG_MV1 LOG_MV2 LOG_RV1 LOG_RV2 Disk array Disk array Figure 3-4 ATgroup Configuration Example The number of ATgroups that can be created is limited by the number of pairs that can be registered to the ATgroups. For details on such ATgroups, refer to 6.1 Specifications of ATgroup. 42

51 Chapter 3 Install Software and Build Environment Stteps ffor Buiilldiing an ATgroup The steps for building ATgroups and configuration examples are described below. (1) Overview of ATgroup Build Steps The following are some configuration examples related to the steps for building ATgroups. First, build an ATgroup comprising two disk arrays whose names are LOCAL_ARRAY and REMOTE_ARRAY. Next, register two pairs of logical disks, one named DB_MV and DB_RV and other named LOG_MV and LOG_RV, to the ATgroup named DB_Group. Access controls are set for the respective volumes, so that only MVs in the ATgroup can be access from an application server, only RVs in the ATgroup can be accessed by the standby server, and only the ATgroup s DDR RVs at each site can be accessed from the backup server. Main site Backup site Application server Standby server Backup server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array Figure 3-5 Configuration Example for Steps in Building ATgroup 43

52 Chapter 3 Install Software and Build Environment (2) Flow of ATgroup Build Steps [Operations at main site] [Operations at backup site] Step 1. Create volume list (application server and backup server) Step 2. Build DDR pair (Main site) Step 3. Set up volumes (application server and backup server) Step 4. Create volume list (application server and backup server) Step 5. Create RDR pair (application server) Step 6. Create ATgroup (application server) Step 7. Register pairs to ATgroup (application server) Step 8. Replicate ATgroup (application server) Step 9. Unmount MV (application server) Step 10. Separate ATgroups (application server) Step 11a. Mount MV in ATgroup (application server) Step 11b. Mount RV in ATgroup (Standby server) Step 12. Create volume list (Standby server and backup server) Step 13. Build DDR pairs (Backup site) Step 14. Set up volumes (Standby server and backup server) * This flow of steps and the operation steps described below are just examples. Be sure to perform operations and steps that are suited to the actual environment. Figure 3-6 Flow of ATgroup Build Steps 44

53 Chapter 3 Install Software and Build Environment (3) Operation Steps Step 1. Create a volume list (application server and main site s backup server) To set up volumes at the main site, create a volume list on the application server and the main site s backup server. If these servers that are used to create the volume list are running ism on HP-UX or Linux, ism must be paused first. (Command example for Windows) ismvollist -cr (Command example for UNIX) ismvollist -r * It is not necessary to create the volume list on main site s backup server with this step for the operation using the GPT disk. Step 2. Build DDR pairs (main site) Next, build DDR pairs. For description of the steps in building DDR pairs, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. The steps described below assume that DDR pairs have been separated, so be sure to separate the DDR pairs beforehand. Step 3. Set up volumes (application server and main site s backup server) Set up volumes using the application server and main site s backup server. For description of the steps in setting up volumes, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. When using LVM on HP-UX servers, fetch the map file from the application server s MV. Next, transfer this map file to all servers and perform a backup. When using LVM on Linux servers, it is not necessary to fetch the map file. (Example of command for fetching map file) vgexport -v -p -m vg01.map /dev/vg01 vgexport -v -p -m vg02.map /dev/vg02 Step 4. Create volume list (application server and main site s backup server) First connect the main site s disk array to the backup site s disk array, and then create a volume list using the application server and the main site s backup server. If these servers that are used to create the volume list are running ism on HP-UX or Linux, ism must be paused first. This step is not necessary if the main site s disk array and backup site s disk array were already connected at Step 1. The main site s disk array and backup site s disk array must be connected for the following steps. (Command example for Windows) ismvollist -cr (Command example for UNIX) ismvollist -r 45

54 Chapter 3 Install Software and Build Environment Step 5. Create RDR pairs (application server) Create RDR pairs to be registered to an ATgroup. (Command example) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_pair -pair -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld OR ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv In addition, they can be created using the main site s ism software. Figure 3-7 Example of Using ism to Create RDR Pairs 46

55 Chapter 3 Install Software and Build Environment Step 6. Create ATgroup (application server) Create an ATgroup. (Command example) ismrc_atg -create -atg DB_Group -arrayname LOCAL_ARRAY -linkarrayname REMOTE_ARRAY In addition, it can be created using the main site s ism software. Figure 3-8 Example of Using ism to Create ATgroup The number of ATgroups that can be created is limited by the number of volume pairs that can be registered to the ATgroups. For details on such ATgroups, refer to 6.1 Specifications of ATgroup. 47

56 Chapter 3 Install Software and Build Environment Step 7. Register pairs to ATgroup (application server) Register RDR pairs to a created ATgroup. (Command example) ismrc_atg -add -atg DB_Group -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_atg -add -atg DB_Group -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld OR ismrc_atg -add -atg DB_Group -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv Pairs can also be registered via the main site s ism software. Figure 3-9 Example of Pair Registration to ATgroup Using ism The number of ATgroups that can be created is limited by the number of volume pairs that can be registered to the ATgroups. For details on such ATgroups, refer to 6.1 Specifications of ATgroup. 48

57 Chapter 3 Install Software and Build Environment Step 8. Replicate ATgroup (application server) Replicate an ATgroup to reflect differences. (Command example) ismrc_replicate -atg DB_Group -wait This replication can also be performed using ism. Figure 3-10 Example of ATgroup Replication Using ism 49

58 Chapter 3 Install Software and Build Environment Step 9. Unmount MV in ATgroup (application server) (Windows) To copy pause point data to an RV in an ATgroup, first stop the current service that is accessing the target volume, then flush and unmount the files in the ATgroup s MV. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (UNIX) To copy pause point data to an RV in an ATgroup, first temporarily unmount the MV volume in the ATgroup if it is already mounted. To unmount, unmount the file system, then set the volume group as inactive or deport the disk group. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is being used, the volume group is set as inactive. If VxVM is being used, deport the disk group. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 50

59 Chapter 3 Install Software and Build Environment Step 10. Separate ATgroup (application server) Once the ATgroup sync state is replicate/synchronous (Rpl/sync), separate the ATgroup, then separate the RVs in the ATgroup. (Command example) ismrc_separate -atg DB_Group -wait This can also be done using ism. Figure 3-11 Example of ATgroup Separation Using ism Step 11. Mount volume (MV or RV) in ATgroup (application server and standby server) (Windows) Mount a volume (MV or RV) in an ATgroup on an application server and a standby server. On an application server, use the ismrc_mount command to mount. Afterward, restart the service that was temporarily stopped for unmounting. On a standby server, start the Windows Computer Management utility to assign a drive letter to the disk. Example of mounting on an application server (i) Mount the file system. (Command example) ismrc_mount -drv E: ismrc_mount -drv F: 51

60 Chapter 3 Install Software and Build Environment (ii) Restart the service. (Command example) NET START ESMCommonService Example of mounting on a standby server Start the Computer Management utility and assign a drive letter to the disk. Even if a drive letter has already been assigned on the standby server, there is a chance that the drive letter may have been changed when building the environment, so be sure to check the assigned drive letter. (UNIX) On an application server and standby server, activate the volume group for a volume (MV or RV) in an ATgroup, or import and mount the disk group. When LVM is being used on a standby server and both servers are HP-UX servers, change the volume group ID, create a directory for the volume group, and import the volume group information before activating the volume group. When using LVM on Linux servers, only activation of the volume group can be performed. When using VxVM on a standby server, use the -C option to import. Example of mounting on an application server (i) Activate the volume group or import the disk group. If using LVM, activate the volume group. If using VxVM, import the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg import dg01 vxdg import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Example of command when using LVM and mounting a VxFS file system) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Example of command when using VxVM and mounting a VxFS file system) 52

61 Chapter 3 Install Software and Build Environment When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log Example of mounting on a standby server (i) Activate the volume group or import the disk group. When using LVM on HP-UX servers, change the volume group ID, create a directory for the volume group, and then import the volume group information from the map file transferred from the application server before activating the volume group. When using LVM on Linux servers, only activation of the volume group can be performed. When using VxVM, use the -C option to clear and import the disk group. (Example of command when using LVM) When using HP-UX servers: vgchgid /dev/rdsk/c2t0d0 vgchgid /dev/rdsk/c2t0d1 mkdir /dev/vg01 mkdir /dev/vg02 mknod /dev/vg01/group c 64 0x mknod /dev/vg02/group c 64 0x vgimport -v -m vg01.map /dev/vg01 /dev/dsk/c2t0d0 vgimport -v -m vg02.map /dev/vg02 /dev/dsk/c2t0d1 vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 When using Linux servers: vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall On Linux servers, to use partition configuration for the first time on the RV side after creating, updating, and replicating a partition, it is necessary to restart the server to make the OS recognize the RV partition information. 53

62 Chapter 3 Install Software and Build Environment (ii) Mount the file system. (Example of command when using LVM and mounting a VxFS file system) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Example of command when using VxVM and mounting a VxFS file system) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log When using RemoteCluster, MV and RV in an ATgroup are operated in the same cluster. Therefore, changing the volume group ID at a standby server is not executed. For environment creation by RemoteCluster, refer to the RemoteCluster with MC/ServiceGuard User s Manual. Step 12. Creation of Volume List (Standby Server and Backup Site s Backup Server) Create a volume list using a standby server and the backup site s backup server. If these servers that are used to create the volume list are running ism on HP-UX or Linux, ism must be paused first. (Command example for Windows) ismvollist -cr (Command example for UNIX) ismvollist -r Step 13. Build DDR pairs (backup site) Next, build DDR pairs. For description of the steps in building DDR pairs, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. The steps described below assume that DDR pairs have been separated, so be sure to separate the DDR pairs beforehand. 54

63 Chapter 3 Install Software and Build Environment Step 14. Set up volumes (standby server and backup site s backup server) Set up volumes using the standby server and backup site s backup server. For description of the steps in setting up volumes, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. When using LVM on HP-UX servers, use the map file transferred from the application server (as was described above in step 3) as the map file required when using LVM. When using LVM on UNIX servers, after the ATgroup has been built, the cluster settings, etc., have been made for the various sites, and setup for starting business tasks has been completed, be sure to fetch and store the volume group setting information on each site s server. This volume group setting information should be backed up once setup for starting business tasks has been completed for the various servers. In some cases, the volume group setting information is used for fault repair. (Example of commands to fetch volume group setting information file) (Command used for application server) When using HP-UX servers: vgcfgbackup -f vg01_mv.conf /dev/vg01 vgcfgbackup -f vg02_mv.conf /dev/vg02 When using Linux servers: vgcfgbackup /dev/vg01 vgcfgbackup /dev/vg02 (Command used for standby server) When using HP-UX servers: vgcfgbackup -f vg01_rv.conf /dev/vg01 vgcfgbackup -f vg02_rv.conf /dev/vg02 When using Linux servers: vgcfgbackup /dev/vg01 vgcfgbackup /dev/vg02 55

64 Chapter 4 Software Operation Methods Chapter 4 Software Operation Methods This chapter describes operation methods for ism s ATgroup functions and ReplicationControl/DisasterRecovery commands Storage Manager s ATgroup Functiion Operatiion Method (GUII) When RemoteDataReplication/DisasterRecovery is installed, ism s replication functions enable ATgroup functions to be used under the following operation privilege. For further description of operation methods for ism and ism s replication functions, start method, operation privileges, event link function, and replication function, refer to either the User s Manual or User s Manual (UNIX) in accordance with your OS in use, or the Data Replication User s Manual (Function Guide). In addition, ism s configuration setting function enables use of the ATgroup s batch setting function. For details on batch settings, see the Configuration Setting Tool User s Manual (GUI). Function ATgroup configuration display, status display Table 4-1 List of ism s ATgroup Functions Description Operation Privilege L1 L2 L3 Display the ATgroup configuration Display the ATgroup state Described in Section Create an ATgroup Delete an ATgroup ATgroup settings Register a pair to an ATgroup - - Delete a pair registered to an ATgroup Rename an ATgroup Change the allowed response time of MV ATgroup replication operations ATgroup forced operation, fault repair Save configuration information Replicate an ATgroup Separate an ATgroup Restore an ATgroup Forced separation of an ATgroup Recovery from fault separation Forced deletion of an ATgroup Output the ATgroup information list in CSV format Save ATgroup and registered pair information The screens displayed by ism s ATgroup functions and the operation of these functions are described below. 56

65 Chapter 4 Software Operation Methods ATgroup IInfformattiion Screen The ATgroup Information screen is opened when using ATgroup functions. Select the target disk array in the Replication screen, then select the ATgroup Information tab. To display the Replication screen, either select [File] [Replication] in the Main (state monitoring) screen or select [Replication] on the toolbar. Select ATgroup Information tab. Figure 4-1 ATgroup Information Screen The items on the ATgroup Information screen s menu bar for ATgroup functions are described below. Among the menus, the [View] and [Help] menus do not contain any items pertaining specifically to ATgroup functions. For details on the [View] and [Help] menus, refer to the Data Replication User s Manual (Function Guide). Outputting the ATgroup Information List in CSV Format (4.1.14) Saving an ATgroup and Registered Pair Information (4.1.15) Close the replication management screen Figure 4-2 File Menu 57

66 Chapter 4 Software Operation Methods Creating an ATgroup (4.1.3) Deleting an ATgroup (4.1.4) Creating/Deleting an ATgroup Volume (4.1.5) Renaming an ATgroup (4.1.6) Changing the Allowed Response Time of MV (4.1.7) Figure 4-3 ATgroup Settings in Operations Menu Replicating an ATgroup (4.1.8) Figure 4-4 ATgroup Operations in Operations Menu Separating an ATgroup (4.1.9) Restoring an ATgroup (4.1.10) Forced separation of an ATgroup (4.1.11) Figure 4-5 ATgroup Forced Operations in Operations Menu Forced Deletion of an ATgroup (4.1.13) Recovery from Fault Separation (4.1.12) 58

67 Chapter 4 Software Operation Methods Diispllayiing tthe ATgroup Conffiigurattiion and Sttatte Properties in the ATgroup information screen and Replication screen can be used to check the ATgroup configuration display and state display. ATgroup information screen The state of the following items can be checked via the ATgroup information screen. ATgroup list Configuration of pairs registered to ATgroup Configuration information on ATgroup s active mode, etc. ATgroup consistency state ATgroup sync state Link state The ATgroup list, which appears near in the upper window of this screen, provides various types of information about ATgroups. When an ATgroup is selected from this list, information about pairs registered to the selected ATgroup is displayed in the lower window. Figure 4-6 ATgroup Configuration and State Display Screen [Description of items in upper window of ATgroup information screen] (i) Type This indicates the ATgroup s location (concentrator or distributor). Display items Concentrator Selected disk array s ATgroup is on the concentrator side (MV side) Distributor Selected disk array s ATgroup is on the distributor side (RV side) 59

68 Chapter 4 Software Operation Methods Icons (green) Concentrator s state is normal. (light blue) Distributor s state is normal. (green) Fault occurred at concentrator. (light blue) Fault occurred at distributor. Link fault has occurred or disk array is frozen. (ii) ATgroup Name This indicates ATgroup name. (iii) ATgroup Consistency State This indicates the ATgroup consistency state. Display items Atomic ATgroup consistency state is Atomic. Consistency of RV data is maintained in ATgroup. Non-atomic ATgroup consistency state is Non-atomic. Consistency of RV data is not maintained in ATgroup. Blank (nothing displayed) ATgroup consistency state is invalid. Either ATgroup contains no pairs or ATgroup has just been created. (iv) ATgroup Sync State This indicates the ATgroup sync state. Display items Separated ATgroup sync state is separated. All pairs included in ATgroup are separated. Sep/exec ATgroup sync state is separating (sep/exec). Pairs included in ATgroup are either separated or being separated. Sep/start ATgroup sync state is separate start (Sep/start). Separation of pairs in ATgroup has been started. Fault ATgroup sync state is fault separation (Fault). Pairs included in ATgroup have experienced a separate state by a fault or forced operation. Fault/recovering ATgroup sync state is fault recovering (Fault/recovering). Pairs included in ATgroup have experienced a separate state by separated, a fault, or forced operation. 60

69 Chapter 4 Software Operation Methods Rpl/sync ATgroup sync state is replicate synchronous (Rpl/sync). All pairs included ATgroup have been synchronized by replication. Rpl/exec ATgroup sync state is replicating (Rpl/exec). Pairs included in ATgroup have been synchronized or are being synchronized by replication. Rpl/start ATgroup sync state is replicate start (Rpl/start). Replication of pairs in ATgroup has started. Rpl/suspend ATgroup sync state is replicate suspend (Rpl/suspend). Pairs included in ATgroup have experienced a fault, causing replication to be suspended. Rst/exec ATgroup sync state is restoring (Rst/exec). Pairs included in ATgroup will be restored or are being restored. Rst/suspend ATgroup sync state is restore suspend (Rst/suspend). Pairs included in ATgroup have experienced a fault, causing restoration to be suspended. Blank (nothing displayed) ATgroup consistency state is invalid. Either ATgroup contains no pairs or ATgroup has just been created. (v) Link Disk Array Name This indicates an identification name given to the link disk array. If blank is given, it indicates that the link disk array name has not been decided. (vi) LINK State This indicates state of linkage between disk arrays. Display items Normal Link state is normal. Fault Communication fault has occurred between disk arrays or link is invalid. Freeze Data replication functions are frozen in the disk array to be connected. Blank (nothing displayed) Either ATgroup contains no pairs or ATgroup has just been created. (vii) Copy Mode This indicates ATgroup s copy mode. Display items Synchronous All replication of pairs in ATgroup is being executed in synchronous mode. Semi-synchronous All replication of pairs in ATgroup is being executed in semi synchronous copy mode of order guarantee. If the ATgroup consistency state is Atomic, copying is executed in 61

70 Chapter 4 Software Operation Methods a guaranteed order. Background Copy All replication of pairs in ATgroup is being executed in background copy mode. Blank (nothing displayed) ATgroup has not been replicated. (viii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (ix) Difference Quantity This indicates the total amount of differential volume between the ATgroup s concentrator (MV side) and distributor (RV side). (x) Atomic-break mode This indicates MV access mode after an atomic-break. Display items MV Access Continue Access to MVs can be continued after an atomic-break is executed. MV Access Stop Access to MVs is stopped after an atomic-break is executed. Blank (nothing displayed) Pairs in ATgroup are not being replicated or copy mode is background copy mode. (xi) Concentrator Number This indicates the concentrator number in hexadecimal. [Description of items in lower part of ATgroup information screen] The lower part of this screen displays information about pairs registered to ATgroups. The pair information items that appear in this screen are the same as those in the Replication Information screen (except for ATgroup name). For details on the pair information items in the Replication Information screen, refer to the Data Replication User s Manual (Function Guide). In the case of a partition user, the volumes for which the partition user does not have the management authorization are not displayed at the bottom of the ATgroup Information screen. 62

71 Chapter 4 Software Operation Methods Replication screen s disk array properties Perform the following steps to display the Replication screen s disk array properties. Select the target disk array in the Replication screen, and then select [View] [Properties] in the menu. Select the target disk array, then right-click the mouse and select [Properties]. Use this properties window to check the properties of the ATgroup. (i) (ii) (iii) (iv) (v) Figure 4-7 Replication Screen s Disk Array Properties [Description of ATgroup information in Replication screen s disk array properties] (i) Maximum Number of Concentrator This indicates the maximum number of concentrators in an ATgroup that can be built for the target disk array. (ii) Maximum Number of Concentrator Simultaneously Executing on Semi Synchronous Copy Mode This indicates the maximum number of concentrators that can be executed with semi synchronous copy mode specified for the target disk array. (iii) Maximum Number of Volume of All ATgroup 63

72 Chapter 4 Software Operation Methods This indicates the maximum number of pairs that can be registered to ATgroup for the target disk array system. (iv) Maximum Number of Volume of One ATgroup This indicates the maximum number of ATgroups to which one pair can be registered. (v) Function of specifying Link Disk Array Name on creating an ATgroup This indicates whether to enable the function of specifying the link disk array name when registering an ATgroup. 64

73 Chapter 4 Software Operation Methods Creattiing an ATgroup The ATgroup Creation screen is used to create ATgroups. Displaying ATgroup Creation screen Perform either of the following steps to open the ATgroup Creation screen. Select a disk array for creating an ATgroup on the concentrator side (MV side), then select [Operation] [ATgroup] [ATgroup Setting] [ATgroup Creation] in the menu. Select a disk array for creating an ATgroup on the concentrator side (MV side), then right-click the mouse on the upper part of the ATgroup information screen and select [ATgroup Setting] [ATgroup Creation]. < Function of specifying Link Disk Array Name on creating an ATgroup is ON in the disk array properties> (display the disk array properties of the disk array to be created from the replication screen of ism to check the setting) (i) (ii) (iii) Figure 4-8 ATgroup Creation Screen [Description of ATgroup Creation screen] (i) Disk Array Name This indicates the name of the disk array for the ATgroup to be created on the concentrator side (MV side). (ii) Link Disk Array Name Specify the link disk array name of the pair to be added to the ATgroup. (iii) ATgroup Name Enter the name of the ATgroup to be created in this text box. The ATgroup name can include up to 32 characters consisting of alphanumeric characters, underbars (_) and/or forward slashes (/). After selecting the link disk array name of an ATgroup to be created and entering the name of the ATgroup, click the [Execute] button to display a confirmation dialog before creating an ATgroup. 65

74 Chapter 4 Software Operation Methods <Function of specifying Link Disk Array Name on creating an ATgroup is OFF in the disk array properties> (i) (ii) Figure 4-9 ATgroup Creation Screen [Description of ATgroup Creation screen] (i) Disk Array Name This indicates the name of the disk array for the ATgroup to be created on the concentrator side (MV side). (ii) ATgroup Name Enter the name of the ATgroup to be created in this text box. The ATgroup name can include up to 32 characters consisting of alphanumeric characters, underbars (_) and/or forward slashes (/). After entering the name of the ATgroup to be created, click the [Execute] button to display a confirmation dialog before creating the ATgroup. 66

75 Chapter 4 Software Operation Methods Handling of errors during ATgroup creation When an error occurs during ATgroup creation, the following message dialog appears. Be sure to execute handling for the respective error. Error Message [02043] Not all the link paths between the specified Disk Arrays are in the normal state. Table 4-2 Handling of Errors During ATgroup Creation Description of Error and Handling The link state between the disk array creating an ATgroup and disk array selected with the link disk array name is invalid. Check the link state of the disk array and restore the link state. [02073] There is no available Link Disk Array. [02074] Specified Link Disk array does not support RemoteDataReplication/Disaster Recovery or the license for RemoteDataReplication/Disaster Recovery is not available. [02237] Please input an ATgroup Name. [02238] Cannot use characters other than alphanumeric characters, "/" and "_". [02239] Cannot set the default ATgroup Name. [02241] Cannot set due to monitor stopped or freezing. [02242] The same ATgroup Name exists. [02243] Cannot create it because maximum registered number is arrived. [02257] Failed to create for exceeding the maximum ATgroup of the Link Disk Array which has different specification. An ATgroup cannot be registered because there is no link disk array. An ATgroup cannot be registered to the link disk array. Select the link disk array with which an ATgroup can be registered and re-execute. The ATgroup name has not been entered. Enter the ATgroup name now. A prohibited character was entered. Enter only alphanumeric characters, underbars (_), or forward slashes (/) when specifying an ATgroup name (up to 32 characters). The default ATgroup name is reserved by the system. Enter a different ATgroup name. ism has suspended monitoring of or has frozen the target disk array. If ism has suspended monitoring, restart the disk array monitoring. If it has frozen the target disk array, defreeze it. The system already includes an ATgroup with the same name. Enter a unique ATgroup name. The maximum number of ATgroups is set in the system properties. Delete unneeded ATgroups. For details on ATgroups, refer to 6.1 Specifications of ATgroup. ATgroup registration failed because ATgroups have been registered to all the concentrator numbers 0000h to 003fh. The maximum number of ATgroups that can be created between the disk arrays with different specifications (disk arrays whose maximum numbers of concentrators in the disk array properties are different) is based on the disk array with the lower specification. Namely, ATgroups are registered to the concentrator numbers 0000h to 003fh. However, when 192 ATgroups or more have already been registered between the disk arrays whose maximum numbers of concentrators in the disk array properties are the same, the maximum number of ATgroups to be additionally created decreases. Delete unnecessary ATgroups created in the concentrator numbers 0000h to 003fh of ATgroup information and re-create a new ATgroup for registration. For ATgroup specifications, refer to 6.1 Specifications of ATgroup. 67

76 Chapter 4 Software Operation Methods Conditions for creation of ATgroup (i) When creating an ATgroup, the ATgroup name you enter cannot exceed 32 characters, can include only alphanumeric characters, underbars (_), and/or forward slashes, and must be unique within the system. (ii) The disk array for the ATgroup to be created on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) On the disk array for the ATgroup to be created on the concentrator side (MV side), replication functions must not be in frozen state. (iv) The specified ATgroup name is not the existing one. (v) The number of ATgroups which have already been registered to the specified disk array must not exceed the maximum number. (vi) When the Function of specifying Link Disk Array Name on creating an ATgroup in the disk array properties is ON (display the disk array properties of the disk array to be created from the replication screen of ism to check the setting), the link state between the disk arrays specified with the link disk array name must be the normal state. (vii) When the Function of specifying Link Disk Array Name on creating an ATgroup in the disk array properties is ON (display the disk array properties of the disk array to be created from the replication screen of ism to check the setting), the concentrator number must be smaller than or equal to the number of specifications available for registering ATgroup of the disk array specified with the link disk array name. 68

77 Chapter 4 Software Operation Methods Dellettiing an ATgroup The ATgroup Deletion screen is used to delete ATgroups. Items displayed on ATgroup Deletion screen Perform either of the following steps to open the ATgroup Deletion screen. Select the concentrator for the ATgroup to be deleted in the ATgroup information screen, then select [Operation] [ATgroup] [ATgroup Setting] [ATgroup Deletion] in the menu. Select the concentrator for the ATgroup to be deleted in the ATgroup information screen, then right-click the mouse and select [ATgroup Setting] [ATgroup Deletion]. (i) (ii) Figure 4-10 ATgroup Deletion Screen [Description of ATgroup Deletion screen] (i) Disk Array Name This indicates the name of the disk array for the ATgroup to be deleted on the concentrator side (MV side). (ii)atgroup Name This indicate the name of the ATgroup to be deleted in this text box. Click the [Execute] button to open a confirmation dialog before deleting the ATgroup. 69

78 Chapter 4 Software Operation Methods Handling of errors during ATgroup deletion When an error occurs during ATgroup deletion, the following message dialog appears. Be sure to execute handling for the respective error. Error Message [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) Table 4-3 Handling of Errors During ATgroup Deletion Description of Error and Handling Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup to normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Created volume exists) ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. A pair has been registered to the specified ATgroup. Delete the registered pair from the ATgroup. Conditions for deletion of ATgroup (i) All pairs registered to the target ATgroup must be deleted beforehand and the ATgroup sync state must be invalid. (ii) The disk array for the ATgroup to be deleted on the concentrator side (MV side) must be set for monitoring by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be deleted on the concentrator side (MV side) must not have frozen replication functions. 70

79 Chapter 4 Software Operation Methods Creattiing//Dellettiing an ATgroup Vollume The ATgroup Volume Creation/Deletion screen is used to register pairs to or delete pairs from an ATgroup. Items displayed on ATgroup Volume Creation/Deletion screen Perform either of the following steps to open the ATgroup Volume Creation/Deletion screen. In the ATgroup information screen, select the concentrator of the ATgroup whose pair will be registered or deleted, then select [Operation] [ATgroup] [ATgroup Setting] [ATgroup Volume Creation/Deletion] in the menu. In the ATgroup information screen, select the concentrator of the ATgroup whose pair will be registered or deleted, then right-click the mouse and select [ATgroup Setting] [ATgroup Volume Creation/Deletion]. (i) (ii) (iii) (iv) (v) (vi) (vii) (viii) (ix) Figure 4-11 ATgroup Volume Creation/Deletion Screen 71

80 Chapter 4 Software Operation Methods [Description of ATgroup Volume Creation/Deletion screen] (i) Disk Array Name This indicates the name of the disk array on the concentrator side (MV side) where a pair will be added or deleted. (ii)atgroup Name This indicates the name of the ATgroup where a pair will be added or deleted. (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (iv) Registered Volume List This is a list of currently registered pairs. When deleting a registered pair from an ATgroup, select the target pair from this list. (v) [Add] Select this to add (register) a pair. (vi) [Delete] Select this to delete a registered pair. (vii) OS Type Select a format to narrow the list of candidate volumes. (viii) Link Disk Array Name Select this to narrow the list of candidate pairs to those registered to the link disk array. If a pair has already been registered or the link disk array has been specified at ATgroup creation, the link disk array is specified automatically. (ix) Candidate Volume List This displays a list of registered candidate volumes. When registering a pair to an ATgroup, select the pair from this list. If you select a pair from the candidate volume list and then select the [Add] button, a confirmation dialog will appear before the selected pair is registered to the ATgroup. If you select a pair from the registered volume list and then select the [Delete] button, a confirmation dialog will appear before the selected registered pair is deleted from the ATgroup. In the case of a partition user, the volumes in the pairs for which the partition user does not have the management authorization are not displayed in (iv) Registered Volume List and (ix) Candidate Volume List. 72

81 Chapter 4 Software Operation Methods Handling of errors when adding or deleting pair in ATgroup If an error occurs when adding or deleting a pair in an ATgroup, the following message dialog will appear. Be sure to execute handling for the respective error. Table 4-4 Handling of Errors when Adding or Deleting Pair in ATgroup Error Message Description of Error and Handling [02058] ATgroup Volume Creation failed(number of faulted volume). [02059] ATgroup Volume Deletion failed (number of faulted volume). Either the volume state is unexecutable or a disk array is abnormal. If the volume state is unexecutable, check the execution conditions of pair creation and volume state. If a disk array is abnormal, recover the disk array and restore the ATgroup to normal state. A disk array is abnormal. Check the disk array. Recover the disk array and restore the ATgroup to normal state. [02243] Cannot create it because maximum registered number is arrived. [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Link failure) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) [02258] Specified ATgroup cannot involve the volume paired with the disk array which has different specification. Please check the Concentrator Number, and retry creating an ATgroup. The maximum number of volumes that can be registered is set in the system properties. Delete unneeded volumes. For details on such ATgroups, refer to 6.1 Specifications of ATgroup. Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup to normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. After ATgroup Volume Creation/Deletion screen was displayed, the ATgroup was deleted. Check whether the specified ATgroup has already been deleted. The link between the disk arrays is invalid. Check and restore the link status. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. It indicates that the volume paired with the disk array with different specification cannot be added to the ATgroup registered to the concentrator number 0040h or more. The maximum number of ATgroups that can be created is based on the disk array with lower specifications between the disk arrays with different specifications (disk arrays whose maximum number of concentrators in the disk array properties are different). Namely, ATgroups are registered to the concentrator numbers 0000h to 003fh. Re-create an ATgroup to add a new volume to the ATgroup. For ATgroup specifications, refer to 6.1 Specifications of ATgroup. 73

82 Chapter 4 Software Operation Methods Conditions for adding/deleting pair in ATgroup (i) Any pairs to be registered to an ATgroup must meet the following conditions. DDR pairs cannot be registered to an ATgroup. Only RDR pairs can be registered to an ATgroup. A pair that is not the top-level pair cannot be registered to an ATgroup. Only the top-level pair can be registered. If the MV of the pair to be registered includes several RDR pairs, it cannot be registered to an ATgroup. Each RDR pair must be set as a single pair registered to an ATgroup. A registered pair in an ATgroup and the pair to be registered to an ATgroup must be separated. An RDR pair that exists among different disk arrays cannot be registered to a single ATgroup. The RDR pair must be in the same disk array. Pairs cannot be registered when an RV is also used as a snapshot. It is required that an RV is not used as a snapshot. Concentrator number must be 003fh or less when registering a pair between the disk arrays whose ATgroup specifications are different with the ATgroup. If the MV or RV of the pair to be registered is a protected volume, it cannot be registered to an ATgroup. (ii) To add a pair to an ATgroup, the ATgroup sync state must be invalid, separated, or fault separation. (iii) To delete a registered pair from an ATgroup, the ATgroup sync state must be separated, or fault separation. (iv) The disk array for the ATgroup where a pair will be added or deleted on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (v) On the disk array for the ATgroup where a pair will be added or deleted on the concentrator side (MV side), replication functions must not been in frozen state. (vi) When adding and deleting a pair, the link state between disk arrays must be normal. (vii) In the case of the partition user, the management authorization for all the pairs registered in the ATgroup to which the pair is registered or from which the pair is deleted is required. 74

83 Chapter 4 Software Operation Methods Renamiing an ATgroup The ATgroup Rename screen is used to rename ATgroups. Items displayed on ATgroup Rename screen Perform either of the following steps to open the ATgroup Rename screen. In the ATgroup information screen, select the concentrator of the ATgroup where the name will be changed, then select [Operation] [ATgroup] [ATgroup Setting] [ATgroup Rename] in the menu. In the ATgroup information screen, select the concentrator of the ATgroup where the name will be changed, then right-click the mouse and select [ATgroup Setting] [ATgroup Rename]. (i) (ii) (iii) Figure 4-12 ATgroup Rename Screen [Description of ATgroup Rename screen] (i) Disk Array Name This indicates the name of the disk array on the concentrator side (MV side) where the ATgroup name will be changed. (ii) Current ATgroup Name This indicates the current ATgroup name that will be changed. (iii) New ATgroup Name Enter the new ATgroup name in this text box. The ATgroup name can include up to 32 characters consisting of alphanumeric characters, underbars (_) and/or forward slashes (/). After entering the name of the ATgroup to be created, click the [Execute] button to display a confirmation dialog before creating the ATgroup. 75

84 Chapter 4 Software Operation Methods Handling of errors when during ATgroup name revision When an error occurs during ATgroup name revision, the following message dialog appears. Be sure to execute handling for the respective error. Error Message [02043] Not all the link paths between the specified Disk Arrays are in the normal state. Table 4-5 Handling of Errors During ATgroup Revision Description of Error and Handling A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. [02237] Please input an ATgroup Name. [02238] Cannot use characters other than alphanumeric characters, "/" and "_". [02239] Cannot set the default ATgroup Name. [02240] Demanded process had already been executed. [02241] Cannot set due to monitor stopped or freezing. [02242] The same ATgroup Name exists. [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) The ATgroup name has not been entered. Enter the ATgroup name now. A prohibited character was entered. Enter only alphanumeric characters, underbars (_), or forward slashes (/) when specifying an ATgroup name (up to 32 characters). The default ATgroup name is reserved by the system. Enter a different ATgroup name. The old ATgroup name and new ATgroup name are the same. Enter a different ATgroup name. ism has suspended monitoring of or has frozen the target disk array. If ism has suspended monitoring, restart the disk array monitoring. If it has frozen the target disk array, defreeze it. The system already includes an ATgroup with the same name. Enter a unique ATgroup name. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for revising ATgroup name (i) When revising an ATgroup, the ATgroup name you enter cannot exceed 32 characters, can include only alphanumeric characters, underbars (_), and/or forward slashes, and must be unique within the system. (ii) The disk array for the ATgroup to be changed on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be changed on the concentrator side (MV side) must not have frozen replication functions. (iv) The link state for the disk array must be normal (when a pair has been registered to ATgroup). (v) In the case of the partition user, the management authorization for all the pairs registered in the ATgroup to be changed is required. 76

85 Chapter 4 Software Operation Methods Changiing tthe Allllowed Response Tiime off MV The Change of the Allowed Response Time of MV screen is used to change the allowed response time of MV. Items displayed on the Change of the Allowed Response Time of MV screen Perform either of the following steps to display the Change of Allowed Response Time of MV screen: From the ATgroup Information screen, select the concentrator of the ATgroup you want to change. Then, from the displayed menu, select [Operation] [ATgroup] [ATgroup Setting] [The Change of the Allowed Response Time of MV]. From the ATgroup Information screen, select the concentrator of the ATgroup you want to change, click the right-mouse button, and select [ATgroup Setting] [The Change of the Allowed Response Time of MV] from the displayed menu. (i) (ii) (iii) Figure 4-13 The Change of the Allowed Response Time of MV Screen [Description of the Change of the Allowed Response Time of MV screen] (i) ATgroup Name This indicates the name of the disk array on the concentrator side (MV side) on which the allowed response time of MV is to be changed. (ii) The Allowed Response Time of MV This indicates the allowed response time of MV range. (iii) Change Enter a new allowed response time of MV. You can specify a new allowed response time of MV within the allowed response 77

86 Chapter 4 Software Operation Methods time of MV range. Clicking the [Change] button displays the confirmation dialog box and changes the allowed response time of MV. Handling of errors when changing an allowed response time of MV If an allowed response time of MV change error occurs, the following message dialog will appear. Take an appropriate action for each error. Table 4-6 Handling of Errors When Changing the Allowed Response Time of MV Error Message Description of Error and Handling [02043] All the link paths between the specified Disk Arrays are in the abnormal state. A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. [02081] Specify a value within the range of <minimum-value> -<maximum-value>. [02241] Setting cannot be done because monitoring is suspended or frozen in the link destination. [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) A value that is not within the range is being used. Set an allowed response time of MV within the range of <minimum-value> to <maximum-value> seconds. ism has suspended monitoring of or has frozen the target disk array. If ism has suspended monitoring, restart the disk array monitoring. If it has frozen the target disk array, defreeze it. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for changing the allowed response time of MV (i) The allowed response time of MV to be changed must be specified within the range. (ii) The disk array for the ATgroup to be changed on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be changed on the concentrator side (MV side) must not have frozen replication functions. (iv) The link state for the disk array must be normal (when a pair has been registered to ATgroup). (v) In the case of the partition user, the management authorization for all the pairs registered in the ATgroup to be changed is required. 78

87 Chapter 4 Software Operation Methods Replliicattiing an ATgroup The ATgroup Replicate screen is used to execute ATgroup replication. Items displayed on ATgroup Replicate screen Perform either of the following steps to open the ATgroup Replicate screen. In the ATgroup information screen, select [ATgroup], then select [Operation] [ATgroup] [ATgroup Operation] [ATgroup Replicate] in the menu. In the ATgroup information screen, select [ATgroup], then right-click the mouse and select [ATgroup Operation] [ATgroup Replicate]. (i) (ii) (iv) (iii) (v) (vi) (vii) (viii) (ix) (x) Figure 4-14 ATgroup Replicate Screen 79

88 Chapter 4 Software Operation Methods [Description of ATgroup Replicate screen] (i) ATgroup Name This indicates the name of the ATgroup to be replicated. (ii) Type This indicates the ATgroup s location (concentrator or distributor). (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (iv) ATgroup Consistency State This indicates the ATgroup consistency state. (v) ATgroup Sync State This indicates the ATgroup sync state. (vi) Registered Volume List This section provides a list of pairs included in the ATgroup to be replicated, the status of each pair, and the replication results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). (vii) Range This indicates the copy range for each pair during replication. Settings Only Difference Only the differences between MV and RV for each pair will be copied. All The full range for each pair will be copied. (viii) Sync Mode of Copy Data Enter the copy mode to be used during replication. Settings Synchronous This sets synchronous copy mode. When the ATgroup sync state is replicate synchronous (Rpl/sync), the ATgroup consistency state becomes Atomic. Semi-synchronous This sets semi synchronous copy mode of order guarantee. When the ATgroup sync state is replicate synchronous (Rpl/sync), the ATgroup consistency state becomes Atomic, and copying is performed in a guaranteed order. Background Copy This sets background copy mode as the copy mode. When the ATgroup sync state is replicate synchronous (Rpl/sync), the ATgroup consistency state remains Non-atomic. (ix) RV Status This indicates the restriction for access from a host until replication is complete. Settings Not Ready This disables to perform operation for a volume from a host. 80

89 Chapter 4 Software Operation Methods Read Only This enables only reading a volume from a host. * You must be careful when specifying Read Only. Refer to the description about RV access restrictions in Data Replication User s Manual (Function Guide). (x) MV Access after Atomic-break This indicates MV access after an atomic-break. Settings MV Access Continue This enables access to MVs to be continued after an atomic-break is executed. MV Access Stop This sets MV access to be stopped after an atomic-break. The MV is normally stopped after an atomic-break, so this setting is normally not necessary. Click the [Execute] button to display a confirmation dialog before replicating the ATgroup. 81

90 Chapter 4 Software Operation Methods Handling of errors during ATgroup replication The following message dialog appears if an error occurs during ATgroup replication. Be sure to execute handling for the respective error. Error Message [02250] Failed in processing execution (No created volume) Table 4-7 Handling of Errors During ATgroup Replication Description of Error and Handling No (volume) pairs have been registered to the specified ATgroup. Register a pair. [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (MV Outside ism Management) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (Volume status is wrong) [02250] Failed in processing execution (All Link Path Abnormal) [02250] Failed in processing execution (Maximum Number of Concentrator Simultaneously Executing on Semi Synchronous Copy Mode arrived) [02250] Failed in processing execution (There are pairs without the management in the ATgroup) Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. The specified ATgroup s concentrator side (MV side) is not being managed by ism. Use ism that manages the ATgroup s concentrator side (MV side) to execute replication. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the ATgroup Replicate Execution screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the ATgroup Replicate Execution screen and re-execute replication. The ATgroup s sync state is unexecutable. Change to a sync state that does enable the processing. A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. It has reached the maximum number of concentrators executable in semi synchronous copy mode. Adjust the number to be executed within the limit. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. 82

91 Chapter 4 Software Operation Methods Conditions for replication of ATgroup (i) The ATgroup sync state must be one of the following. Separated Fault separation (Fault) Replicate suspend (Rpl/suspend) Replicate start (Rpl/start) (ii) When replication will be executed for an ATgroup whose ATgroup sync state is replicate suspend (Rpl/suspend), replication must be set before the ATgroup sync state becomes replicate suspend (Rpl/suspend). (iii) The disk array for the ATgroup to be created on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iv) The disk array for the ATgroup to be created on the concentrator side (MV side) must not have frozen replication functions. (v) The link state for the disk array must be normal. (vi) Any RVs included in the ATgroup must be unmounted from the system. (vii) When you specify semi synchronous copy mode, the maximum number of concentrators simultaneously executing in semi synchronous copy mode must not be exceeded. (viii) The update prevention state for the RVs included in the ATgroup must not be prevent. (ix) When using the asynchronous remote data replication function (RemoteDataRelication Asynchronous), the background copy mode must be specified. (x) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. Notes on replicating an ATgroup (i) When ATgroup replication is executed, the ATgroup consistency state becomes Non-atomic, regardless of the ATgroup s copy mode. When the ATgroup s copy mode is either synchronous mode or semi synchronous copy mode of order guarantee, if the ATgroup sync state is switched to replicate synchronous (Rpl/sync), the ATgroup consistency state will change to Atomic. When the copy mode is background copy mode, if the ATgroup sync state is switched to replicate synchronous (Rpl/sync), the ATgroup consistency state will remain Non-atomic. (ii) If the ATgroup s replication is executed while in background copy mode, an atomic-break will not be executed. (iii) If MV Access Stop is set for MV Access after Atomic-break, business operations may be stopped unexpectedly, so this setting should normally be avoided. (iv) When ATgroup replication is executed for an ATgroup whose sync state is replicate suspend (Rpl/suspend), any suspended copying is restarted. The ATgroup s replication settings specified before replicate suspend (Rpl/suspend) are set to MV Access after Atomic-break, RV Status, and Copy Mode. (v) There is an upper limit for the number of ATgroups that can be replicated concurrently in semi synchronous copy mode of order guarantee. Replication cannot be executed if the upper limit is exceeded. 83

92 Chapter 4 Software Operation Methods (vi) Before executing ATgroup replication, make sure that any RVs included in the ATgroup have been unmounted. If replication is executed while one or more RVs are mounted, the system s buffer image will no longer match the disk s image, which can result in abnormal RV data or abnormalities in the system connected to the RV. 84

93 Chapter 4 Software Operation Methods Separattiing an ATgroup The ATgroup Separate screen is used to separate ATgroups. Items displayed on ATgroup Separate screen Perform either of the following steps to open the ATgroup Separate screen. In the ATgroup information screen, select an ATgroup, then select [Operation] [ATgroup] [ATgroup Operation] [ATgroup Separate] in the menu. In the ATgroup information screen, select an ATgroup, then right-click the mouse and select [ATgroup Operation] [ATgroup Separate]. (iv) (i) (ii) (vi) (iii) (v) (vii) (viii) Figure 4-15 ATgroup Separate Screen [Description of ATgroup Separate screen] (i) ATgroup Name (ii) Type This indicates the name of the ATgroup to be separated. This indicates the ATgroup s location (concentrator or distributor). (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (iv) ATgroup Consistency State This indicates the ATgroup consistency state. 85

94 Chapter 4 Software Operation Methods (v) ATgroup Sync State (vi) Copy Mode (vii) Atomic-break mode (viii) Registered Volume List This indicates the ATgroup sync state. This indicates the ATgroup s copy mode. This indicates MV access after an atomic-break. This section provides a list of pairs included in the ATgroup to be separated, the status of each pair, and the separation results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). Click the [Execute] button to display a confirmation dialog before separating the ATgroup. Handling of errors during ATgroup separation The following message dialog appears if an error occurs during ATgroup separation. Be sure to execute handling for the respective error. Error Message [02250] Failed in processing execution (No created volume) Table 4-8 Handling of Errors During ATgroup Separation Description of Error and Handling No (volume) pairs have been registered to the specified ATgroup. Register a pair. [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (MV Outside ism Management) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (Volume status is wrong) [02250] Failed in processing execution (All Link Path Abnormal) Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. The specified ATgroup s concentrator side (MV side) is not being managed by ism. Use ism that manages the ATgroup s concentrator side (MV side) to execute separation. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the ATgroup Separate screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the ATgroup Separate screen and re-execute separation. The ATgroup s sync state is unexecutable. Change to a sync state that does enable the processing. A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. 86

95 Chapter 4 Software Operation Methods Error Message [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) Description of Error and Handling A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for separation of ATgroup (i) The ATgroup consistency state and ATgroup sync state must be one of the following. The ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) The ATgroup consistency state is Non-atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) The ATgroup consistency state is Non-atomic and the ATgroup sync state is replicating (Rpl/exec) The ATgroup consistency state is Non-atomic and the ATgroup sync state is separate start (Sep/start) (ii) The disk array for the ATgroup to be created on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be created on the concentrator side (MV side) must not have frozen replication functions. (iv) The link state for the disk array must be normal. (v) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. Notes on separating an ATgroup (i) If the ATgroup consistency state is Non-atomic when ATgroup separation is executed, separation is executed for each pair in the ATgroup. During this processing, the ATgroup consistency state remains Non-atomic. (ii) When the ATgroup sync state is separated, the access restriction on RVs included in the ATgroup is Read/write accessible. (iii) When the ATgroup consistency state is Non-atomic, pause point data can be maintained in an RV by unmounting the MV s file system or by executing using the database function s backup mode before separating the ATgroup. (iv) When separating ATgroup, Separation (immediate) which can use RV immediately after the separation start cannot be used. 87

96 Chapter 4 Software Operation Methods Resttoriing an ATgroup The ATgroup Restore screen is used for ATgroup restoration. Items displayed on ATgroup Restore screen Perform either of the following steps to open the ATgroup Restore screen. In the ATgroup information screen, select an ATgroup, then select [Operation] [ATgroup] [ATgroup Operation] [ATgroup Restore] in the menu. In the ATgroup information screen, select an ATgroup, then right-click the mouse and select [ATgroup Operation] [ATgroup Restore]. (i) (iv) (ii) (iii) (v) (vi) (vii) (viii) Figure 4-16 ATgroup Restore Screen [Description of ATgroup Restore screen] (i) ATgroup Name This indicates the name of the ATgroup to be restored. (ii) Type This indicates the ATgroup s location (concentrator or distributor). (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the 88

97 Chapter 4 Software Operation Methods (iv) ATgroup Consistency State (v) ATgroup Sync State (vi) Registered Volume List user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. This indicates the ATgroup consistency state. This indicates the ATgroup sync state. This section provides a list of pairs included in the ATgroup, each pair s state, and the restoration results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). (vii) Copy Range This indicates the copy range for each pair during restoration. Settings Only Difference Only the differences between MV and RV for each pair will be copied. All The full range for each pair will be copied. (viii) RV Status This indicates the restriction for access from a host until restoration is complete. Settings Not Ready This disables to perform operation for a volume from a host. Read Only This enables only reading a volume from a host. * You must be careful when specifying Read Only. Refer to the description about RV access restrictions in Data Replication User s Manual (Function Guide). Click the [Execute] button to display a confirmation dialog before restoring the ATgroup. 89

98 Chapter 4 Software Operation Methods Handling of errors during ATgroup restoration The following message dialog appears if an error occurs during ATgroup restoration. Be sure to execute handling for the respective error. Error Message [02250] Failed in processing execution (No created volume) Table 4-9 Handling of Errors During ATgroup Restoration Description of Error and Handling No (volume) pairs have been registered to the specified ATgroup. Register a pair. [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (MV Outside ism Management) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (Volume status is wrong) [02250] Failed in processing execution (All Link Path Abnormal) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. The specified ATgroup s concentrator side (MV side) is not being managed by ism. Use ism that manages the ATgroup s concentrator side (MV side) to execute replication. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the ATgroup Restore Execution screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the ATgroup Restore Execution screen and re-execute restoration. The ATgroup s sync state is unexecutable. Change to a sync state that does enable the processing. A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for restoration of ATgroup (i) The ATgroup sync state must be one of the following. Separated Restore suspend (Rst/suspend) (ii) The disk array for the ATgroup to be created on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be created on the concentrator side (MV side) must not have 90

99 Chapter 4 Software Operation Methods frozen replication functions. (iv) The link state for the disk array must be normal. (v) The MV and RVs must be unmounted from the system. (vi) The update prevention state for the MV and RVs included in the ATgroup must not be prevent. (vii) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. Notes on restoring an ATgroup (i) When an ATgroup is restored, it is only for an RV store (protect) operation that does not include any updating of RVs. When restoration of the ATgroup is completed, the ATgroup sync state becomes separated. (ii) When ATgroup restoration is executed, the access restriction setting for any RV included in the ATgroup becomes the specified state. When ATgroup restoration is completed and the ATgroup sync state becomes separated, the access restriction on RVs included in the ATgroup is Read/write accessible. (iii) Before executing ATgroup replication, make sure that any MVs or RVs included in the ATgroup have been unmounted. If replication is executed while one or more MVs or RVs are mounted, the system s buffer image will no longer match the disk s image, which can result in abnormal MV data or abnormalities in the system connected to the MV and RV. 91

100 Chapter 4 Software Operation Methods Forced Separattiion off an ATgroup The ATgroup Forced Separate screen is used to forcibly separate ATgroups. When using an RV in the ATgroup after a link fault has occurred or when the ATgroup itself is abnormal, forced separation of the ATgroup is executed on the concentrator side (MV side) and distributor side (RV side) respectively. Items displayed on ATgroup Forced Separate screen Perform either of the following steps to open the ATgroup Forced Separate screen. In the ATgroup information screen, select an ATgroup, then select [Operation] [ATgroup] [ATgroup Forced Operation] [ATgroup Forced Separate] in the menu. In the ATgroup information screen, select an ATgroup, then right-click the mouse and select [ATgroup Forced Operation] [ATgroup Forced Separate]. (i) (ii) (iv) (vi) (iii) (v) (vii) (viii) Figure 4-17 ATgroup Forced Separate Screen [Description of ATgroup Forced Separate screen] (i) ATgroup Name This indicates the name of the ATgroup to be forcibly separated. (ii) Type This indicates the ATgroup s location (concentrator or distributor). (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the 92

101 Chapter 4 Software Operation Methods (iv) ATgroup Consistency State (v) ATgroup Sync State (vi) Copy Mode (vii) Atomic-break mode (viii) Registered Volume List user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. This indicates the ATgroup consistency state. This indicates the ATgroup sync state. This indicates the ATgroup s copy mode. This indicates MV access after an atomic-break. This section provides a list of pairs included in the ATgroup to be forcibly separated, the status of each pair, and the forced separation results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). Click the [Execute] button to display a confirmation dialog before forcibly separating the ATgroup. Handling of errors during Forced separation of ATgroup The following message dialog appears if an error occurs during Forced separation of ATgroup. Be sure to execute handling for the respective error. Table 4-10 Handling of Errors During Forced Separation of ATgroup Error Message Description of Error and Handling [02250] Failed in processing execution (No created volume) No (volume) pairs have been registered to the specified ATgroup. Register a pair. [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the ATgroup Forced Separate screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the Forced Separation of ATgroup screen and re-execute forced separation. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. 93

102 Chapter 4 Software Operation Methods Conditions for Forced separation of ATgroup (i) The ATgroup sync state must not be invalid (Invalid), separated (Separated), fault separation (Fault), or fault recovering (Fault/recovering). (ii) The target disk array will be monitored by ism and must be in monitoring state. (iii) The target disk array must not have frozen replication functions. (iv) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. Caution point for Forced separation of ATgroup (i) Forced separation of ATgroup is performed on the concentrator side (MV side) and distributor side (RV side) respectively. 94

103 Chapter 4 Software Operation Methods Recovery ffrom Faulltt Separattiion Recovery from fault separation is performed on the Recover from Fault separation state screen. It is executed to make the ATgroup be in the separated state, which has been in the fault separation state due to a fault or forced operation. Items displayed on Recover from Fault separation state screen Perform either of the following steps to open the Recover from Fault separation state screen. In the ATgroup information screen, select ATgroup, then select [Operation] [ATgroup] [ATgroup Forced Operation] [Recover from Fault separation state] in the menu. In the ATgroup information screen, select ATgroup, then right-click the mouse and select [ATgroup Forced Operation] [Recover from Fault separation state]. (i) (ii) (iv) (iii) (v) (vi) Figure 4-18 Recover from Fault separation state Screen [Description of Recover from Fault separation state screen] (i) ATgroup Name This indicates the name of the ATgroup to be recovered from fault separation. (ii) Type This indicates the ATgroup s location (concentrator or distributor). 95

104 Chapter 4 Software Operation Methods (iii) Number of Registered Volume (iv) ATgroup Consistency State (v) ATgroup Sync State (vi) Registered Volume List This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. This indicates the ATgroup consistency state. This indicates the ATgroup sync state. This section provides a list of pairs included in the target ATgroup, the status of each pair, and the execution results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). Click the [Execute] button to display a confirmation dialog before recovering from fault separation. Handling of errors when recovering from fault separation The following message dialog appears if an error occurs when recovering from fault separation. Be sure to execute handling for the respective errors. Table 4-11 Handling of Errors when Recovering from Fault Separation Error Message Description of Error and Handling [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the Recover from Fault separation state screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the Recover from Fault separation state screen and then re-execute. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. 96

105 Chapter 4 Software Operation Methods Conditions for recovery from fault separation (i) The ATgroup sync state must be fault separation state. (ii) The operation target disk array will be monitored by ism and must be in monitoring state. (iii) The operation target disk array must not have frozen replication functions. (iv) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. 97

106 Chapter 4 Software Operation Methods Forced Dellettiion off an ATgroup The ATgroup Forced Deletion screen is used to forcibly delete ATgroups. Forced deletion of ATgroup is executed when an ATgroup becomes abnormal due to a fault, etc., or when invalid ATgroup information remains on the distributor side (RV side). Forced deletion of ATgroup is executed from the distributor side (RV side). Also, if invalid ATgroup information remains on the concentrator side (MV side), the ATgroup is first separated, or is first forcibly separated from the concentrator side (MV side), so that the ATgroup can then be deleted. Next, Forced deletion of ATgroup is executed in order to delete the invalid ATgroup information. Items displayed on ATgroup Forced Deletion screen Perform either of the following steps to open the ATgroup Forced Deletion screen. In the ATgroup information screen, select the ATgroup, then select [Operation] [ATgroup] [ATgroup Forced Operation] [ATgroup Forced Deletion] in the menu. In the ATgroup information screen, select the ATgroup, then right-click the mouse and select [ATgroup Forced Operation] [ATgroup Forced Deletion]. (i) (ii) (iv) (iii) (v) Figure 4-19 ATgroup Forced Deletion Screen [Description of ATgroup Forced Deletion screen] (i) ATgroup Name This indicates the name of the ATgroup to be forcibly deleted. (ii) Type This indicates the ATgroup s location. (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (iv) ATgroup Consistency State This indicates the ATgroup consistency state. (v) ATgroup Sync State This indicates the ATgroup sync state. Click the [Execute] button to display a confirmation dialog before forcibly deleting the ATgroup. 98

107 Chapter 4 Software Operation Methods Handling of errors during Forced deletion of ATgroup The following message dialog appears if an error occurs during forced deletion of an ATgroup. Be sure to execute handling for the respective error. Error Message [02064] Volume Deletion was failed. Table 4-12 Handling of Errors During Forced Deletion of ATgroup Description of Error and Handling The disk array is abnormal. Check the state of the disk array. Recover the disk array and restore ATgroup to normal state. [02065] ATgroup Deletion was failed. [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (ATgroup status is wrong: execution condition disagreement) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) Failed in deleting ATgroups. Check the state of the unit and re-execute. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. The specified ATgroup is not normal or ATgroup sync state is not in the executable state. If the specified ATgroup is not normal, restore it to the normal state. If ATgroup sync state is not the executable state, change it to the executable state. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for Forced deletion of ATgroup (i) When deleting the distributor (a) The disk array on the ATgroup s distributor side (RV side) will be monitored by ism and must be in monitoring state. (b) The disk array on the ATgroup s distributor side (RV side) must not have frozen replication functions. (c) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. (ii) When deleting the concentrator (a) The disk array on the ATgroup s concentrator side (MV side) will be monitored by ism and must be in monitoring state. (b) Replication functions of the disk array on the ATgroup s concentrator side (MV side) is not in the freeze state. (c) ATgroup sync state of the specified ATgroup is separated, fault separation, or invalid. (d) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. 99

108 Chapter 4 Software Operation Methods Caution point for Forced deletion of ATgroup Forced deletion of ATgroup is executed when an ATgroup becomes abnormal due to a fault, etc., or when invalid ATgroup information remains on the distributor side (RV side). Forced deletion of ATgroup is executed from the distributor side (RV side). Also, if invalid ATgroup information remains on the concentrator side (MV side), the ATgroup is first separated, or is first forcibly separated from the concentrator side (MV side), so that the ATgroup can then be deleted. Next, Forced deletion of ATgroup is executed in order to delete the invalid ATgroup information Outtputtttiing tthe ATgroup IInfformattiion Liistt iin CSV fformatt The ATgroup information list that is displayed in the ATgroup information screen can be output in CSV format by selecting [File] [CSV Output of Information List] in the ATgroup information screen s menu. For details on outputting this list information display in CSV format, refer to the Data Replication User s Manual (Function Guide). In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. 100

109 Chapter 4 Software Operation Methods Saviing an ATgroup and Regiisttered Paiir IInfformattiion Both pair information and the information about pairs registered to an ATgroup can be saved by selecting [File] [Save the Pair Setting Information]. This saved ATgroup and registered pair information can be used to make batch settings via Storage Manager Configuration Setting screen. For details on saving pair information, refer to the Data Replication User s Manual (Function Guide). For details on batch settings, refer to the Configuration Setting Tool User s Manual (GUI). In the case of a partition user, if the user does not have the management authorization for some pairs in the ATgroup, the setting information on those pairs cannot be saved. Only the information on the pairs that the partition user has the management authorization is saved. When you attempt such an operation, the following message screen appears. To save all the pair information, the storage user must perform the operation. Figure 4-20 Registration Pair Information Saving Confirmation Screen When a partition user uses the [Replication Setting] function of the [Batch Setting] in configuration setting with the file saved by the storage user, the partition user can set only the pairs for which the user has the authority. 101

110 Chapter 4 Software Operation Methods 4..2 ReplliicatiionControll//DiisasterRecovery Command Operatiion Methods (CLII) Commands provided by ReplicationControl/DisasterRecovery can be used to perform the following operations. ATgroup operations ATgroup operations include building an ATgroup, ATgroup replication control, and ATgroup status checking. Remote operation Replication control of pairs established within a remote disk array that is not directly connected to the business application server erver can be executed, along with replication control of ATgroups. Table 4-13 List of ATgroup Operation Commands Command Name Operation Description Section ismrc_replicate Replicate Executes replication of ATgroup ismrc_separate Separate Executes separation of ATgroup ismrc_restore Restore Executes restoration of ATgroup ismrc_change Recovery from fault Executes recovery from fault separation separation ismrc_wait Wait for change in ATgroup sync state Waits for change in ATgroup sync state ismrc_query Display ATgroup s copy state Displays the copy state of ATgroup and pairs registered to ATgroup ismrc_sense Display ATgroup list Display ATgroup list information ismrc_atg Build ATgroup Creates/deletes ATgroup, or registers/deletes pairs in ATgroup Table 4-14 List of Remote Operation Commands Command Name Operation Description Section ismrc_replicate Replicate Executes replication of remote pair. ismrc_separate Separate Executes separation of remote pair. ismrc_restore Restore Executes restoration of remote pair. ismrc_change Change copy mode Changes remote pair s copy mode ismrc_wait Wait for state change Waits for change in remote pair s state. ismrc_query Display copy state Displays remote pair s copy state. ismrc_sense Display setting information Displays remote pair s setting information. 102

111 Chapter 4 Software Operation Methods Replliicattiing an ATgroup ((iismrc_replliicatte)) To replicate an ATgroup, specify the target ATgroup in the ismrc_replicate command and execute the command. Executable construct of ismrc_replicate command (Executable construct of replicate ATgroup command) ismrc_replicate -atg ATgroup name [-cprange diff full ] [ -atmode continue stop ] [-cpmode sync semi bg ] [-rvacc rv_access ] [-nowait -wait [sense interval] ] [-arrayname disk array name ] [-iopath direct manager ] (Executable construct of ordinary pair replication) ismrc_replicate -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] [-cprange copy_range ] [-cpmode copy_mode ] [-rvacc rv_access ] [-nowait -wait [second] ] [ -iopath iopath ] ismrc_replicate -file file_name [-cprange copy_range ] [-cpmode copy_mode ] [-rvacc rv_access ] [-nowait -wait [second] ] [ -iopath iopath ] (Executable construct of command version check) ismrc_replicate -ver (Executable construct of command usage display) ismrc_replicate -?...(Windows) ismrc_replicate -help...(unix) The following describes replication of ATgroups. For description of normal pair replication, refer to the ControlCommand Command Reference. Specifiable options and parameters for ATgroup replication (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required for execution of ATgroup replication. (ii) Specification of copy range -cprange diff full This specifies the copy range. When diff is specified, only the differential between each pair s pair MV and RV data will be copied. When full is specified, the entire range of data in each pair s MV and RV will be copied. The default value when this option is omitted is diff. 103

112 Chapter 4 Software Operation Methods (iii) Specification of MV access after atomic-break -atmode continue stop This specifies enabling MV access after an atomic-break. When continue is specified, MV access continue is set and MV access is enabled even after an atomic-break has been executed. When stop is specified, an MV access stop is set so that MV access is disabled after an atomic-break has been executed. Ordinarily, stop should not be specified since setting an MV access stop can result in unexpected stops in business operations. The default value when this option is omitted is continue. (iv) Specification of copy mode -cpmode sync semi bg This specifies the copy mode during replication. When sync is specified, synchronous mode is set. When semi is specified, semi synchronous copy mode of order guarantee is set. When bg is specified, background copy mode is set. The default value when this option is omitted is sync. (v) Specification of RV access restriction -rvacc ro nr This specifies the RV access restriction. When ro is specified, only referencing (reading) RV is permitted. When nr is specified, accessing RV is not permitted. The default value when this option is omitted is nr. (vi) Specification of wait for copy completion -nowait -wait [sense interval] This specifies whether or not to wait for completion of copying before using a command to change the ATgroup sync state to replicate synchronous (Rpl/sync). When -nowait is specified, the ATgroup Replicate Execution command can be executed immediately, without waiting for completion of copying. When -wait is specified, execution of this command is held pending until copying is completed. When a sense interval value is specified after -wait, completion of copying is monitored for the interval time set by the value. The specifiable range for this value is 1 to 30 (seconds). If no value is specified here, the value set via the replication control option settings file (Windows) or by the RPL_WATCHDEV environment variable (UNIX) becomes valid. If neither sense interval nor RPL_WATCHDEV is specified, the default sense interval is 5 seconds. The default value when this option is omitted is -nowait. 104

113 Chapter 4 Software Operation Methods (vii) Specification of disk array name -arrayname disk array name (viii) Specification of I/O issue path -iopath direct manager This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Messages output during ATgroup replication (Windows) [Start message] Replicate Start ATG:atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Replicating... [End message] Replicate Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Start message] Replicate Start YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path 105

114 Chapter 4 Software Operation Methods [Message during execution] Replicating... [End message] Replicate Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path Description of message YYYY/MM/DD hh:mm:ss atgroup disk_number ld_name type volume name path Replicating... Start/end date/time (YYYY/MM/DD hh:mm:ss) ATgroup name Disk number Logical disk name Use format Mounting point volume name Drive path (mounted to drive letter or NTFS folder) This message is displayed to indicate when ATgroup replication is being executed. The trailing dots are refreshed until the ATgroup sync state becomes replicate synchronous. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of copying. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the [WAIT] section in the replication control option settings file. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Start message] Replicate Start ATG: atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Replicating

115 Chapter 4 Software Operation Methods [End message] Replicate Normal End ATG: atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Start message] Replicate Start YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type [Message during execution] Replicating... [End message] Replicate Normal End YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type Description of message YYYY/MM/DD hh:mm:ss atgroup special_file_name ld_name type Replicating... Start/end date/time (YYYY/MM/DD hh:mm:ss) ATgroup name Special file name Logical disk name Use format This message is displayed to indicate when ATgroup replication is being executed. The trailing dots are refreshed until the ATgroup sync state becomes replicate synchronous. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of copying. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the environment variable. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. 107

116 Chapter 4 Software Operation Methods Conditions for replication of ATgroup (i) The target ATgroup s sync state must be one of the following. Separated Fault separation (Fault) Replicate suspend (Rpl/suspend) Replicate start (Rpl/start) (ii) When replication is executed for an ATgroup whose ATgroup sync state is replicate start (Rpl/start) or replicate suspend (Rpl/suspend), MV access after atomic-break, RV access restriction, and copy mode should not be set, or the setting at replication starting must be specified. (iii) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (iv) Creation and updating of the volume list must be completed. (v) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (vi) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. (vii) The link state for the disk array must be normal. (viii) Any RVs included in the ATgroup must be unmounted from the system. (ix) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (x) The MV or RV in the ATgroup must not be in the not in use state. (xi) The pool to which the MV or RV in the ATgroup belongs must not be in the rotation stop state. (xii) The RV in the ATgroup must not be in the prevent state. However, the above conditions (iii) to (v) are not applied to the operation via ism. Notes on replicating an ATgroup (i) When ATgroup replication is executed, the ATgroup consistency state becomes Non-atomic, regardless of the ATgroup s copy mode. When the ATgroup s copy mode is either synchronous mode or semi synchronous copy mode of order guarantee, if the ATgroup sync state is switched to replicate synchronous (Rpl/sync), the ATgroup consistency state will change to Atomic. When the copy mode is background copy mode, if the ATgroup sync state is switched to replicate synchronous (Rpl/sync), the ATgroup consistency state will remain Non-atomic. (ii) If the ATgroup s replication is executed while in background copy mode, an atomic-break will not be executed. 108

117 Chapter 4 Software Operation Methods (iii) If MV Access Stop is set for MV Access after Atomic-break, business operations may be stopped unexpectedly, so this setting should normally be avoided. (iv) When ATgroup replication is executed for an ATgroup whose sync state is replicate suspend (Rpl/suspend), any suspended copying is restarted. (v) There is an upper limit for the number of the ATgroups that can be replicated concurrently in semi synchronous copy mode of order guarantee. Replication cannot be executed when this limit is exceeded. (vi) Before executing ATgroup replication, make sure that any RVs included in the ATgroup have been unmounted. If replication is executed while one or more RVs are mounted, the system s buffer image will no longer match the disk s image, which can result in abnormal RV data or abnormalities in the system connected to the RV. (vii) If a UNIX server is used, the operation privilege immediately after installation is root authority. Operations using privilege levels other than root authority can also be enabled by changing the operation privilege for the command and directory. 109

118 Chapter 4 Software Operation Methods Separattiing an ATgroup ((iismrc_separatte)) To separate an ATgroup, specify the target ATgroup in the ismrc_separate command and execute the command. Executable construct of ismrc_separate command (Executable construct of ATgroup separation) ismrc_separate -atg ATgroup name [-force ] [-nowait -wait [sense interval] ] [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of ordinary pair separation) ismrc_separate -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] [-rvacc rv_access ] [-rvuse rv_use -force force_arg ] [-nowait -wait [second] ] [ -iopath iopath ] ismrc_separate -file file_name [-rvacc rv_access ] [-rvuse rv_use -force force_arg ] [-nowait -wait [second] ] [ -iopath iopath ] (Executable construct of command version check) ismrc_separate -ver (Executable construct of command usage display) ismrc_separate -?... (Windows) ismrc_separate -help... (UNIX) The following describes separation of ATgroups. For description of normal pair separation, refer to the ControlCommand Command Reference. Specifiable options and parameters for ATgroup separation (i) Specification of ATgroup (required) -atg ATgroup name (ii) Specification of forced separation -force This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required for execution of ATgroup separation. This option is specified to forcibly separate an ATgroup. It is used when normal separation cannot be executed, such as when using an RV in an ATgroup after a link fault has occurred, or when a system fault or a line fault between 110

119 Chapter 4 Software Operation Methods disk arrays has occurred. If this option is omitted, normal separation is executed. (iii) Specification of wait for completion of ATgroup separation -nowait -wait [sense interval] This specifies whether or not to wait for completion of ATgroup separation before using a command to change the ATgroup sync state to separated. When -nowait is specified, the ATgroup Separate command can be executed immediately, without waiting for completion of separation. When -wait is specified, execution of this command is held pending until separation is completed. When a sense interval value is specified after -wait, completion of ATgroup separation is monitored for the interval time set by the value. The specifiable range for this value is 1 to 30 (seconds). If no value is specified here, the value set via the replication control option settings file (Windows) or by the RPL_WATCHDEV environment variable (UNIX) becomes valid. If neither sense interval nor RPL_WATCHDEV is specified, the default sense interval is 5 seconds. The default value when this option is omitted is -nowait. (iv) Specification of disk array name -arrayname disk array name This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. (v) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Messages output during ATgroup separation (Windows) [Start message] Separate Start ATG:atgroup YYYY/MM/DD hh:mm:ss 111

120 Chapter 4 Software Operation Methods [Message during execution] Separating... [End message] Separate Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Start message] Separate Start YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path [Message during execution] Separating... [End message] Separate Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path Description of message YYYY/MM/DD hh:mm:ss atgroup disk_number ld_name type Start/end date/time (YYYY/MM/DD hh:mm:ss) ATgroup name Disk number Logical disk name Use format 112

121 Chapter 4 Software Operation Methods volume name path Separating... Mounting point volume name Drive path (mounted to drive letter or NTFS folder) This message is displayed to indicate when ATgroup separation is being executed. The trailing dots are refreshed until the ATgroup s separation is completed. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of copying. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the [WAIT] section in the replication control option settings file. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Start message] Separate Start ATG:atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Separating... [End message] Separate Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Start message] Separate Start YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type [Message during execution] Separating

122 Chapter 4 Software Operation Methods [End message] Separate Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: special_file_name ld_name type RV: special_file_name ld_name type Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name special_file_name Special file name ld_name Logical disk name type Use format Separating... This message is displayed to indicate when ATgroup separation is being executed. The trailing dots are refreshed until the ATgroup has been separated. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of separation. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the environment variable. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for separation of ATgroup (i) This command s target ATgroup consistency state and the ATgroup sync state must be one of the following. However, this does not apply to forced separation (when the -force option has been specified). The ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) The ATgroup consistency state is Non-atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) The ATgroup consistency state is Non-atomic and the ATgroup sync state is replicating (Rpl/exec) The ATgroup consistency state is Non-atomic and the ATgroup sync state is separate start (Sep/start) 114

123 Chapter 4 Software Operation Methods (ii) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (iii) Creation and updating of the volume list must be completed. (iv) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (v) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. However, this is not applied to forced separation when the -force option has been specified. (vi) The link state for the disk array must be normal. However, this does not apply to forced separation (when the -force option has been specified). (vii) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (viii) The MV or RV in the ATgroup must not be in the not in use state. (ix) The pool to which the MV or RV in the ATgroup belongs must not be in the rotation stop state. However, the above conditions (ii) to (iv) are not applied to the operation via ism. Notes on reparating an ATgroup (i) If the ATgroup consistency state is Non-atomic when ATgroup separation is executed, separation is executed for each pair in the ATgroup. During this processing, the ATgroup consistency state remains Non-atomic. (ii) When the ATgroup sync state is separated, the access restriction on RVs included in the ATgroup is Read/write accessible. (iii) When the ATgroup sync state is Atomic and forced separation is performed, the access restriction for RVs in the ATgroup remains in Read Only or Not Ready. To enable read/write access to RVs in the ATgroup, execute the operation to recover from fault separation on the distributor side (RV side). Recovery from fault separation also changes the ATgroup sync state to separated. (iv) When the ATgroup consistency state is Non-atomic, pause point data can be maintained in an RV by unmounting the MV s file system or by executing using the database function s backup mode before separating the ATgroup. (v) When volumes on both the concentrator side (MV side) and the distributor side (RV side) are not being recognized from the save server, if Forced separation of ATgroup is executed by specifying the -force option, volumes on the concentrator side (MV side) and distributor side (RV side) will be recognized and executed by the respective servers. (vi) Immediately after ATgroup separation, the RV can be used, so RV Separate (immediate) cannot be used. 115

124 Chapter 4 Software Operation Methods (vii) If a UNIX server is used, the operation privilege immediately after installation is root authority. Operations using privilege levels other than root authority can also be enabled by changing the operation privilege for the command and directory. 116

125 Chapter 4 Software Operation Methods Resttoriing an ATgroup ((iismrc_resttore)) To restore an ATgroup, specify the target ATgroup in the ismrc_restore command and execute the command. Executable construct of ismrc_restore command (Executable construct of ATgroup restoration) ismrc_restore -atg ATgroup name [-cprange diff full ] [-rvacc rv_access ] [-nowait -wait [sense interval] ] [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of normal pair restoration) ismrc_restore -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] [-cprange copy_range ] [-cpmode copy_mode ] [-rvacc rv_access ] [-nowait -wait [second] ] [-mode operation_mode ] [ -iopath iopath ] ismrc_restore -file file_name [-cprange copy_range ] [-cpmode copy_mode ] [-rvacc rv_access ] [-nowait -wait [second] ] [-mode operation_mode ] [ -iopath iopath ] (Executable construct of command version check) ismrc_restore -ver (Executable construct of command usage display) ismrc_restore -?...(Windows) ismrc_restore -help...(unix) The following describes restoration of ATgroups. For description of normal pair restoration, refer to the ControlCommand Command Reference. Specifiable options and parameters for ATgroup restoration (i) Specification of ATgroup (required) -atg ATgroup name (ii) Specification of copy range -cprange diff full This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required for execution of ATgroup restoration. This specifies the copy range. When diff is specified, only the differential between each pair s pair MV and RV data will be copied. When full is specified, the entire range of 117

126 Chapter 4 Software Operation Methods data in each pair s MV and RV will be copied. The default value when this option is omitted is diff. (iii) Specification of RV access restriction -rvacc ro nr This specifies the RV access restriction. When ro is specified, only referencing (reading) RV is permitted. When nr is specified, accessing RV is not permitted. The default value when this option is omitted is nr. (iv) Specification of wait for completion of ATgroup restoration -nowait -wait [sense interval] This specifies whether or not to wait for completion of ATgroup restoration before using a command to change the ATgroup sync state to restored. When -nowait is specified, the ATgroup Restore Execution command can be executed immediately, without waiting for completion of restoration. When -wait is specified, execution of this command is held pending until restoration is completed. When a sense interval value is specified after -wait, completion of ATgroup restoration is monitored for the interval time set by the value. The specifiable range for this value is 1 to 30 (seconds). If no value is specified here, the value set via the replication control option settings file (Windows) or by the RPL_WATCHDEV environment variable (UNIX) becomes valid. If neither sense interval nor RPL_WATCHDEV is specified, the default sense interval is 5 seconds. The default value when this option is omitted is -nowait. (v) Specification of disk array name -arrayname disk array name This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. (vi) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. 118

127 Chapter 4 Software Operation Methods Messages output during ATgroup restoration (Windows) [Start message] Restore Start ATG:atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Restoring... [End message] Restore Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Start message] Restore Start YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path [Message during execution] Restoring... [End message] Restore Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path 119

128 Chapter 4 Software Operation Methods Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name disk_number Disk number ld_name Logical disk name type Use format volume name Mounting point volume name path Drive path (mounted to drive letter or NTFS folder) Restoring... This message is displayed to indicate when ATgroup restoration is being executed. The trailing dots are refreshed until the ATgroup s restoration is completed. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of restoration. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the environment variable. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Start message] Restore Start ATG:atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Restoring... [End message] Restore Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Start message] Restore Start YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type 120

129 Chapter 4 Software Operation Methods [Message during execution] Restoring... [End message] Restore Normal End YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name special_file_name Special file name ld_name Logical disk name type Use format Restoring... This message is displayed to indicate when ATgroup restoration is being executed. The trailing dots are refreshed until the ATgroup s restoration is completed. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of restoration. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the environment variable. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for restoration of ATgroup (i) This command s target ATgroup sync state must be one of the following. Separated Restore suspend (Rst/suspend) (ii) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (iii) Creation and updating of the volume list must be completed. (iv) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (v) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. 121

130 Chapter 4 Software Operation Methods (vi) The link state for the disk array must be normal. (vii) The MV and RVs must be unmounted from the system. (viii) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (ix) The MV or RV in the ATgroup must not be in the not in use state. (x) The pool to which the MV or RV in the ATgroup belongs must not be in the rotation stop state. (xi) The MV in the ATgroup must not be in the prevent state. However, the above conditions (ii) to (iv) are not applied to the operation via ism. Notes on restoring an ATgroup (i) ATgroup restoration is performed only as RV restore (protect). When ATgroup restoration is completed, the ATgroup sync state changes to separated. (ii) When ATgroup restoration is completed, the ATgroup sync state changes to separated and the access restriction on RVs included in the ATgroup is read/write accessible. (iii) Before executing ATgroup restoration, make sure that any MVs or RVs included in the ATgroup have been unmounted. If ATgroup restoration is executed while one or more MVs or RVs are mounted, the system s buffer image will no longer match the disk s image, which can result in abnormal MV data or abnormalities in the system connected to the MV and RV. (iv) If a UNIX server is used, the operation privilege immediately after installation is root authority. Operations using privilege levels other than root authority can also be enabled by changing the operation privilege for the command and directory. 122

131 Chapter 4 Software Operation Methods Recovery ffrom Faulltt Separattiion ((iismrc_change)) Recovery from fault separation, specify the target ATgroup in the ismrc_change command and execute the command. Recovery from fault separation is executed when using MVs or RVs that have been in the fault separation state due to a fault or forced operation. Executable construct of ismrc_change command (Executable construct of recovery from fault separation) ismrc_change -atg ATgroup name -force separated [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of copy control state change for ordinary pair) ismrc_change -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] -suspend -resume -sync -semi -bg [-iopath iopath ] ismrc_change -file file_name -suspend -resume -sync -semi -bg [-iopath iopath ] (Executable construct of command version check) ismrc_change -ver (Executable construct of command usage display) ismrc_change -?...(Windows) ismrc_change -help...(unix) Recovery from fault separation is described below. For description of copy control state changes for ordinary pairs, refer to the ControlCommand Command Reference. Specifiable options and parameters for recovery from fault separation (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required when recovering from fault separation. (ii) Specification of ATgroup sync state change (required) -force separated This option is specified to forcibly change an ATgroup s active mode. When recovering from fault separation, specifying separated forcibly changes the ATgroup sync state to the separated state, which enables to use MV or RV. Specification of this option is required when recovering 123

132 Chapter 4 Software Operation Methods (iii) Specification of disk array name -arrayname disk array name (iv) Specification of I/O issue path -iopath direct manager from fault separation. This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Messages output when recovering from fault separation (Windows) [Start message] Change Start ATG:atgroup YYYY/MM/DD hh:mm:ss [End message] Change Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Start message] Change Start YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path 124

133 Chapter 4 Software Operation Methods [End message] Change Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name disk_number Disk number ld_name Logical disk name type Use format volume name Mounting point volume name path Drive path (mounted to drive letter or NTFS folder) For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Start message] Change Start ATG:atgroup YYYY/MM/DD hh:mm:ss [End message] Change Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Start message] Change Start YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type 125

134 Chapter 4 Software Operation Methods [End message] Change Normal End YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name special_file_name Special file name ld_name Logical disk name type Use format For detailed description of this command s display items and return values, and environment variables, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for recovery from fault separation (i) This command s target ATgroup sync state must be fault separation. (ii) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (iii) Creation and updating of the volume list must be completed. (iv) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (v) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. (vi) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. However, the above conditions (ii) to (iv) are not applied to the operation via ism. Notes for recovery from fault separation (i) If a UNIX server is used, the operation privilege immediately after installation is root authority. Operations using privilege levels other than root authority can also be enabled by changing the operation privilege for the command and directory. 126

135 Chapter 4 Software Operation Methods Waiittiing ffor tthe ATgroup Sync Sttatte tto Change ((iismrc_waiitt)) To wait for a change in the ATgroup sync state, specify the target ATgroup in the ismrc_wait command and execute the command. This command to wait for change in ATgroup sync state is executed in order to wait for the ATgroup sync state to change from Replicate ATgroup, Separate ATgroup, or Restore ATgroup. Once the sync state changes to the target sate, execution of this command is ended normally. Executable construct of ismrc_wait command (Executable construct of wait for change in ATgroup sync state) ismrc_wait -atg ATgroup name -cond sync sep fault atsep atfault [ -interval sense interval ] [ -limit limit time nolim ] [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of wait for completion of normal pair replication, restoration, or separation) ismrc_wait -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] -cond condition [-interval interval_time ] [-limit limit_time ] [ -iopath iopath ] ismrc_wait -rv volume -rvflg rv_flg -cond condition [-interval interval_time ] [-limit limit_time ] [ -iopath iopath ] ismrc_wait -file file_name -cond condition [-interval interval_time ] [-limit limit_time ] [ -iopath iopath ] (Executable construct of command version check) ismrc_wait -ver (Executable construct of command usage display) ismrc_wait -?...(Windows) ismrc_wait -help...(unix) Waiting for in a change in the ATgroup sync state is described below. For description of waiting for completion of normal pair replication, restoration, or separation, refer to the ControlCommand Command Reference. 127

136 Chapter 4 Software Operation Methods Specifiable options and parameters for wait for change in ATgroup sync state (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required when executing wait for change in ATgroup sync state. (ii) Specification of condition for ending wait (required) -cond sync sep fault atsep atfault This specifies conditions for ending a wait. When sync is specified, this command terminates normally when ATgroup sync state becomes replicate synchronous (Rpl/sync). When sep is specified, this command terminates normally when the ATgroup sync state becomes separated. When fault is specified, this command terminates normally when the ATgroup sync state becomes either separated or fault separation. When atsep is specified, this command terminates normally when the ATgroup consistency state becomes Atomic and the ATgroup sync state becomes separated. When atfault is specified, this command terminates normally when the ATgroup consistency state becomes Atomic and the ATgroup sync state becomes separated or fault separation. This option is required when waiting for a change in the ATgroup sync state. (iii) Specification of sense interval -interval sense interval This specifies the sense (monitoring) interval when waiting for a change in the ATgroup sync state. Any integer from 1 to 30 (seconds) can be specified in sense interval as the sense interval. The default value when this option is omitted is the value set by replication control option settings file (Windows) or the RPL_WATCHDEV environment variable (UNIX). If neither sense interval nor RPL_WATCHDEV has been specified, the sense interval becomes 5 seconds. (iv) Specification of monitoring limit time -limit limit time nolim This specifies the limit time when waiting for a change in the ATgroup sync state. Any integer from 0 to (seconds) can be specified in limit time as the limit time, or nolim (unlimited) can be specified. The default value when this option is omitted is the value set by the replication control option settings file (Windows) or the environment variable RPL_LIMWATCHDEV (UNIX). If neither this 128

137 Chapter 4 Software Operation Methods (v) Specification of disk array name -arrayname disk array name option nor RPL_LIMWATCHDEV has been specified, the wait time is unlimited. This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. (vi) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Messages output when waiting for change in ATgroup sync state (Windows) [Message during execution] Waiting... [End message] activity_state Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the end message by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Message during execution] Waiting... [End message] activity_state Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name 129

138 Chapter 4 Software Operation Methods path RV: disk_number ld_name type volume name path Description of message activity_state Activity state of pair for which wait has ended YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name disk_number Disk number ld_name Logical disk name type Use format volume name Mounting point volume name path Drive path (mounted to drive letter or NTFS folder) Waiting... This message is displayed to indicate when waiting is in progress. The trailing dots are refreshed until the waiting has ended. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the [WAIT] section in the replication control option settings file. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Message during execution] Waiting... [End message] activity_state Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the end message by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Message during execution] Waiting... [End message] activity_state Normal End YYYY/MM/DD hh:mm:ss 130

139 Chapter 4 Software Operation Methods ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type Description of message activity_state Activity state of pair for which wait has ended YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name special_file_name Special file name ld_name Logical disk name type Use format Waiting... This message is displayed to indicate when waiting is in progress. The trailing dots are refreshed until the waiting has ended. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the [WAIT] section in the replication control option settings file. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for waiting for change in ATgroup sync state (i) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (ii) Creation and updating of the volume list must be completed. (iii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (iv) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. (v) The link state for the disk array must be normal. (vi) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. The above conditions (i) to (iii) are not applied to the operation via ism. 131

140 Chapter 4 Software Operation Methods Notes on waiting for the ATgroup sync state to change (i) While waiting, if the ATgroup sync state becomes replicate suspend (Rpl/suspend) or restore suspend (Rst/suspend), the wait will stop and end abnormally. (ii) While waiting, if any of the following occurs, the wait will stop and end abnormally. Abnormal link ATgroup link state is detected Copy of ATgroup s differential quantity of volume cannot be obtained Frozen target disk array is detected (iii) If a UNIX server is used and the operation privilege immediately after installation is root authority. This operation can be enabled for a privilege level other than root authority by changing the command and directory operation privilege. 132

141 Chapter 4 Software Operation Methods Diispllayiing tthe ATgroup Copy Sttatte ((iismrc_query)) To display an ATgroup s copy state, specify the target ATgroup in the ismrc_query command and execute the command. This command is used in order to display a particular ATgroup s copy state. Executable construct of ismrc_query command (Executable construct of ATgroup copy state display) ismrc_query -atg ATgroup name [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of replication state display for normal pair) ismrc_query -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] [ -iopath iopath ] ismrc_query -rv volume -rvflg rv_flg [-iopath iopath ] ismrc_query -file file_name [-iopath iopath ] (Executable construct of command version check) ismrc_query -ver (Executable construct of command usage display) ismrc_query -?...(Windows) ismrc_query -help...(unix) The following describes the ATgroup s copy state. For description of displaying the replication state of normal pairs, refer to the ControlCommand Command Reference. Options and parameters available for ATgroup s copy state display (i) Specification of ATgroup (required) -atg ATgroup name (ii) Specification of disk array name -arrayname disk array name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required when executing this command to display an ATgroup s copy state. This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. When disk arrays are connected to both the concentrator side (MV side) and the distributor side (RV side), specify the disk array to be displayed. If this option is omitted, the target disk array is automatically searched. 133

142 Chapter 4 Software Operation Methods (iii) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Information displayed by ATgroup copy state display (Windows) [Displayed information] ATG: atgroup Disk Array Name diskarray_name Site site ATG State atg_consistency_state(atg_sync_state) ATG Path State atg_path_state Atomic-break Mode atomic_break_mode Copy Mode copy_mode ATG Copy Diff atg_copy_diffsize Pair Entry pair_entry_count Allowed Response allowed_response_time MV: Disk No. disk_number LD Name ld_name Type type Volume Name volume_name Path path RV: Disk No. disk_number LD Name ld_name Type type Volume Name volume_name Path path Activity State activity Sync State sync Copy Control State copy <state> Start Time YYYY/MM/DD hh:mm:ss <state> End Time YYYY/MM/DD hh:mm:ss Separate Diff separate_diffsize Copy Diff copy_diffsize 134

143 Chapter 4 Software Operation Methods RV Access Previous Active rvacc prev_state Description of display items atgroup ATgroup name diskarray_name Target disk array name site ATgroup s location (concentrator or distributor) atg_consistency_state ATgroup consistency state atg_sync_state ATgroup s sync state atg_path_state ATgroup s link state atomic_break_mode MV access after atomic-break copy_mode ATgroup s copy mode atg_copy_diffsize Copy differential quantity of volume in ATgroup pair_entry_count Number of registered pairs in ATgroup allowed_response_time Allowance time for MV delay that is an ATgroup attribute disk_number Disk number ld_name Logical disk name type Use format volume_name Mounting point volume name path Drive path (mounted to drive letter or NTFS folder) activity Pair s activity state sync Pair s sync state copy Copy control state YYYY/MM/DD hh:mm:ss <state> Start Time indicates this operation s start date/time and <state> End Time indicates its end date/time (<state> differs according to the state) separate_diffsize Copy differential quantity of volume when separate state occurs copy_diffsize Copy differential quantity of volume when replicate or restore state occurs rvacc RV access restriction prev_state Pair s sync state before state transition For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Displayed information] ATG: atgroup Disk Array Name diskarray_name 135

144 Chapter 4 Software Operation Methods Site site ATG State atg_consistency_state(atg_sync_state) ATG Path State atg_path_state Atomic-break Mode atomic_break_mode Copy Mode copy_mode ATG Copy Diff atg_copy_diffsize Pair Entry pair_entry_count Allowed Response allowed_response_time MV: Special File special_file_name LD Name ld_name Type type RV: Special File special_file_name LD Name ld_name Type type Activity State activity Sync State sync Copy Control State copy <state> Start Time YYYY/MM/DD hh:mm:ss <state> End Time YYYY/MM/DD hh:mm:ss Separate Diff separate_diffsize Copy Diff copy_diffsize RV Access rvacc Previous Active prev_state Description of display items atgroup ATgroup name diskarray_name Target disk array name site ATgroup s location (concentrator or distributor) atg_consistency_state ATgroup consistency state atg_sync_state ATgroup s sync state atg_path_state ATgroup s link state atomic_break_mode MV access after atomic-break copy_mode ATgroup s copy mode atg_copy_diffsize Copy differential quantity of volume in ATgroup pair_entry_count Number of registered pairs in ATgroup allowed_response_time Allowance time for MV delay that is an ATgroup attribute special_file_name Special file name ld_name Logical disk name type Use format activity Pair s activity state 136

145 Chapter 4 Software Operation Methods sync copy YYYY/MM/DD hh:mm:ss separate_diffsize copy_diffsize rvacc prev_state Pair s sync state Copy control state <state> Start Time indicates this operation s start date/time and <state> End Time indicates its end date/time (<state> differs according to the state) Copy differential quantity of volume when separate state occurs Copy differential quantity of volume when replicate or restore state occurs RV access restriction Pair s sync state before state transition For detailed description of this command s display items and return values, and environment variables, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for displaying ATgroup s copy state (i) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be displayed belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. However, if the target ATgroup does not include any registered pairs and if a disk array set to the ATgroup can be accessed, only information about that ATgroup is displayed. (ii) Creation and updating of the volume list must be completed. (iii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (iv) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. However, the above conditions (i) to (iii) are not applied to the operation via ism. Notes on displaying the ATgroup copy state (i) If a UNIX server is used, the operation privilege immediately after installation is root authority. This operation can be enabled for a privilege level other than root authority by changing the command and directory operation privilege. 137

146 Chapter 4 Software Operation Methods Diispllayiing tthe ATgroup Liistt ((iismrc_sense)) The ismrc_sense command is used to display an ATgroup list. This command is used in order to display an ATgroup list or setting information about a particular ATgroup. Executable construct of ismrc_sense command (Executable construct for displaying ATgroup list) ismrc_sense -atg [ATgroup name] [ -site conc dist ] [ -linkarrayname] [ -arrayname disk array name ] [ -iopath direct manager ] (Executable construct for displaying specific volume information) ismrc_sense -vol volume -volflg vol_flg [-attr ] [-protect ] [ -iopath iopath ] ismrc_sense -file file_name [-attr ] [-protect ] [ -iopath iopath ] (Executable construct of command version check) ismrc_sense -ver (Executable construct of command usage display) ismrc_sense -?... (Windows) ismrc_sense -help... (UNIX) The following describes the displaying of ATgroup lists. For description of displaying specific volume information, refer to the ControlCommand Command Reference. Specifiable options and parameters for ATgroup list display (i) Specification of ATgroup information (options must be specified, parameters are not required) -atg [ATgroup name] This is specified to fetch ATgroup information. If no parameters are specified, the ATgroup list is displayed. To display particular ATgroup s information, specify the target ATgroup s name as ATgroup name. Specification of this option is required in order to display an ATgroup list. (Specification of parameters is not required.) (ii) Specification of site information -site conc dist This specifies the site (concentrator or distributor) of the ATgroup whose information will be displayed. Specify conc to display an ATgroup connected to the concentrator side (MV side), or dist to specify an ATgroup connected to the distributor side (RV side). If this option is omitted, all accessible ATgroup information on both the concentrator 138

147 Chapter 4 Software Operation Methods (iii) Specification of disk array name -arrayname disk array name side (MV side) and distributor side (RV side) will be displayed. An error will occur if incorrect site information is entered in order to display particular ATgroup s information, or in order to specify a disk array name. This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. (iv) Specification of link disk array name -linkarrayname This specifies the disk array name of the link disk array configuring the ATgroup. (v) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Information displayed when displaying ATgroup list (Windows) [Displayed information] No. ATG Name Site Entry Disk Array Name xxxxh atgroup site entry diskarray_name Specifying -linkarrayname can display the disk array name of the link disk array configuring the ATgroup. [Displayed information] No. ATG Name Site Entry Disk Array Name Link Disk Array Name xxxxh atgroup site entry 139

148 Chapter 4 Software Operation Methods diskarray_name link_diskarray_name Description of display items xxxx Concentrator number (a unique number is assigned to each concentrator, for each related disk array) atgroup ATgroup name site ATgroup s location (concentrator or distributor) entry Number of registered pairs in ATgroup diskarray_name Disk array name link_diskarray_name Link disk array name For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Displayed information] No. ATG Name Site Entry Disk Array Name xxxxh atgroup site entry diskarray_name Specifying -linkarrayname can display the disk array name of the link disk array configuring the ATgroup. [Displayed information] No. ATG Name Site Entry Disk Array Name Link Disk Array Name xxxxh atgroup site entry diskarray_name link_diskarray_name Description of display items xxxx atgroup site entry diskarray_name Concentrator number (a unique number is assigned to each concentrator, for each related disk array) ATgroup name ATgroup s location (concentrator or distributor) Number of registered pairs in ATgroup Disk array name 140

149 Chapter 4 Software Operation Methods link_diskarray_name Link disk array name For detailed description of this command s display items and return values as well as environment variables, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for displaying ATgroup list (i) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be displayed belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. However, if the target ATgroup does not include any registered pairs and if a disk array set to the ATgroup can be accessed, only information about that ATgroup is displayed. (ii) Creation and updating of the volume list must be completed. (iii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (iv) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. However, the above conditions (i) to (iii) are not applied to the operation via ism. Notes on displaying the ATgroup list (i) If a UNIX server is used, the operation privilege immediately after installation is root authority. This operation can be enabled for a privilege level other than root authority by changing the command and directory operation privilege. 141

150 Chapter 4 Software Operation Methods Buiilldiing an ATgroup ((iismrc_attg)) The ismrc_atg command is used to build ATgroups. This command is used to create or delete an ATgroup, to change attributes, or to register or delete a volume pair to an ATgroup. Executable construct of ismrc_atg command (Executable construct for building an ATgroup) ismrc_atg -create -delete -add -remove -update -atg ATgroup name [-force ] [-mv volume -mvflg ld drv mdir mvol sfn [-rv volume -rvflg ld drv mdir mvol sfn ] -file file ] [-arrayname disk array name ] [-linkarrayname link disk array name ] [ -allowedresponse allowed response time of MV ] [ -iopath direct manager ] (Executable construct of command version check) ismrc_atg -ver (Executable construct of command usage display) ismrc_atg -?... (Windows) ismrc_atg -help... (UNIX) Specifiable options and parameters for build ATgroup (i) Specification of operation (required) -create -delete -add -remove -update (ii) Specification of ATgroup (required) -atg ATgroup name This specifies whether the operation will be to create or delete an ATgroup, to change attributes, or to register or delete a pair in an ATgroup. When -create is specified, the target ATgroup will be created. When -delete is specified, the target ATgroup will be deleted. When -add is specified, a volume pair will be registered to a previously created ATgroup. When -update is specified, the allowed response time of MV, which is an ATgroup attribute, is updated. When -remove is specified, a previously registered pair will be deleted from an ATgroup. Specification of one of these options is required. This specifies the ATgroup to be built. Enter the name of the ATgroup to be built as ATgroup name. The ATgroup name can include up to 32 characters consisting of alphanumeric characters, underbars (_) and/or forward slashes (/). This option must be specified in order to build an ATgroup. 142

151 Chapter 4 Software Operation Methods (iii) Specification of forced deletion (required for Forced deletion of ATgroup) -force This specifies that the target ATgroup will be forcibly deleted. This option is valid only when deletion of an ATgroup has been specified. Forced deletion of ATgroup is specified when an ATgroup becomes abnormal due to a fault, etc., or when invalid ATgroup information remains on the concentrator side (MV side) or distributor side (RV side). (iv) Specification of pair to be registered or delete (required for registration or deletion of volume pair in ATgroup) -mv volume -mvflg ld drv mdir mvol sfn [-rv volume -rvflg ld drv mdir mvol sfn] -file file These options specify the volume pair to be registered to or deleted from an ATgroup. These options are valid only when specifying either -add to register a pair or -remove to delete a previously registered pair, in which case these options are required. The value specified in volume indicates the type of volume for the -mvflg or -rvflg option. When ld is specified in -mvflg or -rvflg, the volume specification is a logical disk name. When drv is specified, (for Windows only), the volume specification becomes a drive letter. When mdir is specified (for Windows only), the volume specification becomes the mount destination for the NTFS folder. When mvol is specified (for Windows only), the volume specification is the mounting point volume name. When sfn is specified (for UNIX only), the volume specification becomes a special file name. When only one pair has been registered in a configuration, specification of -rv and -rvflg can be omitted for the specified MV. Instead of -mv, -mvflg, -rv, or -rvflg option, the pair information can be entered in the -file option to enable specification of the replication control file. An example of a replication control file specification is shown below. For details, refer to the ControlCommand Command Reference. Format of replication control file MV_Type:MV RV_Type:RV 143

152 Chapter 4 Software Operation Methods Description of replication control file MV_Type This specifies the volume format specified for the MV. Formats that can be specified to build an ATgroup are: ld, drv (Windows only), mdir (Windows only), mvol (Windows only), and sfn (UNIX only). MV This specifies the MV_Type format for the MV volume. RV_Type This specifies the volume format for the RV. Formats that can be specified to build an ATgroup are: ld, drv (Windows only), mdir (Windows only), mvol (Windows only), and sfn (UNIX only). RV This specifies the RV_Type format for the RV volume. (v) Specification of disk array name (required for creation or deletion of ATgroup) -arrayname disk array name This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. When creating an ATgroup or registering a pair to an ATgroup, specify the name of the disk array on the concentrator side (MV side) which will be accessible. This option is required when creating or deleting an ATgroup. This option is used to explicitly specify the target disk array when registering or deleting a pair in an ATgroup. (vi) Specification of link disk array name (required for creation of ATgroup) -linkarrayname link disk array name This specifies the disk array name of the link (distributor side) disk array when creating an ATgroup with the disk array on the concentrator side which supports the function specifying the link disk array. With the disk array which does not support this function, it is not required to specify this option. And it is ignored even if specified. The supporting situation of the function specifying the link disk array can be checked by viewing the disk array properties of the disk array to be created from the replication screen of ism. (vii) Specification of allowed response time of MV (required for ATgroup updating) -allowedresponse allowed response time of MV This specifies, in seconds, the allowed response time of MV that is an ATgroup attribute. A value from 1 to 18 seconds (integer) can be specified. The default value is 18 seconds. 144

153 Chapter 4 Software Operation Methods (viii) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Information displayed when building an ATgroup (Windows) [End message] ism13247: Command has completed successfully. (code=pppp-xxxx-yyyy-zzzz) Description of display items pppp-xxxx-yyyy-zzzz This differs according to the processing. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [End message] ism13247: Command has completed successfully. (code=pppp-xxxx-yyyy-zzzz) Description of display items pppp-xxxx-yyyy-zzzz This differs according to the processing. For detailed description of this command s display items and return values, and environment variables, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for building an ATgroup (i) When creating an ATgroup, the disk array on the target ATgroup s concentrator side (MV side) must be accessible. (ii) When creating an ATgroup, the ATgroup name you enter cannot exceed 32 characters, can include 145

154 Chapter 4 Software Operation Methods only alphanumeric characters, underbars (_), and/or forward slashes (/), and must be unique within the system. (iii) When creating an ATgroup with the disk array on the concentrator side which supports the function specifying the link disk array, it is required to specify the link disk array using the -linkarrayname option. (iv) When deleting an ATgroup, the disk array on the target ATgroup s concentrator side (MV side) must be accessible. However, if using the -force option to set forced deletion, either the disk array on the concentrator side (MV side) or the disk array on the distributor side (RV side) of the ATgroup where the forced operation is executed can be accessible. (v) To delete an ATgroup, all registered pairs in that ATgroup must be deleted, and the ATgroup s sync state must become invalid. (vi) To register a volume pair to an ATgroup, the target pair must meet the following conditions. DDR pairs cannot be registered to an ATgroup. (Only RDR pairs can be registered to an ATgroup.) A pair that is not the top-level pair cannot be registered to an ATgroup. (Only the top-level pair can be registered.) If the MV of the pair to be registered includes several RDR pairs, it cannot be registered to an ATgroup. Each RDR pair must be set as a single pair registered to an ATgroup. A registered pair in an ATgroup must be separate from the pair to be registered to an ATgroup. An RDR pair that exists among different disk arrays cannot be registered to a single ATgroup. (The RDR pair must be in the same disk array.) If the RV of the RDR pair to be registered is used as a BV (base-volume) of the snapshot function, it cannot be registered to an ATgroup. If the MV or RV of the RDR pair to be registered is protected by the data retention function, it cannot be registered to an ATgroup. (vii) When registering a pair to an ATgroup, the target pair s MV or RV must be registered to a volume list and must be accessible. (viii) When registering a pair to an ATgroup, the ATgroup sync state must be invalid, separated, or fault separation. (ix) When deleting a pair from an ATgroup, the ATgroup sync state must be either separated or fault separation. (x) When executing Forced deletion of ATgroup on the concentrator side (MV side), ATgroup sync state must be invalid, separated, or fault separation. (xi) When updating the allowed response time of MV that is an ATgroup attribute, the ATgroup sync state must be Invalid, Separated, or Fault. (xii) Creation and updating of the volume list must be completed. (xiii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (xiv) There must not be any frozen replication functions in any disk array that belongs to the target ATgroup. 146

155 Chapter 4 Software Operation Methods (xv) When creating an ATgroup and registering a pair to an ATgroup, the link state for the disk array must be normal. (xvi) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (xvii) If any of the following functions is executed when operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list: Forced deletion of ATgroup Deletion of registered pair from ATgroup Pair registration in ATgroup However, the above conditions (vii), (xi), (xii), and (xvii) are not applied to the operation via ism. Notes on building an ATgroup (i) Forced deletion of ATgroup (using the -force option) is executed when the ATgroup becomes abnormal due to a fault, etc., and invalid ATgroup information remains on the concentrator side (MV side) or distributor side (RV side). When invalid ATgroup information remains on the concentrator side (MV side), separate the ATgroup or perform Forced separation of ATgroup on the concentrator side (MV side) to enable the ATgroup to be deleted. Next, execute Forced deletion of ATgroup to delete the invalid ATgroup information. At this time, it may take several minutes to complete Forced deletion of ATgroup. (ii) If a UNIX server is used, the operation privilege immediately after installation is root authority. Operations using privilege levels other than root authority can also be enabled by changing the operation privilege for the command and directory. 147

156 Chapter 4 Software Operation Methods Remotte Operattiion Commands Replication control and display commands (ismrc_replicate, ismrc_separate, ismrc_restore, ismrc_change, ismrc_wait, ismrc_query, ismrc_sense, ismrc_updprevent, and ismrc_swap) can be executed for remote pairs that have already been set. Remote operations cannot be executed using commands for disk array information display, pair setting, and ATgroup, or commands that display server information (ismrc_arrayinfo, ismrc_ldlist, ismrc_pair, and ismvollist). Executable construct of remote operation commands for pairs The executable construct of remote operation commands for pairs is the same as for commands that involve ordinary pairs. The pair targeted for remote operations can be specified in command parameters. However, when a remote operation is executed for a pair, only the logical disk name can be specified in the pair specification. For description of executable constructs related to operations involving ordinary pairs, refer to the Data Replication User s Manual (Function Guide) and the ControlCommand Command Reference. Range of remote operation commands for pairs Remote operation commands for pairs can be used for operations involving linked pairs in a range that includes the accessible volumes in the command s target pair and extends to the next level of disk arrays. The following figures illustrate configurations of pairs that can be used in remote operations and configurations of pairs that cannot be used in remote operations. Operation Range for the ATgroup The operation commands for the ATgroup can only be executed from the server that has access to the MV or RV of the pair belonging to the ATgroup (main site s application server or backup site s standby server). Note that the command to build the ATgroup (ismrc_atg) can only be executed from the main site. The following shows configuration examples of pairs that can be operated, pairs that cannot be operated, and ATgroup operation range. 148

157 Chapter 4 Software Operation Methods This first figure shows pairs for which remote operation commands can be executed from the main site. Main site Application server Backup site Standby server Operation instruction MV (i) RV/MV Operation instruction (ii) RV MV (iii) RV/MV (iv) RV/MV (v) RV Operation instruction MV (vi) (vii) RV/MV (viii) RV RV Disk array Disk array [Description of pairs available for operations] (i) RDR pairs accessible via MV (ii) DDR pairs linked via disk array of this command s target pair MV and next level s disk array (iii) DDR pairs accessible via MV (iv) RDR pairs linked via disk array of this command s target pair MV (v) DDR pairs linked via disk array of this command s target pair MV and next level s disk array (vi) DDR pairs accessible via RV (vii) RDR pairs linked via disk array of this command s target pair RV (viii) DDR pairs linked via disk array of this command s target pair RV and next level s disk array Figure 4-21 Pairs Available for Remote Operation Commands Executed from Main Site 149

158 Chapter 4 Software Operation Methods The next first shows pairs for which remote operation commands can be executed from a backup site. Main site Application server Backup site Standby server MV (i) Operation instruction RV/MV (ii) RV Operation instruction MV (ii) RV (iii) RV MV (iv) RV/MV (v) RV/MV (vi) Operation instruction RV MV (vii) (v) RV/MV (vi) Operation instruction RV RV Disk array Disk array [Description of pairs available for operations] (i) DDR pairs linked via disk array of this command s target pair RV and next level s disk array (ii) RDR pairs accessible via RV (iii) DDR pairs linked via disk array of this command s target pair RV and next level s disk array (iv) DDR pairs linked via disk array of this command s target pair RV and next level s disk array (v) RDR pairs linked via disk array of this command s target pair RV (vi) DDR pairs accessible via RV (vii) DDR pairs linked via disk array of this command s target pair RV and next level s disk array Figure 4-22 Pairs Available for Remote Operation Commands Executed from Backup Site 150

159 Chapter 4 Software Operation Methods This figure shows pairs for which remote operation commands cannot be executed. Main site Backup site (1) Application server Standby server (1) Backup site (2) Standby server (2) Operation instruction MV RV/MV (i) RV Operation instruction MV (ii) RV/MV Operation instruction RV MV RV/MV (iii) RV Disk array Disk array Disk array [Reasons why remote operation commands cannot be executed] (i) Disk array is two levels from MV for this command s target pair. (ii) Disk array is two levels from RV for this command s target pair. (iii) Disk array is two levels from MV for this command s target pair. Figure 4-23 Pairs for which Remote Operation Commands Cannot be Executed 151

160 Chapter 4 Software Operation Methods This figure shows the operation range for the ATgroup. Main site Application server Backup site Standby site Operation instruction Operation instruction (i) MV ATgroup (ii) RV MV RV MV RV Disk array Disk array [Description of ATgroup that can be operated] (i) ATgroup containing the MV accessible from the main site. However, if the operating mode of the main site from the application server is set to secure mode, the MVs of all the pairs in the ATgroup must be registered in the volume list. (ii) ATgroup containing the RV accessible from the backup site. However, if the operating mode of the main site from the standby server is set to secure mode, the RVs of all the pairs in the ATgroup must be registered in the volume list. Figure 4-24 Operation Range for the ATgroup 152

161 Chapter 4 Software Operation Methods Conditions for execution of remote operation commands (i) Remote operation commands can be used for operations involving linked pairs in a range that includes the accessible volumes in the command s target pair (which have been registered in the volume list) and extends to the next level of disk arrays. (ii) The target pairs for remote operations must be specified in the command s parameters as the target pairs. (iii) Logical disk names must be specified for remote operation pairs. (iv) Replication functions must not be frozen in the disk arrays for the pairs for which remote operations are executed. (v) When remote operations are executed for pairs at the next level, the link state for the disk array must be normal. (vi) Creation and updating of the volume list must be completed. (vii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress) (viii) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. 153

162 Chapter 5 Operation Examples Chapter 5 Operation Examples If a disaster occurs at the main site in a disaster recovery system, operations executed for the scenarios described below must be taken into consideration to enable recovery of the assumed essentials for business. For example, in a scenario where an equipment fault occurs during recovery from a disaster, one must consider what operation would be needed to perform a time-based backup at a backup site. In a scenario where an equipment fault or line fault occurs and then a disaster occurs, one must devise a recovery method that prevents the loss of the latest backup data. Key points to consider for operations under various scenarios are listed in the following table. Table 5-1 Considerations for Operations Under Various Scenarios Scenario Considerations for Operations During normal operations When an equipment fault or line fault occurs When a disaster occurs When restoring from backup site to main site Copy mode and sync state during operations Time-based backup Switchover of business tasks at scheduled times or at night, etc. Steps for recovery from line fault, Steps for recovery from disk fault (MV or RV) Steps for recovery from server fault Countermeasures to a disaster following an equipment fault or line fault Fault detection and reporting methods Etc. Recovery steps at backup site Operation of backup site after recovery Countermeasures to a disaster at a backup site Disaster detection and reporting methods Etc. Data restoration steps Etc. The considerations for these operations differ somewhat depending on the specific disaster recovery configuration and the content of business tasks. The steps described below are for a scenario in which the current copy mode is semi synchronous copy mode of order zguarantee and the current sync state is continual replication. Backup during normal operations Recovery steps from line fault (after a line fault occurs) Steps for use of RV data in ATgroup after a disaster Restoration of data from MV to RV when recovering from backup site to main site 154

163 Chapter 5 Operation Examples 5..1 Backup During normal operations, DynamicDataReplication and tape backup are performed periodically at the main site and backup site. When recovering business operations at a backup site after a disaster, RVs in the ATgroup are used for data recovery. The backup data is used for data recovery from a partial fault at the main site, or for data recovery as part of business recovery after a fault has occurred at a backup site. For example, DynamicDataReplication or backup data on tape (backed up at the main site) is used when a fault has occurred among some of the logical disks at the main site. The following describes a configuration example in which the tape backup steps include the use of DynamicDataReplication at various sites. 155

164 Chapter 5 Operation Examples (1) Overview of backup steps The following configuration example is used to describe a set of backup steps. During normal operations, the ATgroup sync state is replicate synchronous (Rpl/sync) and the copy mode is semi synchronous copy mode of order guarantee. DDR pairs are set up at the main site and backup site, and during normal operations the activity state of each site s DDR pairs is separate and the sync state is separated. Backup consists of tape backup that is performed after the DDR pause point data has been secured by replication control. During these steps, business operations are temporarily paused in order to secure pause points. Main site Backup site Application server Backup server Standby server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array [Status of ATgroup and DDR pair before backup] ATgroup s copy mode: Semi synchronous copy mode of order guarantee MV access after atomic-break: MV access continue ATgroup consistency state: Atomic ATgroup sync state: Replicate synchronous (Rpl/sync) Link state: Normal State of DDR pair: Separated Figure 5-1 Configuration Example of Backup Steps 156

165 Chapter 5 Operation Examples (2) Flow of Backup Steps [Operations at main site] Step 1a. Unmount DDR pair s RV (backup server) [Operations at backup site] Step 1b. Unmount DDR pair s RV (backup server) Step 2a. Replicate DDR pair (application server) Step 2b. Replicate DDR pair in background copy mode (standby server) Step 3. Secure pause point (application server) Step 4. Separate ATgroup (application server) Step 5a. Separate DDR pair (application server) Step 5b. Separate DDR pair (standby server) Step 6. Replicate ATgroup (application server) Step 7. Resume business tasks (application server) Step 8a. Mount DDR pair s RV (backup server) Step 8b. Mount DDR pair s RV (backup server) Step 9a. Backup to tape (backup server) Step 9b. Backup to tape (backup server) * The flow of these steps and the following operation steps are provided as an example. Be sure to adjust the specific operations for the particular environment. Figure 5-2 Flow of Backup Steps 157

166 Chapter 5 Operation Examples (3) Operation Steps Step 1. Unmount DDR pair s RV (main site and backup site s backup server) (Windows) At this step, the file system buffers are cleared in the respective DDR pair s RVs which are recognized by the main site and backup site s backup server. The server that is already accessing the target volume is stopped, the file system is flushed and unmounted, and then the file system s buffers are cleared. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system and clear the file system buffers. Since the volumes to be unmounted are RVs, their drive path (mounted to drive letter or NTFS folder) is automatically deleted so that these RVs cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (UNIX) The respective DDR pair s RVs, which are recognized by the main site and backup site s backup server, are unmounted. Since the RV information is overwritten over the MV information, if an LVM is used, the volume group information is temporarily deleted. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Either set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. When using LVM on HP-UX servers, the volume group information is temporarily deleted after the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) When using HP-UX servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 vgexport -m vg01.map /dev/vg01 vgexport -m vg02.map /dev/vg02 When using Linux servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 158

167 Chapter 5 Operation Examples vxdg deport dg02 vxdisk offline c2t0d0 vxdisk offline c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. Step 2. Replicate DDR pairs (application server and standby server) The main site and backup site DDR pairs are replicated. Since the backup site s DDR pairs are serially configured pairs that are simultaneously replicated, the -nowait option is set and replications performed during background copy mode. (Command example at main site) ismrc_replicate -file pair_ddr.txt -wait * pair_ddr.txt setting example #Type:MV Type:RV ld:db_mv ld:db_bk_local ld:log_mv ld:log_bk_local (Command example at backup site) ismrc_replicate -file pair_ddr_r.txt -cpmode bg -nowait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote Step 3. Secure pause point (application server) (Windows) A pause point is secured by the application server. Specifically, business operations and other aspects of the current service that access the target volume are stopped, the business volume s file system is flushed and unmounted, and a pause point is secured. (i) Stop business operations. (ii) Stop service. (Command example) NET STOP ESMCommonService (iii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: 159

168 Chapter 5 Operation Examples (UNIX) A pause point is secured by the application server. Specifically, business operations are stopped, the business volume is unmounted, either the volume group is set as inactive or the disk group is deported, and a pause point is secured. (i) Stop business operations. (ii) Unmount the file system. (Command example) umount /db umount /log (iii) The volume group is set as inactive or the disk group is deported. If LVM is used, set the volume group as inactive. If VxVM is used, deport the disk group. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 Step 4. Separate ATgroup (application server) The ATgroup is separated in order to secure pause point data in the ATgroup s RVs at the backup site. (Command example) ismrc_separate -atg DB_Group -wait Step 5. Separate DDR pair (applicaition server and standby server) A DDR pair is separated to secure pause point data in the RV of the DDR pair where backup data will be secured. (Command example at main site) ismrc_separate -file pair_ddr.txt -wait * pair_ddr.txt setting example #Type:MV Type:RV ld:db_mv ld:db_bk_local ld:log_mv ld:log_bk_local (Command example at backup site) ismrc_separate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote 160

169 Chapter 5 Operation Examples Step 6. Replicate ATgroup (application server) ATgroup is replicated and then the ATgroup is set to normal consistency state. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -wait Step 7. Resume business tasks (application server) (Windows) All business tasks that were stopped to secure a pause point are resumed. Specifically, the business volume is mounted, stopped service is restarted, and then business tasks are resumed. (i) Mount the file system. (Command example) ismrc_mount -drv E: ismrc_mount -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (iii) Resume business tasks. (UNIX) Business tasks that were stopped to secure a pause point are resumed. Specifically, the volume group is set as active or the disk group is imported, the business volume is mounted, and then business tasks are resumed. (i) Activate the volume group or import the disk group. If LVM is used, the volume group is set as active. If VxVM is used, the disk is imported. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg import dg01 vxdg import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Command example when using LVM and mounting VxFS s file system) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) 161

170 Chapter 5 Operation Examples (iii) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log Resume business tasks. Step 8. Mount DDR pair s RVs (main site and backup site s backup server) (Windows) DDR pair RVs that are recognized by the backup server are mounted. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RVs. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (UNIX) DDR pair RVs that are recognized by the backup server are set as an active volume group or the disk group is imported and then mounted. If LVM is used on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and volume group information is imported before the volume group is set as active. If LVM is used on Linux servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and the volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) When using HP-UX servers: vgchgid /dev/rdsk/c2t0d0 vgchgid /dev/rdsk/c2t0d1 mkdir /dev/vg01 mkdir /dev/vg02 162

171 Chapter 5 Operation Examples (ii) mknod /dev/vg01/group c 64 0x mknod /dev/vg02/group c 64 0x vgimport -v -m vg01.map /dev/vg01 /dev/dsk/c2t0d0 vgimport -v -m vg02.map /dev/vg02 /dev/dsk/c2t0d1 vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 When using Linux servers: vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online c2t0d0 vxdisk online c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall Mount the file system. (Command example when using LVM and mounting VxFS s file system) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) When using HP-UX or Solaris servers mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log Step 9. Backup to tape (main site and backup site s backup server) Backup software is used to back up to tape data that was previously copied to an RV. 163

172 Chapter 5 Operation Examples 5..2 Liine Faullt Recovery When a line fault occurs, a link fault is detected by ism. Once a link fault has been detected, a log with the message ID ism10402 from ism is recorded. For details on ism s logs, refer to the Messages Handbook. If a line fault makes it impossible to guarantee the order when writing to RVs in an ATgroup, an atomic-break is executed. After recovery from a line fault, the ATgroup is replicated and the consistency state is returned to normal consistency state. However, consistency in the ATgroup RVs is temporarily lost during the period from replication start until replicating (i.e., from when the ATgroup sync state is replicate start (Rpl/start) until it is replicating (Rpl/exec). If the risk of a disaster occurring during ATgroup replication must be considered, then before restarting ATgroup replication any RV that has data consistency during execution of an atomic-break must be backed up, such as by using DynamicDataReplication. The following description is based on a configuration of line fault recovery steps. Figure 5-3 Link Fault Display by ism when Line Fault has Occurred 164

173 Chapter 5 Operation Examples (1) Overview of Line Fault Recovery Steps The following configuration example is used to describe line fault recovery steps. In this example, an atomic-break is assumed to be executed when a line fault occurs during operations where the ATgroup sync state is replicate synchronous (Rpl/sync) and the ATgroup s copy mode is semi synchronous copy mode of order guarantee. After recovering from the line fault, consistent data that was saved immediately after the line fault was saved from the backup site to a DDR RV (in this case, the logical disk name is DB_BK_REMOTE, LOG_BK_REMOTE), is used to again replicate the ATgroup so that recovery is to the original consistency state. Main site Backup site Backup server Application server Standby server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array [ATgroup and DDR pair states during line fault recovery] ATgroup s copy mode: Semi synchronous copy mode of order guarantee MV access after atomic-break: MV access continue ATgroup consistency state: Atomic Concentrator s ATgroup sync state: Fault separation (Fault) Distributor s ATgroup sync state: Replicate synchronous (Rpl/sync) Link state: Fault DDR pair state: Separated Figure 5-4 Example of Line Fault Recovery Steps 165

174 Chapter 5 Operation Examples (2) Flow of Line Fault Recovery Steps (example) [Operations at main site] [Operations at backup site] Step 1. Execute ATgroup forced separation (standby server) Step 2. Unmount DDR pair s RV (backup server) Step 3. Replicate DDR pair (standby server) Step 4. Separate DDR pair (standby server) Step 5a. Replicate ATgroup (application server) Step 5b. Mount DDR pair s RV (backup server) * This flow of steps and the operation steps described below are just examples. Be sure to perform operations and steps that are suited to the actual environment. Figure 5-5 Flow of Line Fault Recovery Steps (Example) 166

175 Chapter 5 Operation Examples (3) Operation Steps Step 1. Execute Forced separation of ATgroup (standby server) First, separate the RVS in the ATgroup in order to save consistent data immediately after the line fault occurs. Because the link state is a fault condition, forced separation must be performed to separate the RVs in the ATgroup. (Command example) ismrc_separate -atg DB_Group -force -wait This can also be done using ism at the backup site. Figure 5-6 Execution of Forced separation of ATgroup by ism (Example) Step 2. Unmount DDR pair s RV (backup site s backup server) (Windows) In order to save consistent data immediately after the line fault to a DDR RV, the file system buffers are cleared in the respective DDR pair s RVs which are recognized by the backup site s backup server, to prepare for replication of DDR pairs at the backup site. Specifically, the current service that accesses the target volume is stopped, the file system is flushed and unmounted, and then the file system buffers are cleared. (i) Stop service. (Command example) NET STOP ESMCommonService (ii) File system is flushed and unmounted and file system buffers are cleared. Since the volumes to be unmounted are RVs, their drive path (mounted to drive letter or NTFS folder) is 167

176 Chapter 5 Operation Examples automatically deleted so that these RVs cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (UNIX) In order to save consistent data immediately to a DDR RV after the line fault, respective DDR pair s RVs, which are recognized by the main site and backup site s backup server, are unmounted to prepare for replication of DDR pairs at the backup site. Since the RV information is overwritten over the MV information, if LVM is used, the volume group information is deleted. (i) The file system is unmounted. (Command example) umount /db umount /log (ii) The volume group is set as inactive or the disk group is deported. If LVM is used, volume group is set as inactive. When using HP-UX servers, the volume group information is temporarily deleted after the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) When using HP-UX servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 vgexport -m vg01.map /dev/vg01 vgexport -m vg02.map /dev/vg02 When using Linux servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline c2t0d0 vxdisk offline c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. 168

177 Chapter 5 Operation Examples Step 3. Replicate DDR pair (standby server) The backup site s DDR pairs are replicated, and the data is then copied. In this case, it is assumed that the ATgroup sync state of the host ATgroup s distributor side (RV side) has been changed by ism to fault separation. (Command example) ismrc_replicate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote This can also be done using ism. Figure 5-7 Execution of DDR Pair Replication by ism (Example) 169

178 Chapter 5 Operation Examples Step 4. Separate DDR pair (standby server) The DDR pair is separated in order to save consistent data to the DDR pair s RV immediately after a line fault. (Command example) ismrc_separate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote This can also be done using ism. Figure 5-8 Execution of DDR Pair Separation by ism (Example) 170

179 Chapter 5 Operation Examples Step 5a. Replicate ATgroup (application server) ATgroup is replicated and then the ATgroup is set to normal consistency state. Consistent of RV data in the ATgroup is lost until the ATgroup sync state becomes replicate synchronous (Rpl/sync). If a disaster occurs during this period, the data is recovered from the DDR pair s RV that was secured at Step 4. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -wait This can also be done using ism. Figure 5-9 Execution of ATgroup Replication by ism (Example) 171

180 Chapter 5 Operation Examples Step 5b. Mount DDR pair s RV (backup site s backup server) (Windows) DDR pair RVs that are recognized by the backup server is mounted and normal consistency state is restored. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RVs. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (UNIX) DDR pair RVs that are recognized by the backup site s backup server are set as an active volume group or the disk group is imported and then mounted, after which the normal consistency state is restored. This DDR pair s RV data is data that is copied after being mounted in the application server. Accordingly, after the volume group is set as active or the disk group is imported, the volume s mount bits must be cleared. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and the volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) When using HP-UX servers: vgchgid /dev/rdsk/c2t0d0 vgchgid /dev/rdsk/c2t0d1 mkdir /dev/vg01 mkdir /dev/vg02 mknod /dev/vg01/group c 64 0x mknod /dev/vg02/group c 64 0x vgimport -v -m vg01.map /dev/vg01 /dev/dsk/c2t0d0 vgimport -v -m vg02.map /dev/vg02 /dev/dsk/c2t0d1 172

181 Chapter 5 Operation Examples (ii) (iii) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 When using Linux servers: vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online c2t0d0 vxdisk online c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall Clear the volume s mount bits. (Command example when clearing mount bits while using LVM and VxFS file system) When using HP-UX servers: fsck -F vxfs -y /dev/vg01/lvol1 fsck -F vxfs -y /dev/vg02/lvol1 When using Linux servers: fsck -t vxfs /dev/vg01/lvol1 fsck -t vxfs /dev/vg02/lvol1 (Command example when clearing mount bits while using VxVM and VxFS file system) When using HP-UX or Solaris servers: fsck -F vxfs -y /dev/vx/dsk/dg01/vol1 fsck -F vxfs -y /dev/vx/dsk/dg02/vol1 When using Linux servers: fsck -t vxfs /dev/vx/dsk/dg01/vol1 fsck -t vxfs /dev/vx/dsk/dg02/vol1 Mount the file system. (Command example when mounting VxFS s file system while using LVM) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when mounting VxFS s file system while using VxVM) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db 173

182 Chapter 5 Operation Examples mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log 174

183 Chapter 5 Operation Examples 5..3 Use of RV when Diisaster Occurs When a disaster occurs at the main site and is followed by a disk array fault or line fault, a link fault is detected by the backup site s ism. Once a link fault has been detected, a log with the message ID ism10402 from ism is recorded. For details on ism s logs, refer to the Messages Handbook. When a disaster occurs at the main site, an RV in the ATgroup is used to recover business operations at a backup site. When using an RV in the ATgroup, first check the ATgroup state to determine whether or not the RV data in the ATgroup is consistent. If the ATgroup consistency state is Atomic, it means the RV data in the ATgroup can be confirmed as consistent and can be recovered. If the ATgroup consistency state is not Atomic, it means the RV data in the ATgroup cannot be confirmed as consistent, so backup data must be used for recovery. When using an RV in the ATgroup, the ATgroup sync state must be separated. To set the ATgroup sync state as separated, first execute separate ATgroup, then ATgroup forced separate, and finally recovery from fault separation. After restarting business operations at the backup site, if the ATgroup s settings do not have to be preserved, the ATgroup should be deleted and its pairs should be cancelled to prevent future operation errors. The following describes, based on a configuration example, the steps when using an RV after a fault occurs. Figure 5-10 Link Fault Display by ism After a Disaster 175

184 Chapter 5 Operation Examples (1) Overview of RV use steps after a disaster (example) A configuration example used to describe RV use steps after a disaster is illustrated below. Before the disaster, data was copied while the ATgroup sync state was replicate synchronous (Rpl/sync) and the copy mode was semi synchronous copy mode of order guarantee. When a disaster occurs at the main site, ATgroup state at the backup site is determined, after which RVs in the ATgroup are used. Main site Backup site Backup server Application server Standby server Backup server Tape Tape DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E Disaster occurs DB_MV Mount /dev/vg01 and dg01 in /db or drive E DB_Group DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array [Status of ATgroup and DDR pair before disaster] ATgroup s copy mode: Semi synchronous copy mode of order guarantee MV access after atomic-break: MV access continue ATgroup consistency state: Atomic ATgroup sync state Replicate synchronous (Rpl/sync) Link state: Fault State of DDR pair: Separated Figure 5-11 Configuration Example for RV Use Steps After a Disaster 176

185 Chapter 5 Operation Examples (2) Confirmation of ATgroup State When a disaster has occurred at the main site, prompting use of RVs in the ATgroup at a backup site to restore business operations, it is determined whether or not RVs in the ATgroup can be used, and then the steps for using RVs in the ATgroup are confirmed. These matters are determined by checking the ATgroup consistency state, ATgroup sync state, and link state. ism s replication management functions, ATgroup functions, or ReplicationControl/DisasterRecovery s commands can be used to check the ATgroup consistency state, ATgroup sync state, and link state. The details on these confirmation steps are described below. (i) Confirmation of ATgroup consistency state This step confirms the ATgroup consistency state. When the ATgroup consistency state is Atomic, then the RV data in the ATgroup can be judged as consistent and can be used for recovery. If the ATgroup consistency state is Non-atomic, the link state is normal, and replication control can be executed, then the ATgroup is replicated, separated, and restored in that order. If replication control cannot be executed or if the link state is abnormal, recovery is not possible unless a pause point has been secured, the ATgroup has been separated, and the ATgroup sync state is not separated. (Example of confirmation by command) ismrc_query -atg DB_Group * The ATG State in the command s results is used to confirm the ATgroup consistency state. This can also be confirmed via ism. Figure 5-12 Example of Confirmation of ATgroup Consistency State by ism 177

186 Chapter 5 Operation Examples (ii) Confirmation of link state This step confirms the link state. If the link state is confirmed as normal, the ATgroup is replicated and then separated, so that data from just before a disaster can be used to prevent any data loss. However, when an ATgroup is replicated, if a fault occurs during the period from replication start until replicating (i.e., from when the ATgroup sync state is replicate start (Rpl/start) until it is replicating (Rpl/exec), RVs in the ATgroup can no longer be used. Consequently, before replicating an ATgroup, software such as DynamicDataReplication must be used to back up the RVs in the ATgroup. When the link state is abnormal, if the copy mode is synchronous mode and the MV access after atomic-break setting is MV access stop, the latest data will be retained in the ATgroup s RVs. Some data loss may occur if these settings are different. (Example of confirmation by command) ismrc_query -atg DB_Group * The ATG Path State in the command s results is used to confirm the link state. This can also be confirmed via ism. Figure 5-13 Example of Link State Confirmation by ism 178

187 Chapter 5 Operation Examples (iii) Confirmation of ATgroup sync state This step confirms the ATgroup sync state. The ATgroup sync state is used to determine the recovery steps for using RVs in the ATgroup. (Example of confirmation by command) ismrc_query -atg DB_Group * The ATG State in the command s results is used to confirm the ATgroup sync state. This can also be confirmed via ism. Figure 5-14 Example of ATgroup Sync State Confirmation by ism 179

188 Chapter 5 Operation Examples (3) Steps for Using RV in ATgroup When using RVs in the ATgroup, the ATgroup sync state must be separated. To set the ATgroup sync state as separated, first execute separate ATgroup, ATgroup forced separate, and recovery from fault separation. The following describes the steps for using RVs in an ATgroup when the ATgroup consistency state, link state, and ATgroup sync state are determined to be as described below when confirmed from a backup site. These states are set up in anticipation of a scenario in which a fault has occurred in a server, disk array, or line at the main site. [States confirmed from backup site] ATgroup consistency state Link state ATgroup sync state State of DDR pair at backup site... Atomic... Fault... Replicate synchronous (Rpl/sync)... Separated * Since a disk array fault has occurred, states cannot be checked from the concentrator side (MV side), so the ATgroup sync state checked from a backup site is retained before any disaster occurs. Flow of Steps for Using RV in ATgroup Step 1. Execute forced separation (standby server) Step 2. Execute recovery from fault separation (standby server) Step 3. Mount RVs in ATgroup (standby server) Step 4. Prepare to resume business operations, and resume business operations (standby server) Step 5. Forced deletion of ATgroup and forced unpairing of pair (standby server) * Step 5 can be omitted if the main site s disk array is operating normally or if the ATgroup must be set by a host application. * This flow of steps and the operation steps described below are just examples. Be sure to perform operations and steps that are suited to the actual environment. Figure 5-15 Flow of Steps for Using RV in ATgroup 180

189 Chapter 5 Operation Examples Operation Steps Step 1. Execute forced separation (standby server) Once the ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) for the ATgroup s distributor side (RV side) which recovers business operations, the ATgroup sync state set as separated. Since the link state is fault, a forced separation is executed first to separate the RVs in the ATgroup. When forced separation is executed, the ATgroup sync state becomes fault separation. (Command example) ismrc_separate -atg DB_Group -force -wait This can also be done using ism at the backup site. Figure 5-16 Execution of Forced separation of ATgroup by ism (Example) 181

190 Chapter 5 Operation Examples Step 2. Execute recovery from fault separation (standby server) Once the ATgroup consistency state is Atomic and the ATgroup sync state is fault separation on the ATgroup s distributor side (RV side) where business operations are performed, recovery from fault separation is executed and the distributor side (RV side) ATgroup s sync state becomes separated. (Command example) ismrc_change -atg DB_Group -force separated This can also be done using ism at the backup site. Figure 5-17 Execution of Recovery from Fault Separation by ism (Example) 182

191 Chapter 5 Operation Examples Step 3. Mount RV in ATgroup (standby server) (Windows) The RVs that were used by the standby server to recover from fault separation are mounted as RVs in the ATgroup. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RVs. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (UNIX) The RVs that were used by the standby server to recover from fault separation are mounted as RVs in the ATgroup, and the volume group is set as active or the disk group is imported. The RV data in the ATgroup is data that is copied after being mounted in the application server. Consequently, after the volume group is set as active or the disk group is imported, the volume s mount bits must be cleared. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and the volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) When using HP-UX servers vgchgid /dev/rdsk/c2t0d0 vgchgid /dev/rdsk/c2t0d1 mkdir /dev/vg01 mkdir /dev/vg02 mknod /dev/vg01/group c 64 0x mknod /dev/vg02/group c 64 0x vgimport -v -m vg01.map /dev/vg01 /dev/dsk/c2t0d0 vgimport -v -m vg02.map /dev/vg02 /dev/dsk/c2t0d1 vgchange -a y /dev/vg01 183

192 Chapter 5 Operation Examples (ii) (iii) vgchange -a y /dev/vg02 When using Linux servers vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online c2t0d0 vxdisk online c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall Clear the volume s mount bits. (Command example when clearing mount bits while using LVM and VxFS file system) When using HP-UX servers: fsck -F vxfs -y /dev/vg01/lvol1 fsck -F vxfs -y /dev/vg02/lvol1 When using Linux servers: fsck -t vxfs /dev/vg01/lvol1 fsck -t vxfs /dev/vg02/lvol1 (Command example when clearing mount bits while using VxVM and VxFS file system) When using HP-UX or Solaris servers: fsck -F vxfs -y /dev/vx/dsk/dg01/vol1 fsck -F vxfs -y /dev/vx/dsk/dg02/vol1 When using Linux servers: fsck -t vxfs /dev/vx/dsk/dg01/vol1 fsck -t vxfs /dev/vx/dsk/dg02/vol1 Mount the file system. (Command example when mounting VxFS s file system while using LVM) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when mounting VxFS s file system while using VxVM) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log 184

193 Chapter 5 Operation Examples When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log Step 4. Prepare to resume business operations, and resume business operations (standby server) The standby server is used to recover the cluster configuration, confirm data in the mounted RVs in the ATgroup, revise the network configuration that it uses for business operations, prepare for resuming business operations, and to resume business operations. Step 5. Forced deletion of ATgroup and forced deletion of pair (standby server) After resuming business operations, forced deletion is executed for the ATgroup and forced unpairing for the pair, and then settings that are no longer needed are deleted to prevent operation errors. However, if an ATgroup must be set when the main site s disk array is operating normally, then neither forced deletion of the ATgroup nor forced unpairing is executed. (Command example of Forced deletion of ATgroup) ismrc_atg -delete -atg DB_Group -arrayname REMOTE_ARRAY -force (Command example of forced unpairing of pair) ismrc_pair -unpair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld -force rv ismrc_pair -unpair -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld -force rv or ismrc_pair -unpair -file pair.txt -force rv * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv This can also be done using ism at the backup site. 185

194 Chapter 5 Operation Examples Figure 5-18 Execution of Forced Deletion of ATgroup by ism (Example) Figure 5-19 Execution of Forced Unpairing of Pairs by ism (Example) 186

195 Chapter 5 Operation Examples 5..4 Copy Back Mode from RV to MV The following describes how, after a disaster has caused business operations to be switched to a backup site, the fault at the main site is repaired and data is restored from an RV to the MV so that business operations can be switched back to the main site. Factors such as the fault state at the main site and the time required for recovery are used to determine whether to simply restore data from an RV to the MV or to swap the pair configuration before restoring data from an RV to the MV Resttorattiion ffor Copy Back Mode The following describes the method used to restore data from RVs to MVs. During this restoration, business operations can be confirmed and restarted via the main site where such operations they be resumed. However, access to the restored volume (MV) depends on the line speed. In addition, if a line fault occurs before this restoration has been completed, it may become impossible to access normal data on the restored volume (MV). Consequently, business operations are usually merely checked during the restore operation, and once restoration has been completed business operations are resumed at the main site. Note that during restoration business operations cannot be performed at the backup site that is the source for reverting data. Consequently, differences in information between disk arrays might occur, such as when data has been lost in a disk array, so this is not suitable for full-range copying of data from an RV to the MV. Restoring data from an RV to an MV is suitable when differential data between disk arrays has not been lost or when ample time for reverting business operations can be secured. Below, a configuration example is provided for describing the restoration steps for Copy Back Mode. 187

196 Chapter 5 Operation Examples (1) Overview of Restoration Steps for Copy Back Mode The restoration steps for Copy Back Mode are described below. After a disaster occurs, business operations that are being performed via a standby server at a backup site are reverted to the main site s application server and data is copied back. Main site Backup site Backup server Application server Standby server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E Restore DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F Restore LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array Copy Back Mode by restoration [ATgroup settings after Copy Back Mode] ATgroup s copy mode: MV access after atomic-break: Semi synchronous copy mode of order guarantee MV access continue Figure 5-20 Configuration of Restoration for Copy Back Mode (Example) 188

197 Chapter 5 Operation Examples (2) Flow of Restoration Steps for Copy Back Mode (example) [Operations at main site] [Operations at backup site] Step 1. Repair fault at main site (main site) Step 2a. Create volume list (application server and backup server) Step 2b. Create volume list (standby server and backup server) Step 3. Create new ATgroup (application server) Step 4. Stop business operations (standby server) Step 5a. Unmount restoration volume (MV in ATgroup) (application server) Step 5b. Unmount business volume (RVs in ATgroup) (standby server) Step 6. Restore ATgroup (application server) Step 7. Mount restoration volume (MV in ATgroup) (application server) Step 8. Restore and check business operations (application server) Step 9a. Prepare Copy Back Mode of business operations (main site) Step 9b. Prepare Copy Back Mode of business operations (backup site) Step 10. Wait for completion of ATgroup restoration (application server) Step 11. Replicate ATgroup (application server) Step 12. Resume business operations (application server) * The flow of these steps and the following operation steps are provided as an example. Be sure to adjust the specific operations for the particular environment. Figure 5-21 Flow of Restoration Steps for Copy Back Mode (Example) 189

198 Chapter 5 Operation Examples For steps for Copy Back Mode of RemoteCluster, refer to the RemoteCluster with MC/ServiceGuard User s Manual. (3) Operation Steps Step 1. Repair fault at main site (main site) The fault at the main site is repaired at this step. Software installation and environment settings are also required. When using LVM on HP-UX servers, if the map files on the servers have been lost, the backed up map files are copied when rebuilding the environment. Step 2. Create volume list (application server, standby server, and backup server) If the main site s disk array is connected to a backup site s disk array, a volume list is created to include all servers connected to these disk arrays. If the servers in the volume list are HP-UX or Linux servers running ism, ism must be temporarily stopped. At this step and afterward, the main site s disk array must be connected to a backup site s disk array. (Command example for Windows) ismvollist -cr (Command example for UNIX) ismvollist -r * Do not perform the disk signature writing, partition creation, file system creation, and drive letter setting when making the application server recognize the rebuilt MV for the operation using a GPT disk. Step 3. Create new ATgroup (application server) If an ATgroup has been deleted, such as in order to prevent operation errors or as a result of repairing faults in volumes, a new ATgroup is created at this step. If the target ATgroup is valid, this step can be omitted. (Command example for pair settings) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_pair -pair -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv 190

199 Chapter 5 Operation Examples (Command example of ATgroup creation) ismrc_atg -create -atg DB_Group -arrayname LOCAL_ARRAY -linkarrayname REMOTE_ARRAY (Command example of pair registration to ATgroup) ismrc_atg -add -atg DB_Group -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_atg -add -atg DB_Group -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_atg -add -atg DB_Group -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv These settings can also be made via the main site s ism. Figure 5-22 Pair Settings via ism (Example) 191

200 Chapter 5 Operation Examples Figure 5-23 ATgroup Creation via ism (Example) Figure 5-24 Registration of Pair to ATgroup by ism (Example) 192

201 Chapter 5 Operation Examples Step 4. Stop business operations (standby server) Business operations being handled at the backup site s standby server are stopped to enable their Copy Back Mode. Step 5. Unmount volume (MV or RV in ATgroup) (application server and standby server) (Windows) At this step, file system buffers are cleared in the main site s restored volume (MV in ATgroup) and at the backup site s business volume (RV in ATgroup). The current service that is accessing the target volume is stopped, the file system is flushed and unmounted, and then file system s buffers are cleared. Once these buffers have been cleared, the stopped service can be resumed since the standby server s file system has already been unmounted. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system, then clear the file system s buffers. At the standby server, where the target volume for unmounting is an RV, the drive path (mounted to drive letter or NTFS folder) is automatically deleted to disable access to this RV. (Command example for application server) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (Command example for standby server) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (iii) Start service at standby server. (Command example) NET START ESMCommonService * This step is not required on the application server side for the operation using a GPT disk. (UNIX) To prepare for restoration of the ATgroup, the main site s restoration volume (MV in ATgroup) and the backup site s business volume (RV in ATgroup) are unmounted. When restoring the information in the restoration volume (MV in ATgroup), information in the business volumes (RV in ATgroup) is overwritten during the replication that follows information restoration, so when using LVM on HP-UX servers, the volume group information must be temporarily deleted. (i) Unmount the file system. (Command example) 193

202 Chapter 5 Operation Examples (ii) umount /db umount /log Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. When using LVM on HP-UX servers, the volume group information is deleted after the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) When using HP-UX servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 vgexport -m vg01.map /dev/vg01 vgexport -m vg02.map /dev/vg02 When using Linux servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline c2t0d0 vxdisk offline c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. * If the volume has already been unmounted, such as when repairing a server fault or logical disk fault, this step can be omitted. 194

203 Chapter 5 Operation Examples Step 6. Restore ATgroup (application server) Once the data has been restored, the ATgroup is restored at this step. When restoring the ATgroup, business operations can be confirmed without waiting for restoration to be completed by specifying this command s -nowait option. (Command example) ismrc_restore -atg DB_Group -nowait This can also be done using ism. Figure 5-25 Execution of ATgroup Restoration by ism (Example) For the operation using a GPT disk, the following steps must be executed on the application server before executing Step 7: Restarting the application server Restart the application server to write the disk signature for MV. Recognizing the MV-side disk Start the [Disk Management] of Windows, and assign a drive letter to the disk. Re-creating the volume list 195

204 Chapter 5 Operation Examples Since the drive letter has been newly assigned to MV, re-create the volume list by using the ReplicationControl ismvollist command on the application server. Step 7. Mount restoration volume (MV in ATgroup) (application server) (Windows) At this step, the restoration volume (MV in ATgroup) that will be used after reverting business operations is mounted in the main site s application server, and the stopped service is resumed. (i) Mount the file system. (Command example) ismrc_mount -drv E: ismrc_mount -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (UNIX) At the main site s application server, the volume group is set as active or the disk group is imported for the restoration volume (MV in ATgroup) that will be used after reverting business operations, which is then mounted. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and the volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) When using HP-UX servers: vgchgid /dev/rdsk/c2t0d0 vgchgid /dev/rdsk/c2t0d1 mkdir /dev/vg01 mkdir /dev/vg02 mknod /dev/vg01/group c 64 0x mknod /dev/vg02/group c 64 0x vgimport -v -m vg01.map /dev/vg01 /dev/dsk/c2t0d0 vgimport -v -m vg02.map /dev/vg02 /dev/dsk/c2t0d1 vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 196

205 Chapter 5 Operation Examples (ii) When using Linux servers: vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall Mount the file system. (Command example when using LVM and mounting VxFS s file system) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log 197

206 Chapter 5 Operation Examples Step 8. Restore and check business operations (application server) At this step, the restoration volume (MV in ATgroup) is used to recover the cluster configuration, check the mounted data, and to recover and check business operations in the main site s application server. When using LVM on UNIX servers, after business operations have been recovered and checked, be sure to fetch and store the volume group setting information at the main site s application server. This volume group setting information should be backed up once business operations have been recovered and checked. In some cases, the volume group setting information is used for fault repair. (Command example for fetching volume group setting information file) When using HP-UX servers: vgcfgbackup -f vg01_mv.conf /dev/vg01 vgcfgbackup -f vg02_mv.conf /dev/vg02 When using Linux servers: vgcfgbackup /dev/vg01 vgcfgbackup /dev/vg02 Step 9. Prepare Copy Back Mode of business operations (main site and backup site) Once the recovered business operations have been checked in the main site s application server, the network configuration is changed to enable Copy Back Mode of business operations to the main site. 198

207 Chapter 5 Operation Examples Step 10. Wait for completion of ATgroup restoration (application server) At this step, a wait is set until data copying has been completed as part of ATgroup restoration. (Command example of wait for completion of ATgroup restoration) ismrc_wait -atg DB_Group -cond sep Completion of ATgroup restoration can also be checked by ism. * Restoration is completed when the ATgroup sync state changes from restoring to separated. Figure 5-26 Confirmation of ATgroup Restoration by ism (Example) 199

208 Chapter 5 Operation Examples Step 11. Replicate ATgroup (application server) After data copying has been completed as part of ATgroup restoration, the ATgroup is replicated and then restored to normal state. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -wait This can also be done using ism. Figure 5-27 Execution of ATgroup Replication by ism (Example) Step 12. Resume business operations (application server) Business operations are resumed at the main site s application server. 200

209 Chapter 5 Operation Examples Copy Back Mode by Paiir Conffiigurattiion Substtiittuttiions This is a Copy Back Mode method that includes temporarily substituting pair configurations by registering a newly created ATgroup in which a backup site s volume is copied back to the MV while the main site s volume is copied back to an RV, after which this ATgroup is replicated. While business operations are continued at the backup site, this ATgroup created for Copy Back Mode is replicated and the data is copied back. Once the Copy Back Mode is completed, business operations are switched back. Since this method enables data to be copied back while business operations are continued, business operations can be switched with very little downtime. However, when pair configurations are substituted in this way, ATgroup replication requires full-range copying, which means that some time is needed to reach the replicated state (Rpl/sync) after business operations are switched. Until the replicated state (Rpl/sync) is reached, data in the ATgroup s RVs is not consistent. Accordingly, DynamicDataReplication (or other software) is used to back up the copied back data after business operations are switched and before ATgroup replication. Copy Back Mode by pair substitution is therefore a method that is appropriate in cases such as when there is not enough time to switch over business operations for Copy Back Mode by restoration. The following describes Copy Back Mode by pair substitution based on a configuration example. 201

210 Chapter 5 Operation Examples (1) Overview of Copy Back Mode by Pair Configuration Substitutions The example used to describe the steps of Copy Back Mode by pair configuration substitution is shown below. When a disaster occurs, business operations that had been handled by a backup site s standby server are switched to the main site s application server. The backup site s volume is temporarily copied to the MV and the main site s volume is copied to an RV (hence a pair substitution), while a new ATgroup created for Copy Back Mode is registered and replicated, after which the data is copied back. Main site Backup site Backup server Application server Standby server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E Replicate DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F Replicate LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array Copy Back Mode by substitution of MV and RV pairs [ATgroup settings after Copy Back Mode] ATgroup s copy mode: MV access after atomic-break: Semi synchronous copy mode of order guarantee MV access continue Figure 5-28 Copy Back Mode by Pair Configuration Substitutions (Example) 202

211 Chapter 5 Operation Examples (2) Flow of steps in Copy Back Mode by pair configuration substitution [Operations at main site] [Operations at backup site] Step 1. Repair fault at main site (main site) Step 2a. Create volume list (application server and backup server) Step 2b. Create volume list (standby server and backup server) Step 3. Create new ATgroup for Copy Back Mode during MV and RV pair configuration substitution (standby server) Step 4. Unmount restoration volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 5. Replicate ATgroup used for Copy Back Mode (standby server) Step 6. Secure pause point (standby server) Step 7. Separate ATgroup used for Copy Back Mode (standby server) Step 8. Resume business operations (application server) Step 9. Mount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 10. Restore and check business operations (application server) Step 11. Unmount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 12. Replicate ATgroup used for Copy Back Mode (standby server) 203

212 Chapter 5 Operation Examples Step 13. Stop business operations (standby server) Step 14. Unmount business volume (MV in the ATgroup used for Copy Back Mode) (standby server) Step 15. Separate ATgroup used for Copy Back Mode (standby server) Step 16. Mount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 17a. Prepare to switch over business operations (main site) Step 17b. Prepare to switch over business operations (backup site) Step 18. Resume business operations (application server) Step 19. Delete ATgroup used for Copy Back Mode where pair configuration of MV and RV has been switched (standby server) Step 20. Unmount DDR pair s RV (backup server) Step 21. Replicate DDR pair (standby server) Step 22. Separate DDR pair (standby server) Step 23. Mount DDR pair s RV (backup server) Step 24. Reset ATgroup (application server) Step 25. Unmount ATgroup s RV (standby server: if a Windows server) Step 26. Replicate ATgroup (application server) 204

213 Chapter 5 Operation Examples * When business operations have been stopped, if it is possible to confirm whether or not business operations can be resumed at the main site s application server, then steps 6 to 12 are not necessary. * If a UNIX server is used, step 25 is not necessary. * After restoring the main site fault from steps 13 through 26, you can also copy back data using the swap function as described later. * The flow of these steps and the following operation steps are provided as an example. Be sure to adjust the specific operations for the particular environment. Figure 5-29 Flow of Steps in Copy Back Mode by Pair Configuration Substitution (Example) For steps for Copy Back Mode of RemoteCluster, refer to the RemoteCluster with MC/ServiceGuard User s Manual. (3) Operation Steps Step 1. Repair fault at main site (main site) At this step, the main site s fault is repaired. Software installation and environment settings are also required. When using LVM on HP-UX servers, if the map files on the servers have been lost, the backed up map files are copied when rebuilding the environment. Step 2. Create volume list (application server, standby server, and backup server) If the main site s disk array is connected to a backup site s disk array, a volume list is created to include all servers connected to these disk arrays. If the servers in the volume list are HP-UX or Linux servers running ism, ism must be temporarily stopped. At this step and afterward, main site s disk array must be connected to a backup site s disk array. (Command example for Windows) ismvollist -cr (Command example for UNIX) ismvollist -r * This step is not required on the application server side for the operation using a GPT disk. Make the application server recognize MV rebuilt in Step 8, and re-create the volume list (on the application server) after checking the drive letter. 205

214 Chapter 5 Operation Examples Step 3. Create new ATgroup for Copy Back Mode during MV and RV pair configuration substitution (standby server) When ATgroup replication is used to perform Copy Back Mode, a new ATgroup in which of the backup site s volume is the MV and of the main site s volume is the RV. If any previously set ATgroup remains, it is deleted beforehand. (Command example for pair settings) ismrc_pair -pair -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_pair -pair -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_pair -pair -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv (Command example for creating ATgroup) ismrc_atg -create -atg DB_Group_swap -arrayname REMOTE_ARRAY -linkarrayname REMOTE_ARRAY (Command example for registering pair to ATgroup) ismrc_atg -add -atg DB_Group_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_atg -add -atg DB_Group_swap -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_atg -add -atg DB_Group_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv * Make sure that the backup site s volume is the MV and the main site s volume is the RV. * Before creating the ATgroup, make sure that the backup site s disk array has been specified. * The new ATgroup used for Copy Back Mode is temporarily created as an ATgroup named DB_Group_swap. These settings can also be made via ism. 206

215 Chapter 5 Operation Examples Figure 5-30 Settings in ism for Substitution of MV and RV Pair Configuration (Example) Figure 5-31 Settings in ism for Creation of New ATgroup Used for Substitution of MV and RV Pair Configuration (Example) 207

216 Chapter 5 Operation Examples Figure 5-32 Registration of Pair to ATgroup by ism (Example) 208

217 Chapter 5 Operation Examples Step 4. Unmount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) (Windows) The file system buffers are cleared in the main site s Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) in order to copy back data to the main site. Specifically, the current service that accesses the target volume is stopped, the file system is flushed and unmounted, and then the file system buffers are cleared. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system, then clear the file system s buffers. Since the volume to be unmounted is an RV, the drive path (mounted to drive letter or NTFS folder) is automatically deleted to disable access to this RV. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline * This step is not required on the application server side for the operation using a GPT disk. (UNIX) To prepare for Copy Back Mode of data to the main site, the main site s restoration volume (RV in the ATgroup used for Copy Back Mode) is unmount. The information in the restoration volume (RV in the ATgroup used for Copy Back Mode) overwrites the information in the business volume (MV in the ATgroup used for Copy Back Mode), so when using LVM on HP-UX servers, the volume group information must be temporarily deleted. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. When using LVM on HP-UX servers, the volume group information is temporarily deleted after the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) When using HP-UX servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 vgexport -m vg01.map /dev/vg01 vgexport -m vg02.map /dev/vg02 209

218 Chapter 5 Operation Examples When using Linux servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline c2t0d0 vxdisk offline c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. * If the volume has already been unmounted, such as when repairing a server fault or logical disk fault, this step can be omitted. 210

219 Chapter 5 Operation Examples Step 5. Replicate ATgroup used for Copy Back Mode (standby server) At this step, the ATgroup used for Copy Back Mode is replicated and data is copied to the copy back volume (RV in ATgroup used for Copy Back Mode). (Command example) ismrc_replicate -atg DB_Group_swap -cpmode semi -cprange full -wait * Since this is done immediately after pair reconfiguration, full range copying is performed explicitly at this step. This can also be done using ism. Figure 5-33 Replication by ism of ATgroup Used for Copy Back Mode (Example) 211

220 Chapter 5 Operation Examples Step 6. Secure pause point (standby server) (Windows) A pause point is reserved in the business operations at the backup site in order to retrieve data used to confirm the switchover of business operations to the Copy Back Mode volume (the RV in the ATgroup used for Copy Back Mode). Specifically, business operations and other aspects of the current service that access the target volume are stopped, the business volume s file system is flushed and unmounted, and a pause point is secured. (i) Stop business operations. (ii) Stop service. (Command example) NET STOP ESMCommonService (iii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (UNIX) A pause point is reserved in the business operations at the backup site in order to retrieve data used to confirm the switchover of business operations to the Copy Back Mode volume (the RV in the ATgroup used for Copy Back Mode). Specifically, business operations are stopped, the business volume is unmounted, either the volume group is set as inactive or the disk group is deported, and a pause point is secured. (i) Stop business operations at backup site. (ii) Unmount backup site s business volume. (Command example) umount /db umount /log (iii) Set the volume group as inactive or deport the disk group. If LVM is used, set the volume group as inactive for the backup site s business volume. If VxVM is used, deport the business volume s disk group at the backup site. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 212

221 Chapter 5 Operation Examples Step 7. Separate ATgroup used for Copy Back Mode (standby server) Separate the ATgroup used for Copy Back Mode and save to the Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) the data that will be checked switching over business operations. (Command example) ismrc_separate -atg DB_Group_swap -wait This can also be done using ism. Figure 5-34 Separation by ism of ATgroup Used for Copy Back (Example) For the operation using a GPT disk, the following steps must be executed on the application server before executing Step 8: Making the application server recognize the rebuilt MV Make the application server recognize the rebuilt MV by setting the AccessControl. Recognizing the MV-side drive letter Start the [Disk Management] of Windows, and assign a drive letter to the disk. Re-creating the volume list Since the drive letter has been newly assigned to MV, re-create the volume list by using the ReplicationControl ismvollist command on the application server. 213

222 Chapter 5 Operation Examples Step 8. Resume business operations (standby server) (Windows) Business operations are resumed in order to secure a pause point. At this point, the business volume is mounted, then the stopped service is started and business operations are resumed. (i) Mount the file system. (Command example) ismrc_mount -drv E: ismrc_mount -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (iii) Resume business operations. (UNIX) Business operations are resumed in order to secure a pause point. At this point, the volume group is set as active or the disk group is imported, and then the business volume is mounted and business operations are resumed. (i) Activate the volume group or import the disk group for the backup site s business volume. If LVM is used, set the backup site s business volume group as active. If VxVM is used, import the backup site s business volume s disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg import dg01 vxdg import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the business volume. (Command example of mounting VxFS s file system when using LVM) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example of mounting VxFS s file system when using VxVM) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: 214

223 Chapter 5 Operation Examples (iii) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log Resume business operations. Step 9. Mount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) (Windows) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is mounted to enable switchover of business operations to the main site s application server. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RV. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (UNIX) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is set as in an active volume group or the disk group is imported and mounted to enable switchover of business operations to the main site s application server. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and the volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) When using HP-UX servers: vgchgid /dev/rdsk/c2t0d0 vgchgid /dev/rdsk/c2t0d1 mkdir /dev/vg01 mkdir /dev/vg02 mknod /dev/vg01/group c 64 0x

224 Chapter 5 Operation Examples (ii) mknod /dev/vg02/group c 64 0x vgimport -v -m vg01.map /dev/vg01 /dev/dsk/c2t0d0 vgimport -v -m vg02.map /dev/vg02 /dev/dsk/c2t0d1 vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 When using Linux servers: vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online c2t0d0 vxdisk online c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall Mount the file system. (Command example when using LVM and mounting VxFS s file system) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log Step 10. Restore and check business operations (application server) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is used to confirm recovery of the cluster configuration, and mounted data, as well as to confirm recovery of business operations at the main site s application server. 216

225 Chapter 5 Operation Examples Step 11. Unmount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) (Windows) The file system buffers are cleared in the main site s Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) in order to copy differential data to the Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) during confirmation of business operations recovery. Specifically, the current service that accesses the target volume is stopped, the file system is flushed and unmounted, and then the file system buffers are cleared. (i) Stop service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system and clear the file system buffers. Since the volume to be unmounted is an RV, its drive path (mounted to drive letter or NTFS folder) is automatically deleted so that these it cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (UNIX) The main site s Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is unmounted in order to copy differential data to the Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) during confirmation of business operations recovery. Since information in the restoration volume (RV in the ATgroup used for Copy Back Mode) is overwritten by information from the business volume (MV in ATgroup used for Copy Back Mode), when using LVM on HP-UX servers, the volume group information must be temporarily deleted. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. When using LVM on HP-UX servers, the volume information is temporarily deleted after the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) When using HP-UX servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 vgexport -m vg01.map /dev/vg01 vgexport -m vg02.map /dev/vg02 217

226 Chapter 5 Operation Examples When using Linux servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline c2t0d0 vxdisk offline c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. 218

227 Chapter 5 Operation Examples Step 12. Replicate ATgroup used for Copy Back Mode (standby server) At this step, the ATgroup used for Copy Back Mode is replicated and differential data during confirmation of business operations recovery is copied to the restoration volume (RV in the ATgroup used for Copy Back Mode). (Command example) ismrc_replicate -atg DB_Group_swap -cpmode semi -wait This can also be done using ism. Figure 5-35 Replication of ATgroup Used for Copy Back ism (Example) Step 13. Stop business operations (standby server) Business operations are stopped at the backup site s standby server to enable their switchover. 219

228 Chapter 5 Operation Examples Step 14. Unmount business volume (MV in the ATgroup used for Copy Back Mode) (standby server) (Windows) At this step, the file system in the business volume (MV in ATgroup used for Copy Back Mode) is unmounted. Specifically, the current service that accesses the target volume is stopped and the business volume s file system is flushed and unmounted. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (UNIX) At this step, the business volume (MV in ATgroup used for Copy Back Mode) is unmounted. Since the information in the business volume (MV in ATgroup used for Copy Back Mode) is overwritten during replication that follows switchover of business operations, when using LVM on HP-UX servers, the volume group information must be temporarily deleted. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. When using LVM on HP-UX servers, the volume group information is temporarily deleted after the volume group is set as inactive. If VxVM is used, deport the disk group. (Example of command when using LVM) When using HP-UX servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 vgexport -m vg01.map /dev/vg01 vgexport -m vg02.map /dev/vg02 When using Linux servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 220

229 Chapter 5 Operation Examples Step 15. Separate ATgroup used for Copy Back Mode (standby server) At this step, the ATgroup is separated in order to preserve the differential data to the volume used for Copy Back Mode (RV in the ATgroup used for Copy Back Mode). (Command example) ismrc_separate -atg DB_Group_swap -wait This can also be done using ism. Figure 5-36 Separation of ATgroup Used for Copy Back Mode by ism (Example) 221

230 Chapter 5 Operation Examples Step 16. Mount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) (Windows) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) that is used after switching over business operations in the main site s application server is mounted. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RV. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (UNIX) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) that is used after switching over business operations at the main site s application server is set as an active volume group or the disk group is imported and mounted. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and the volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) When using HP-UX servers: vgchgid /dev/rdsk/c2t0d0 vgchgid /dev/rdsk/c2t0d1 mkdir /dev/vg01 mkdir /dev/vg02 mknod /dev/vg01/group c 64 0x mknod /dev/vg02/group c 64 0x vgimport -v -m vg01.map /dev/vg01 /dev/dsk/c2t0d0 vgimport -v -m vg02.map /dev/vg02 /dev/dsk/c2t0d1 vgchange -a y /dev/vg01 222

231 Chapter 5 Operation Examples (ii) vgchange -a y /dev/vg02 When using Linux Servers: vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online c2t0d0 vxdisk online c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall Mount the file system. (Command example when using LVM and mounting VxFS s file system) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log When using LVM on UNIX servers, after business operations have been recovered and checked, be sure to fetch and store the volume group setting information at the main site s application server. This volume group setting information should be backed up once business operations have been recovered and checked. In some cases, the volume group setting information is used during switchover of business operations or during fault repair. (Example of commands to fetch volume group setting information file) When using HP-UX servers: vgcfgbackup -f vg01_mv.conf /dev/vg01 223

232 Chapter 5 Operation Examples vgcfgbackup -f vg02_mv.conf /dev/vg02 When using Linux servers: vgcfgbackup /dev/vg01 vgcfgbackup /dev/vg02 Step 17. Prepare to switch over business operations (main site and backup site) At this step, the Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is used to recover the cluster configuration and change the network configuration in order to enable the resumption of business operations at the main site. Step 18. Resume business operations (application server) Business operations are resumed in the main site s application server. Step 19. Delete ATgroup used for Copy Back Mode where pair configuration of MV and RV has been switched (standby server) At this step, the ATgroup that was created for Copy Back Mode is deleted. (Command example of unpairing of pair registered to ATgroup) ismrc_atg -remove -atg DB_Group_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_atg -remove -atg DB_Group_swap -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_atg -remove -atg DB_Group_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv (Command example of ATgroup deletion) ismrc_atg -delete -atg DB_Group_swap -arrayname REMOTE_ARRAY (Command example of unpairing) ismrc_pair -unpair -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_pair -unpair -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_pair -unpair -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv These deletions can also be performed via ism. 224

233 Chapter 5 Operation Examples Figure 5-37 Unpairing by ism of Pairs Registered to ATgroup Used for Copy Back Mode (Example) Figure 5-38 Deletion by ism of ATgroup Used for Copy Back Mode (Example) 225

234 Chapter 5 Operation Examples Figure 5-39 Unpairing by ism of Pairs Used for Copy Back Mode (Example) 226

235 Chapter 5 Operation Examples Step 20. Unmount DDR pair s RV (backup site s backup server) (Windows) Replication of an ATgroup after recovery of business operations requires full-range copying, which means that some time is needed to reach the replicated state (Rpl/sync) after business operations are switched. Until the replicated state (Rpl/sync) is reached, data in the ATgroup s RVs is not consistent, so the data to be used for Copy Back Mode is backed up to a DDR pair s RV at a backup site. In preparation for backup, buffers are cleared in the respective DDR pair s RV which is recognized by the backup site s backup server. The server that is already accessing the target volume is stopped, the file system is flushed and unmounted, and then the file system s buffers are cleared. (i) (ii) Stop the service. (Command example) NET STOP ESMCommonService Flush and unmount the file system and clear the file system buffers. Since the volume to be unmounted is an RV, its drive path (mounted to drive letter or NTFS folder) is automatically deleted so that the RV cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (UNIX) Replication of an ATgroup after recovery of business operations requires full-range copying, which means that some time is needed to reach the replicated state (Rpl/sync) after business operations are switched. Until the replicated state (Rpl/sync) is reached, data in the ATgroup s RVs is not consistent, so the data to be used for Copy Back Mode is backed up to a DDR pair s RV at a backup site. In preparation for backup, the DDR pair s RV that is recognized by the backup site s backup server is unmounted. Since the RV information is overwritten over the MV information, when using LVM on HP-UX servers, the volume group information must be temporarily deleted. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Either set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. When using LVM on HP-UX servers, the volume group information is temporarily deleted after the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) When using HP-UX servers: vgchange -a n /dev/vg01 227

236 Chapter 5 Operation Examples vgchange -a n /dev/vg02 vgexport -m vg01.map /dev/vg01 vgexport -m vg02.map /dev/vg02 When using Linux servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline c2t0d0 vxdisk offline c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. 228

237 Chapter 5 Operation Examples Step 21. Replicate DDR pair (standby server) At this step, the backup site s DDR pair is replicated so that the data to be used for Copy Back Mode can be copied to the DDR pair s RV at the backup site. (Command example) ismrc_replicate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote This can also be done using ism. Figure 5-40 Replication by ism (Example) 229

238 Chapter 5 Operation Examples Step 22. Separate DDR pair (standby server) At this step, a DDR pair is separated so that the data to be used for Copy Back Mode can be preserved in the DDR s RV at the backup site. (Command example) ismrc_separate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote This can also be done using ism. Figure 5-41 Separation by ism (Example) 230

239 Chapter 5 Operation Examples Step 23. Mount DDR pair s RV (backup site s backup server) (Windows) At this step, the DDR pair s RV that is recognized by the backup site s backup server is mounted. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RVs. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (UNIX) The DDR pair s RV that is recognized by the backup site s backup server is mounted in the volume group is set as active or the disk group is imported and the DDR pair normal consistency state is restored. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on HP-UX servers, the volume group ID is changed, a directory is created for the volume group, and the volume group information is imported before the volume group is set as active. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) When using HP-UX servers: vgchgid /dev/rdsk/c2t0d0 vgchgid /dev/rdsk/c2t0d1 mkdir /dev/vg01 mkdir /dev/vg02 mknod /dev/vg01/group c 64 0x mknod /dev/vg02/group c 64 0x vgimport -v -m vg01.map /dev/vg01 /dev/dsk/c2t0d0 vgimport -v -m vg02.map /dev/vg02 /dev/dsk/c2t0d1 vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 When using Linux servers: 231

240 Chapter 5 Operation Examples (ii) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online c2t0d0 vxdisk online c2t0d1 * The disk specification (c2t0d0 and c2t0d1) mentioned above is an example when using an HP-UX server. Specify disks according to each environment since the specification varies depending on the platform. vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall Mount the file system. (Command example when using LVM and mounting VxFS s file system) When using HP-UX servers: mount -F vxfs /dev/vg01/lvol1 /db mount -F vxfs /dev/vg02/lvol1 /log When using Linux servers: mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) When using HP-UX or Solaris servers: mount -F vxfs /dev/vx/dsk/dg01/vol1 /db mount -F vxfs /dev/vx/dsk/dg02/vol1 /log When using Linux servers: mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log 232

241 Chapter 5 Operation Examples Step 24. Reset ATgroup (application server) Reset the ATgroup so that the main site s volume is the MV and the backup site s volume is the RV, which is the configuration for normal operation. (Command example of pair setting) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_pair -pair -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv (Command example of ATgroup creation) ismrc_atg -create -atg DB_Group -arrayname LOCAL_ARRAY -linkarrayname REMOTE_ARRAY (Command example of registration of pair to ATgroup) ismrc_atg -add -atg DB_Group -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_atg -add -atg DB_Group -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_atg -add -atg DB_Group -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv These settings can also be entered via ism. 233

242 Chapter 5 Operation Examples Figure 5-42 Pair Resetting by ism (Example) Figure 5-43 Re-creation of New ATgroup by ism (Example) 234

243 Chapter 5 Operation Examples Figure 5-44 Re-registration of Pair to ATgroup by ism (Example) Step 25. Unmount ATgroup s RV (when standby server is a Windows server) (Windows) To prepare for replication of the ATgroup after business operations have been restored, the file system buffers are cleared once again in the ATgroup s RV, and the RV cannot be accessed. Once the buffers have been cleared, the file system remains unmounted, so the stopped service is restarted. (i) Flush and unmount the file system and clear the file system s buffers once again. Since the volume to be unmounted is an RV, the drive path (mounted to drive letter or NTFS folder) is automatically deleted and the RV cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (ii) Restart the service. (Command example) NET START ESMCommonService 235

244 Chapter 5 Operation Examples Step 26. Replicate ATgroup (application server) At this step, the ATgroup is replicated and the normal configuration for operations is restored. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -cprange full -wait * Since this is done immediately after pair reconfiguration, full range copying is performed explicitly at this step. Figure 5-45 Replication by ism of ATgroup (Example) 236

245 Chapter 5 Operation Examples 5..5 Copy Back Mode through Swap for Siinglle RDR Paiir How to copy back data from RV to MV by pair configuration substitutions is described in Copy Back Mode by Pair Configuration Substitutions. In this RDR pair substitution, you perform unpairing in Step 19. Delete ATgroup used for Copy Back Mode where pair configuration of MV and RV has been switched (standby server) and reset the pair in Step 24. Reset ATgroup (application server). Although the supporting disk arrays are limited, the swap function for RDR pair enables you to immediately switch the roles of MV and RV, avoid full copy during the replication operation after the processing, and maintain the MV data redundancy by RV. However, when you use the swap function, the operation restarting time and the failure recovery procedures are different from the conventional operation. This section describes the points to be noted using a single RDR pair. Then, application to the ATgroup is described in the next section. 237

246 Chapter 5 Operation Examples Nottes on Copy Back Mode Procedure tthrough Swap ffor Siinglle RDR Paiir Notes on Copy Back Mode procedure using the swap for a single RDR pair is described based on the configuration diagram below. Main site Application server Backup server Backup site Standby server Backup server Tape Tape DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E Replicate DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOCAL_ARRAY LOCAL_ARRAY Disk array REMOTE_ARRAY REMOTE_ARRAY Disk array Copy Back Mode through swap for RDR pair Figure 5-46 Configuration Example of Data Switching Procedure through Swap for RDR Pair 238

247 Chapter 5 Operation Examples (1) Conventional Pair Configuration Substitutions (Unpairing and Resetting the Pair) (i) Conventional Procedure: Unpair the pair for Copy Back Mode where pair configuration of MV and RV has been switched (standby server) Unpair the RDR pair created for Copy Back Mode. (Unpairing example from the command line) ismrc_pair -unpair -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld or ismrc_pair -unpair -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv (ii) Conventional Procedure: Reset the pair (application server) Reset the pair using MV as the main site volume and RV as the backup site volume and configure normal operation configuration. (Pair setting example from the command line) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld or ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv (2) Pair Configuration Substitutions by Swap Procedure by swap: Swap the pair for Copy Back Mode (standby server) Swap the pair for Copy Back Mode and make normal operation configuration. (Command example) ismrc_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld or ismrc_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv 239

248 Chapter 5 Operation Examples * Note that the main site volume returns to MV and the backup site volume returns to RV after performing the swap. However, in pair swap operation, the time when the operation is restarted at the main site application server and the time when the parallel and serial DDR pairs are backed up are different as shown in the table below compared with when you unpair and reset the pair. Table 5-2 Differences between Unpairing/Resetting Operation and Pair Swap Operation Differences Unpairing/resetting operation Pair swap operation MV and RV mounting states It is not required to unmount MV and RV when unpairing or resetting the pair. You can unpair and reset the pair at the main site application server after the operation is restarted. You need to unmount both MV and RV in advance because their update is prohibited. It is not possible to restart the operation before pair swapping because the RV at the main site cannot be updated when swapping the pair. Target pair s replication state Parallel/serial pair s replication state The pair must be in the separated state to perform unpairing. However, if data is different between MV and RV when the pair is unpaired or set, it is not a problem. Because the RV at the main site is separated, the operation can be restarted and data can be different between MV and RV. The parallel/serial pair s replication state is not specified. You can unpair and reset the pair at the main site application server by copying data to DDR after the operation is restarted. MV and RV must be in the replicated state (Rpl/sync) where the data in both volumes is the same when performing the swap. It is not possible to restart the operation before swapping the pair because the RV at the main site is being replicated. Because MV and RV cannot be updated, the parallel/serial pair must be in the replicated or separated state where copy operations to MV or RV are not executed (The restored or separating state is not usable). To ensure the MV redundancy at the backup site, you need to complete copying and separation of the backup to the parallel DDR pair before performing the swap. 240

249 Chapter 5 Operation Examples Recovery Procedure att Paiir Swap Faiillure This section describes the recovery procedure when pair swap is abnormally terminated during Copy Back Mode through swap for RDR pair. (1) Determination of Pair State and Recovery Method When Pair Swap Is Abnormally Terminated Determine the recovery method based on the pair state in the table below if the swap is abnormally terminated. Table 5-3 Pair State and Recovery Method When Pair Swap is Abnormally Terminated Pair state State Recovery method Switching Sync state MV/RV update determination state prevention state result MV and RV roles not swapped Replicate synchronous Replicate synchronous Separate Non-update prevention state MV and/or RV is in the update prevention state. MV and RV are in the update prevention state. Swap not yet performed Failure during preprocessing before swap During swap Remove the cause of failure and then proceed to the recovery procedure. (1) Perform the swap operation again. Remove the cause of failure and then proceed to the recovery procedure. (1) Perform forced cancellation of update prevention to MV or RV that is in the update prevention state. (2) Perform the swap operation again. Remove the cause of failure and then proceed to the recovery procedure. (1) Perform forced cancellation of update prevention to MV and RV. (If the message ID ism13994 * error occurs, use the pair switching by unpairing and resetting. Perform the steps (4) to (6) below.) (2) Replicate the pair. (3) Execute the swap command again. MV and RV roles swapped Separate Separate MV and/or RV is in the update prevention state. Update non-prevention state Failure during postprocessing after swap Swap completed (Execute the following steps if he message ID ism13994 * error occurs.) (4) Perform forced unpairing of MV and RV. (5) Perform forced cancellation of update prevention to the Copy Back Mode and business volumes. (6) Set the pair in the configuration (normal operation configuration) to be switched. Remove the cause of failure and then proceed to the recovery procedure. (1) Perform forced cancellation of update prevention to MV or RV that is in the update prevention state. This pair has been swapped and the recovery process is not needed. * For the error message with message ID ism13994, refer to the Messages Handbook. 241

250 Chapter 5 Operation Examples (2) Checking the Pair State Using the Command (i) Link status check Check that the link state is normal. ismrc_arayinfo -arrayname REMOTE_ARRAY -linfo * Check the link state by referring to LOCAL_ARRAY and Path State in the command execution result. (ii) Pair switching state check Check that the roles of MV and RV in the pair have been switched by the swap operation. You can determine the recovery procedure based on this state. (Example using the command for confirmation) Execute this command for the volume that was MV before the swap. ismrc_sense -vol DB_RV -volflg ld -attr * If Attribute in the command execution result shows MV, MV and RV roles have not yet been switched. When this value is RV or RV/MV, MV and RV roles have been switched. (iii) Pair sync state check Check the pair sync state. You can determine the recovery procedure based on this state. (Example using the command for confirmation) ismrc_query -mv BD_RV -mvflg ld * Check the pair sync state by referring to Sync State in the command execution result. (iv) MV and RV update prevention state check Check the MV and RV update prevention state. You can determine the recovery procedure based on this state. (Command example) ismrc_sense-vol DB_RV -volflg ld -updprevent ismrc_sense -vol DB_MV -volflg ld -updprevent or ismrc_sense -file volume.txt -updprevent * volume.txt setting example #Type:LD ld:db_rv ld:db_mv * Check the MV and RV update prevention state by referring to Update Prevention State in the command execution result. 242

251 Chapter 5 Operation Examples (3) Pair State Check from the ism Screen You can check the pair state and determine the recovery procedure from the ism screen. (i) Link state check You can check this state in the [Path State] field on the Disk Array Link Information tab in the replication management screen. Figure 5-47 Link State Check Example with ism (ii) Pair switching state, pair sync state and MV/RV update prevention state check You can check these states in the [Type], [Sync State], and [Update Prevention State] fields on the Replication Information tab in the replication management screen. 243

252 Chapter 5 Operation Examples Figure 5-48 Example of Checking Pair and Other States with ism (4) Removing Cause of Failure If the pair swap operation is abnormally terminated, refer to the output error message in Messages Handbook and take appropriate action against the error. 244

253 Chapter 5 Operation Examples (5) Example of Recovery Procedure at Pair Swap Failure In this section, a failure that occurred during the swap is used as example (if other pair states are caused by failure, perform one of the steps (1) to (3) to complete the swap). [Operations at main site] [Operations at backup site] Step 1. Perform forced cancellation of update prevention to MV and RV in the pair used for Copy Back Mode (standby server) Step 2. Replicate the pair used for Copy Back Mode (standby server) Step 3. Swap the pair used for Copy Back Mode (standby server) Step 4a. Perform forced unpair of RV in the pair used for Copy Back Mode (application server) Step 4b. Perform forced unpair of MV in the pair used for Copy Back Mode (standby server) Step 5a. Perform forced cancellation of update prevention to the Copy Back Mode volume (application server) Step 5b. Perform forced cancellation of update prevention to the business volume (standby server) Step 6. Set the pair for the configuration to be switched (configuration for normal operation) (application server) * If the error with the message ID ism13994 occurs at step 1, perform steps 4 to 6. * If recovery from swap error is completed at step 3, steps 4 to 6 are unnecessary. Figure 5-49 Flow Example of Recovery Procedure at Pair Swap Failure 245

254 Chapter 5 Operation Examples Step 1. Perform forced cancellation of update prevention to MV and RV in the pair used for Copy Back Mode (standby server) Perform forced cancellation of update prevention for MV and RV in the pair used for Copy Back Mode. (Command example) ismrc_updprevent -vol DB_RV -volflg ld -mode cancel -force ismrc_updprevent -vol DB_MV -volflg ld -mode cancel -force or ismrc_updprevent -file volume.txt -mode cancel -force * volume.txt setting example #Type:LD ld:db_rv ld:db_mv * If the error with the message ID ism13994 occurs at step 1, perform steps 4 to 6. For the error message with message ID ism13994, refer to the Messages Handbook. You can perform this operation from ism by selecting the pair on the replication management screen and performing the forced cancellation of update prevention operation. Figure 5-50 Example of Forced Cancellation of Update Prevention by ism 246

255 Chapter 5 Operation Examples Step 2. Replicate the pair used for Copy Back Mode (standby server) To swap the pair, replicate the pair used for Copy Back Mode and wait until it becomes replicate synchronous. (Command example) ismrc_replicate -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld -wait or ismrc_replicate -file pair_swap.txt -wait * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv You can perform this operation from ism by selecting the pair on the replication management screen and performing the replication operation. Step 3. Swap the pair used for Copy Back Mode (standby server) Perform the pair swap operation again. (Command example) ismrc_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld or ismrc_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv * Note that the main site volume returns to MV and the backup site volume returns to RV. * If recovery from swap error is completed at this step, you do not have to restore steps 4 to 6. Perform the following steps only when the procedure fails due to the error with the message ID ism If recovery succeeds in steps 1 to 3, the steps after 4 are not needed. Step 4. Perform forced unpair of MV and RV in the pair used for Copy Back Mode (business and standby servers) Forcibly unpair MV and RV in the pair used for Copy Back Mode. (Example of forcibly unpairing MV from the command line) (standby server) ismrc_pair -unpair -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld -force mv or ismrc_pair -unpair -file pair_swap.txt -force mv (Example of forcibly unpairing RV from the command line) (application server) ismrc_pair -unpair -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld -force rv or ismrc_pair -unpair -file pair_swap.txt -force rv * pair_swap.txt setting example 247

256 Chapter 5 Operation Examples #Type:MV ld:db_rv Type:RV ld:db_mv You can perform this operation from ism by selecting the pair on the replication management screen and performing the forced unpairing operation. Step 5. Perform forced cancellation of update prevention to the Copy Back Mode and business volumes (business and standby servers) Perform forced cancellation of update prevention from each site. (Command example) (standby server) ismrc_updprevent -vol DB_RV -volflg ld -mode cancel -force or ismrc_updprevent -file volume.txt -mode cancel -force * volume.txt setting example #Type:LD ld:db_rv (Command example) (application server) ismrc_updprevent -vol DB_MV -volflg ld -mode cancel -force or ismrc_updprevent -file volume.txt -mode cancel -force * volume.txt setting example #Type:LD ld:db_mv Step 6. Set the pair for the configuration (configuration for normal operation) to be switched (application server) Set the pair to be switched by the swap as the normal operation configuration pair with the pair setting operation. (Pair setting example from the command line) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld or ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv You can perform this operation from ism by selecting the volume that will be MV on the replication management screen and performing the pair setting operation. 248

257 Chapter 5 Operation Examples 5..6 Copy Back Mode through Paiir Swap iin RDR//DR Confiiguratiion The pair configuration is substituted after the main site is restored in the operation procedure described in Copy Back Mode by Pair Configuration Substitutions. When you substitute the pair configuration by unpairing and resetting the pair, the whole area is copied during the ATgroup replication operation. Therefore, replication takes time and RV in the ATgroup is not consistent until completion of replication. If you use the swap function for RDR pair to substitute pair, you can avoid full copy during the ATgroup replication after the processing and immediately ensure the consistency of RV in the ATgroup. This section describes the procedure for copying back the data by swap for RDR pair by showing differences from Copy Back Mode by Pair Configuration Substitutions using the configuration example shown in Figure 5-28 Copy Back Mode by Pair Configuration Substitutions (Example). 249

258 Chapter 5 Operation Examples Copy Back Mode Procedure tthrough Swap ffor RDR Paiir (1) Flow of Copy Back Mode Procedure through Swap for RDR Pair The steps 1 to 15 are the same as in Copy Back Mode by Pair Configuration Substitutions. [Operations at main site] [Operations at backup site] Step 1. Repair fault at main site (main site) Step 2a. Create volume list (application server and backup server) Step 2b. Create volume list (standby server and backup server) Step 3. Create new ATgroup for Copy Back Mode during MV and RV pair configuration substitution (standby server) Step 4. Unmount restoration volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 5. Replicate ATgroup used for Copy Back Mode (standby server) Step 6. Secure pause point (standby server) Step 7. Separate ATgroup used for Copy Back Mode (standby server) Step 8. Resume business operations (application server) Step 9. Mount restoration volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 10. Restore and check business operations (application server) Step 11. Unmount restoration volume (RV in the ATgroup used for Copy Back Mode) (application server) 250

259 Chapter 5 Operation Examples Step 12. Replicate ATgroup used for Copy Back Mode (standby server) Step 13. Stop business operations (standby server) Step 14. Unmount business volume (MV in the ATgroup used for Copy Back Mode) (standby server) Step 15. Separate Atgroup used for Copy Back Mode (standby server) Step 16. Unmount DDR pair s RV (backup server) Step 17. Replicate DDR pair (standby server) Step 18. Separate DDR pair (standby server) Step 19. Mount DDR pair s RV (backup server) Step 20. Delete ATgroup used for Copy Back Mode during MV and RV pair configuration substitution (standby server) Step 21. Replicate the pair used for Copy Back Mode (standby server) Step 22. Swap the pair used for Copy Back Mode (standby server) Step 23. Recreate ATgroup (application server) Step 24. Replicate ATgroup (application server) Step 25. Mount MV in ATgroup (application server) Step 26a. Prepare for operation substitution (main site) Step 26b. Prepare for operation substitution (backup site) Step 27. Resume business operations (application server) 251

260 Chapter 5 Operation Examples * If you can verify whether to restart operation on the application server at the main site while the operation is stopped, steps 6 to 12 are not needed. * If an error occurs during pair swapping in step 22, refer to Recovery Procedure at Pair Swap Failure for the restoration procedure. * The flow of this procedure and the following operation procedure is an example. Perform the operations and procedures in the specific way for each environment. Figure 5-51 Flow Example of Copy Back Mode Procedure by Swapping the Pair For steps for Copy Back Mode of RemoteCluster, refer to the RemoteCluster with MC/ServiceGuard User s Manual. (2) Operation Steps For the steps 1 to 13, refer to the same steps in Copy Back Mode by Pair Configuration Substitutions. Step 14. Unmount business volume (MV in the ATgroup used for Copy Back Mode) (Windows) At this step, the file system in the business volume (MV in ATgroup used for Copy Back Mode) is unmounted. Specifically, the current service that accesses the target volume is stopped and the business volume s file system is flushed and unmounted. This volume must be unmounted by specifying -offline because it will be swapped. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (UNIX) At this step, the business volume (MV in ATgroup used for Copy Back Mode) is unmounted. Since the information in the business volume (MV in ATgroup used for Copy Back Mode) is overwritten during replication that follows switchover of business operations, when using LVM on HP-UX servers, the 252

261 Chapter 5 Operation Examples volume group information must be temporarily deleted. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. When using LVM on HP-UX servers, the volume group information is temporarily deleted after the volume group is set as inactive. If VxVM is used, deport the disk group. (Example of command when using LVM) When using HP-UX servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 vgexport -m vg01.map /dev/vg01 vgexport -m vg02.map /dev/vg02 When using Linux servers: vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 For step 15, refer to the same step in Copy Back Mode by Pair Configuration Substitutions. For steps 16 to 19, refer to steps 20 to 23 in Copy Back Mode by Pair Configuration Substitutions. Step 20. Delete ATgroup used for Copy Back Mode during MV and RV pair configuration substitution (standby server) Delete the ATgroup created for Copy Back Mode only. (ATgroup pair cancellation example from the command line) ismrc_atg -remove -atg DB_Group_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_atg -remove -atg DB_Group_swap -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_atg -remove -atg DB_Group_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv 253

262 Chapter 5 Operation Examples (ATgroup deletion example using the command) ismrc_atg -delete -atg DB_Group_swap -arrayname REMOTE_ARRAY Step 21. Replicate the pair used for Copy Back Mode (standby server) Replicate the pair used for Copy Back Mode and copy differential data under operation recovery check to the volume used for Copy Back Mode (RV of the pair used for Copy Back Mode). (Command example) ismrc_replicate -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_replicate -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_replicate -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv Step 22. Swap the pair used for Copy Back Mode (standby server) Swap the pair used for Copy Back Mode and switch the sites. (Pair swap example from the command line) ismrc_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_swap -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv Step 23. Recreate ATgroup (application server) Reset the ATgroup and make normal operation configuration because the main site volume is MV and the backup site volume is RV. (ATgroup pair registration example from the command line) ismrc_atg -create -atg DB_Group -arrayname LOCAL_ARRAY -linkarrayname REMOTE_ARRAY (ATgroup pair registration example from the command line) ismrc_atg -add -atg DB_Group -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_atg -add -atg DB_Group -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_atg -add -atg DB_Group -file pair.txt 254

263 Chapter 5 Operation Examples * pair.txt setting example #Type:MV ld:db_mv ld:log_mv Type:RV ld:db_rv ld:log_rv Step 24. Replicate ATgroup (application server) Replicate the ATgroup and restore it to the normal operation configuration. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -wait Step 25. Mount MV in ATgroup (application server) Refer to step 16 in Copy Back Mode by Pair Configuration Substitutions. Step 26. Prepare for operation substitution (main and backup sites) Restore the cluster configuration, change the network configuration and perform other necessary operations using the Copy Back Mode volume (RV in the ATgroup for Copy Back Mode) to restart the operation at the main site. Step 27. Resume business operations (application server) Resume the operation on the application server at the main site. 255

264 Chapter 5 Operation Examples Recovery Procedure att Paiir Swap Faiillure If a failure occurs while multiple pairs consisting of the ATgroup are swapped, check each pair s state according to Recovery Procedure at Pair Swap Failure, categorize each pair and perform appropriate recovery. (1) Swapped pair Swap has already been completed and measures are unnecessary. (2) Pair for which error has occurred during swap Restoration from swap failure is required. (3) Pair not swapped You need to swap the pairs not yet swapped. When recovery from swap failure is completed and all the pairs are swapped, you can register a pair with the new ATgroup. If some failed pairs are restored by pair switching by forced unpairing and setting instead of pair swap, you can register them with other pairs switched through swapping in the ATgroup. However, the pairs restored through forced unpairing and pair setting is fully copied during the ATgroup replication afterward and the replicated state (Rpl/sync) is delayed. 256

265 Chapter 6 Notes Chapter 6 Notes The following are notes related to disaster recovery functions for data replication Speciifiicatiions of ATgroup The specifications of ATgroups are described below. (For S4300, S3300, and 2800 series) (1) Up to 64 ATgroups can be built. (2) Up to 128 pairs can be registered to an ATgroup. (3) The total number of pairs that can be registered to ATgroups is 4096 pairs. (For S4900 and D8 series) (1) Up to 256 ATgroups can be built. (2) For S4900, up to 8192 pairs can be registered to an ATgroup. For D8 series, up to 1024 x the number of nodes (1 to 4) pairs can be registered to an ATgroup. (3) For S4900, the total number of pairs that can be registered to ATgroups is 8192 pairs. For D8 series, the total number of pairs that can be registered to ATgroups is 1024 x the number of nodes (1 to 4). (4) Up to 64 ATgroups (concentrators) can be concurrently replicated in semi synchronous copy mode of order guarantee. (5) The maximum number of ATgroups which can be created between the disk arrays with different specifications of the number of ATgroups ( Maximum Number of Concentrator in the disk array properties) is based on the disk array with lower specification. In this case, concentrator numbers are limited in the 0000h to 003fh range and up to 64 ATgroups can be registered. Note that RDR pair created between the disk arrays with different specifications cannot be registered to the ATgroup whose concentrator number exceeds 003fh. 257

266 Chapter 6 Notes 6..2 Notes on Buiilldiing an Enviironment Notes on building the environment are listed below. (1) Be sure to set unique disk array names and logical disk names for disk arrays in the system. (2) For ATgroup names, be sure to enter only alphanumeric characters, underbars (_), or forward slashes (/) (up to 32 characters), and be sure that each ATgroup name is unique within the system. (3) Each pair registered to an ATgroup must meet the following conditions. DDR pairs cannot be registered to an ATgroup. Only RDR pairs can be registered to an ATgroup. A pair that is not the top-level pair cannot be registered to an ATgroup. Only the top-level pair can be registered. If the MV of the pair to be registered includes several RDR pairs, it cannot be registered to an ATgroup. Each RDR pair must be set as a single pair registered to an ATgroup. A registered pair in an ATgroup and any pair to be registered to an ATgroup must be separated. An RDR pair that exists among different disk arrays cannot be registered to a single ATgroup. (The RDR pair must be in the same disk array.) If the RV is used as a base-volume (BV) of snapshot function, it cannot be registered to an ATgroup. It is required that the RV is an RDR pair not in serially configured with the snapshot function. (4) When a Windows server uses a volume that is in an ATgroup, if the ATgroup is replicated or restored, the disk s author information will be identical for the MV and RVs within the ATgroup. (5) When a Windows server uses a volume that is in an ATgroup, the MV and RVs within that ATgroup cannot be used on the same server. (6) A control volume must be built and set on all the servers where ReplicationControl/DisasterRecovery is used. (7) When a server using ReplicationControl/DisasterRecovery is a Windows server, automount disabling must be set. (8) When building a disaster recovery system for the Windows platform, it is recommended that ActiveDirectory be configured within a single domain in order to facilitate switching over of business operations after a disaster. (9) Exchange Server makes data takeover by using data copy on a storage basis more difficult and is not suitable for a disaster recovery system using RemoteDataReplication/DisasterRecovery. (10) When a pair whose sync state is fault or forced separate (cancel) is registered to ATgroup, the ATgroup sync state is automatically transited to fault state. (11) It is required that versions of ReplicationControl and ReplicationControl/DisasterRecovery to be used support the disk array connected to the server and further support the disk array that is RDR-related to that disk array. (12) When creating an ATgroup with the ismrc_atg command and if the disk array on the concentrator side supports the function specifying the link disk array ( Function of specifying Link Disk Array Name on creating an ATgroup in the disk array properties is ON (display the disk array properties of the disk array to be created from the replication screen of ism to check the setting)), it is required to specify the 258

267 Chapter 6 Notes link disk array with the linkarrayname option. (13) When an error such as SCSI I/O error occurs during creating an ATgroup with the ismrc_atg command, the ismrc_atg command terminates abnormally but an incomplete ATgroup midway through creation may be left. In this case, delete the relevant ATgroup and re-create. (14) Observe notes related to the use of GPT disk when building the disaster recovery system using the GPT disk. For details, refer to the Data Replication User s Manual (Installation and Operation Guide for Windows). (15) If an ATgroup is created in the version that supports the function to change the allowed response time of MV, ATgroups must be deleted in the same version. If an ATgroup is deleted in an unsupported version, the allowed response time of MV may remain. 259

268 Chapter 6 Notes 6..3 Notes for Operatiions Notes for operations are listed below. (1) When the ATgroup consistency state is Non-atomic, the RV data included in that ATgroup is not consistent data except when separation has been completed at a pause point in business operations. Thus, it is not normally used for recovery of business operations. (2) When ATgroup replication is executed, the ATgroup consistency state becomes Non-atomic until replication is completed, and RV data is not consistent. Before replicating the ATgroup, be sure to back up the data to a separate volume or take similar steps to ensure that data consistency is not lost. (3) When ATgroup replication is performed in background copy mode, the ATgroup consistency state will remain Non-atomic even when replicate synchronous (Rpl/sync) is set. (4) If the ATgroup s replication is executed while in background copy mode, an atomic-break will not be executed. (5) If MV Access Stop is set for MV Access after Atomic-break, access for business operations may become disabled. Also, business operations may be stopped unexpectedly, so this setting should normally be avoided. (6) When a disaster occurs at the main site and is followed by a disk array fault or line fault, a link fault is detected by the backup site s ism. Once a link fault has been detected, a log with the message ID ism10402 from ism is recorded. For details on ism s logs, refer to the Messages Handbook. (7) If a link fault occurs, possibly due to a disk array fault or line fault, the link fault is detected by ism but the ATgroup sync state and pair state on the distributor side (RV side) does not change. This is because the state of the copy source for the distributor side (RV side), i.e., the concentrator side (MV side), cannot be referenced, so the state immediately prior to the disaster is maintained on the distributor side (RV side). (8) When disk arrays or ATgroups become abnormal due to a loss of the opponent disk array caused by a disaster or a fault, etc., and invalid ATgroup information remains, be sure to perform the following steps to delete the invalid ATgroup information. (When invalid ATgroup information remains on concentrator side (MV side)) (i) Separate the ATgroup or execute forced separation of the ATgroup from the concentrator side (MV side). (ii) Execute Forced deletion of ATgroups from the concentrator side (MV side). (When invalid ATgroup information remains on distributor side (RV side)) (i) Execute Forced deletion of ATgroups from the distributor side (RV side). (9) The following conditions must be met when executing simultaneous replication of serially configured pairs. Replication at the second and lower levels is enabled only when the copy mode is background copy mode. When copying of former and latter pairs cannot be performed in different directions at the same time (such as performing replication while former and latter pairs are being restored). 260

269 Chapter 6 Notes Replication control cannot be performed when the former or latter pair s activity state is separate, or when its sync state is separating. (10) The following conditions must be met when executing remote operation commands. Operations using pairs on the remote side must be linked to pairs in a volume that can be accessed. Any pair targeted for remote operations and any MV or RV in a pair belonging to a linked accessible volume must be registered in a volume list. Remote operations can be executed for linked pairs up until the next level. Logical disk names must be specified for any MV or RV for which a remote command is set. (11) Immediately after ATgroup separation, the RV can be used, so RV Separate (immediate) cannot be used. (12) ATgroup restoration is only an RV store (protect) operation, and does not include any updating of RVs. When restoration of the ATgroup is completed, the ATgroup sync state becomes separated. (13) When executing ATgroup restoration, be sure to check that all the pairs registered to the ATgroup are in a restoration executable state. If there is a pair that cannot be restored at ATgroup restoration, the start of the pair restoration will fail and pairs to be restored co-exist with those not to be restored. In the result, the ATgroup is placed in the restore execution (Rst/exec) state. In this case, re-execute ATgroup restoration by either of the following methods. Make sure that the ATgroup state automatically changes to separated and troubleshoot the cause of a failure, then re-execute ATgroup restoration. Execute forced separation for ATgroup once and perform the operation to recover from fault separation using the replication management screen or the ismrc_change command to change the ATgroup state from fault to separated. Then re-execute ATgroup restoration. (14) ATgroup restoration cannot be executed from fault state. Recover from fault separation beforehand to restore the ATgroup state from fault to separated, and execute restoration. (15) In Ver 3.4 or earlier, the ATgroup pair-related information was always output to a console in the start or end message of the processing when operating the ATgroup using the ismrc_replicate, ismrc_separate, ismrc_restore, ismrc_change, or ismrc_wait commands of ReplicationControl/DisasterRecovery. However, it is not output for the default in Ver 4.1 or later. To output the ATgroup pair-related information in the start or end message of the processing in Ver 4.1 or later, specify OUTPUT for the RPL_ATGPAIRMSG parameter of the replication operation option setting file (for Windows) or the environment variable (for UNIX). (16) On Linux servers, if an attempt is made to access a volume in the Not Ready state or Read Only state, such as an access-restricted RV, a message indicating an SCSI error (refer to the <message examples> below) may be recorded in syslog. If you execute a command, such as the fsck command, that accesses all volumes in the server, the same message may be recorded for all volumes in the Not Ready state in the server. <Message example> Dec 10 14:48:14 sv001 kernel: SCSI error :< > return code = 0x Dec 10 14:48:14 sv001 kernel: Info fld=0x0, Current sde: sense key Data Protect Dec 10 14:48 14 sv001 kernel: ASC=f4 ASCQ= a Dec 10 14:48:14 sv001 kernel: end_request: I/O error, dev sde, sector 0 261

270 Chapter 6 Notes In an environment that has PathManager installed, if such a message is recorded, the PathManager path to the volume may be broken, preventing the volume from being accessed. Separate the pair or change the RV access restriction to make the volume read/write-enabled, and then recover the broken path using the recoverpath command of PathManager. If the path to a volume a volume in Read Only state is broken, recovery is automatically performed by path monitoring of PathManager after a fixed time period (monitoring interval). For details on PathManager, refer to the PathManager User s Manual (Linux). In an environment that does not have PathManager installed, there is no operational problem even if a message similar to the above example is recorded. (17) When configuring RDR between disk arrays of different models, there may be difference in functions supported by each disk array. Available functions for RDR pair operations and remote operations are limited to the functions that are mutually supported by the disk arrays, thus you cannot perform operations exceeding this function range. (18) If the following events occur or operations are executed while replicate, separate, or restore is being operated for an ATgroup, the synchronous state of the ATgroup will not make an automatic transition to the next state: Abnormal end of ism server, ism client, etc. Forced end by task manager instead of end by the [Finish] button on the screen. Failed in processing execution of some pairs in the ATgroup. In such cases, execute operation for the relevant ATgroup again. (19) When you perform [Rescan Disks] and [Refresh] in Windows [Disk Management] if there is a volume of which state is Not Ready or Not Available, such as an access-restricted RV, on a Windows Server 2008, the following error message may be recorded in the event log. However, the operation is not affected. <Message example> Source Virtual Disk Service Event ID 1 Description: An unexpected error has occurred. Error code: 13@ * A different value may be displayed for the error code. 262

271 Appendix A License Appendix A License A..1 Apache llog4jj Liicense Apache License Version 2.0, January TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. 263

272 Appendix A License "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable 264

273 Appendix A License (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and 265

274 Appendix A License do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, 266

275 Appendix A License incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, 267

276 Appendix A License WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. 268

277 Appendix A License A..2 Fllex Liicense The flex manual is placed under the same licensing conditions as the rest of flex: Copyright c 2001, 2002, 2003, 2004, 2005, 2006, 2007 The Flex Project. Copyright c 1990, 1997 The Regents of the University of California. All rights reserved. This code is derived from software contributed to Berkeley by Vern Paxson. The United States Government has rights in this work pursuant to contract no. DE-AC03-76SF00098 between the United States Department of Energy and the University of California. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the name of the University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED AS IS AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 269

278 Appendix A License A..3 OpenSSL Liicense This is a copy of the current LICENSE file inside the CVS repository. LICENSE ISSUES ============== The OpenSSL toolkit stays under a dual license, i.e. both the conditions of the OpenSSL License and the original SSLeay license apply to the toolkit. See below for the actual license texts. Actually both licenses are BSD-style Open Source licenses. In case of any license issues related to OpenSSL please contact [email protected]. OpenSSL License /* ==================================================================== * Copyright (c) The OpenSSL Project. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * 3. All advertising materials mentioning features or use of this * software must display the following acknowledgment: * "This product includes software developed by the OpenSSL Project * for use in the OpenSSL Toolkit. ( * * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to * endorse or promote products derived from this software without * prior written permission. For written permission, please contact 270

279 Appendix A License * [email protected]. * * 5. Products derived from this software may not be called "OpenSSL" * nor may "OpenSSL" appear in their names without prior written * permission of the OpenSSL Project. * * 6. Redistributions of any form whatsoever must retain the following * acknowledgment: * "This product includes software developed by the OpenSSL Project * for use in the OpenSSL Toolkit ( * * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED * OF THE POSSIBILITY OF SUCH DAMAGE. * ==================================================================== * * This product includes cryptographic software written by Eric Young * ([email protected]). This product includes software written by Tim * Hudson ([email protected]). * */ Original SSLeay License /* Copyright (C) Eric Young ([email protected]) * All rights reserved. * * This package is an SSL implementation written * by Eric Young ([email protected]). * The implementation was written so as to conform with Netscapes SSL. * 271

280 Appendix A License * This library is free for commercial and non-commercial use as long as * the following conditions are aheared to. The following conditions * apply to all code found in this distribution, be it the RC4, RSA, * lhash, DES, etc., code; not just the SSL code. The SSL documentation * included with this distribution is covered by the same copyright terms * except that the holder is Tim Hudson ([email protected]). * * Copyright remains Eric Young's, and as such any Copyright notices in * the code are not to be removed. * If this package is used in a product, Eric Young should be given attribution * as the author of the parts of the library used. * This can be in the form of a textual message at program startup or * in documentation (online or textual) provided with the package. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * "This product includes cryptographic software written by * Eric Young ([email protected])" * The word 'cryptographic' can be left out if the rouines from the library * being used are not cryptographic related :-). * 4. If you include any Windows specific code (or a derivative thereof) from * the apps directory (application code) you must include an acknowledgement: * "This product includes software written by Tim Hudson ([email protected])" * * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, 272

281 Appendix A License * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY * WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * The license and distribution terms for any publically available version or * derivative of this code cannot be changed. i.e. this code cannot simply be * copied and put under another distribution license * [including the GNU Public License.] */ 273

282 Index A AccessControl...35 application servers...34 ATgroup...11 ATgroup build steps...43 ATgroup configuration display function...13 ATgroup consistency states...16 ATgroup operation commands ATgroup replication...19 ATgroup replication control and state transitions...20 ATgroup replication control functions...19 ATgroup setting function...13 ATgroup state display function...16 ATgroup sync state... 17, 18 Atomic...16 atomic group...11 atomic-break function...22 B background copy mode...7 backup servers...34 F Fault...17 Fault recovering...17 Fault separation...17 Fault(Certifying)...29 Fault(Comparing)...29 Fault/recovering...17 freshness of backup data and use format...8 H hardware...34 I Identical...29 install software...39 Install software and build environment...33 installation...40 installation of RemoteDataReplication/DisasterRecovery...39 Invalid...16, 17 C Certified...29 Certifying...29 Comparing...29 concentrator...13 copy mode...4 copy mode options for disaster recovery systems...4 copy of order guarantee function...21 D Different...29 disaster...1 disaster recovery...1 disaster recovery use format...4 disk arrays...34 distances between sites...9 distributor...13 DynamicDataReplication...35 E example of Storage Manager layout...37 L layout and configuration of disaster recovery system...9 logical disk movement command...263, 269, 270 M management servers...34 management terminals...34 method to copy data to remote sites...4 MV Access after Atomic-break...22 MV Access Continue...22 MV Access Stop...22 N Non-atomic...16 Not Certified...29 notes for operations notes on building an environment O operation examples backup considerations for operations under various scenarios

283 P Copy Back Mode by pair configuration substitutions Copy Back Mode of data from RV to MV Copy Back Mode procedure through swap for RDR pair Copy Back Mode through pair swap in RDR/DR configuration Copy Back Mode through swap for single RDR pair line fault recovery notes on Copy Back Mode procedure through swap for single RDR pair recovery procedure at pair swap failure , 256 restoration for data Copy Back Mode use of RV when disaster occurs pairs registered to ATgroup...42 pairs that can be registered to ATgroup...14 pairs that cannot be registered to ATgroup...15 partition...31 partitioning function...31 PerformanceMonitor...35 PerformanceNavigator...35 placement of management servers...37 placement of sites...9 Prevent...28 R RDR quick sync...28 recovery from backup data...8 recovery from replication during replicating...8 recovery point objective...2 recovery time objective...2 remote pair operations...27 RemoteDataReplication...35 RemoteDataReplication/DisasterRecovery... 1, 35 Replicate start...17 Replicate synchronous...17 Replicating...17 ReplicationControl...35 ReplicationControl/DisasterRecovery... 35, 102 ReplicationControl/DisasterRecovery command ismrc_atg ismrc_change ismrc_query ismrc_replicate ismrc_restore ismrc_sense ismrc_separate ismrc_wait ReplicationControl/DisasterRecovery function build an ATgroup build ATgroup display ATgroup list display ATgroup s copy state display the ATgroup copy state display the ATgroup list recovery from fault separation , 123 remote operation commands , 148 replicate an ATgroup replicate ATgroup restore an ATgroup restore ATgroup separate an ATgroup separate ATgroup wait for change in ATgroup sync state wait for the ATgroup sync state to change ReplicationControl/DisasterRecovery installation and environment setting...39 ReplicationControl/DisasterRecovery operation method conditions for building an ATgroup conditions for displaying ATgroup list conditions for displaying ATgroup s copy state conditions for execution of remote operation commands conditions for recovery from fault separation conditions for replication of ATgroup conditions for restoration of ATgroup conditions for separation of ATgroup conditions for waiting for change in ATgroup sync state executable construct of ismrc_atg command executable construct of ismrc_change command executable construct of ismrc_query command executable construct of ismrc_replicate command executable construct of ismrc_restore command executable construct of ismrc_sense command executable construct of ismrc_separate command executable construct of ismrc_wait command executable construct of remote operation commands for pairs information displayed by ATgroup copy state display information displayed when building an ATgroup information displayed when displaying ATgroup list messages output during ATgroup replication messages output during ATgroup restoration messages output during ATgroup separation messages output when recovering from fault separation messages output when waiting for change in ATgroup sync state notes for recovery from fault separation notes on building an ATgroup notes on displaying the ATgroup copy state notes on displaying the ATgroup list notes on reparating an ATgroup notes on replicating an ATgroup notes on restoring an ATgroup notes on waiting for the ATgroup sync state to change options and parameters available for ATgroup s copy state display pairs available for remote operation commands executed from backup site pairs available for remote operation commands executed from main site pairs for which remote operation commands cannot be executed range of remote operation commands for pairs specifiable options and parameters for ATgroup list display specifiable options and parameters for ATgroup replication specifiable options and parameters for ATgroup restoration specifiable options and parameters for ATgroup separation specifiable options and parameters for build ATgroup.142 specifiable options and parameters for recovery from fault separation

284 specifiable options and parameters for wait for change in ATgroup sync state restoration of ATgroup...20 Restore suspend...17 Rpl/exec...17 Rpl/start...17 Rpl/suspend...17 Rpl/sync...17 RPO...2 Rst/exec...17 Rst/suspend...17 RTO...2 S selection of configuration and lines...10 semi-synchronous copy mode of order guarantee...6, 7 Sep/exec...17 Sep/start...17 Separate start...17 Separated...17 Separating...17 separation of ATgroup...19 simultaneous replication of serially configured pair...26 single fault point...10 software...35 specifications of ATgroup specified building disaster...2, 9 specified districts disaster...2, 9 specified system disaster...2 standby servers...34 storage control software...35 Storage Manager...35 Storage Manager s ATgroup function...56 ATgroup configuration display, status display...56 ATgroup forced operation...56 ATgroup information...57 ATgroup Information tab...57 ATgroup replication operations...56 ATgroup settings...56 ATgroup s batch settings...56 change the allowed response time of MV... 32, 56, 77 create an ATgroup... 56, 65 create/delete an ATgroup volume...71 delete a pair registered to an ATgroup... 32, 56 delete an ATgroup... 56, 69 display the ATgroup configuration... 32, 56 display the ATgroup state... 56, 59 fault repair...56 forced deletion of an ATgroup... 32, 56, 98 forced separation of an ATgroup... 32, 56, 92 items on menubar...57 output the ATgroup information list in CSV format...56, 100 recovery from fault separation... 32, 56, 95 register a pair to an ATgroup... 32, 56 rename an ATgroup... 32, 56, 75 replicate an ATgroup... 32, 56, 79 restore an ATgroup... 32, 56, 88 save an ATgroup and registered pair information... 32, 56, 101 save configuration information...56 separate an ATgroup... 32, 56, 85 Storage Manager s ATgroup function operation method...56 ATgroup Configuration and State Display screen...59 ATgroup Deletion screen...69 ATgroup Forced Deletion screen...98 ATgroup Forced Separate screen...92 ATgroup information screen...59 ATgroup Rename screen...75 ATgroup Replicate Execution screen...79 ATgroup Restore screen...88 ATgroup Separate screen...85 ATgroup Volume Creation/Deletion screen...71 caution point for ATgroup forced separation...94 caution point for forced deletion of ATgroup Change of the Allowed Response Time of MV screen...77 conditions for adding/deleting pair in ATgroup...74 conditions for changing the allowed response time of MV...78 conditions for creation of ATgroup...68 conditions for deletion of ATgroup...70 conditions for forced deletion of ATgroup...99 conditions for forced separation of ATgroup...94 conditions for recovering from fault separation...97 conditions for replication of ATgroup...83 conditions for restoration of ATgroup...90 conditions for revising ATgroup name...76 conditions for separation of ATgroup...87 displaying ATgroup Creation screen...65 handling of errors during ATgroup creation...67 handling of errors during ATgroup deletion...70 handling of errors during ATgroup forced separation...93 handling of errors during ATgroup replication...82 handling of errors during ATgroup restoration...90 handling of errors during ATgroup separation...86 handling of errors during forced deletion of ATgroup...99 handling of errors when adding or deleting pair in ATgroup...73 handling of errors when changing an allowed response time of MV...78 handling of errors when during ATgroup name revision...76 handling of errors when recovering from fault separation...96 notes on replicating an ATgroup...83 notes on restoring an ATgroup...91 notes on separating an ATgroup...87 Recover from Fault separation state screen...95 replication screen s disk array properties...63 summary of copy mode features...7 swap function for RDR pair...30 synchronous mode...5, 7 system configuration...33 T tape libraries...34 target hardware for software installation...36 U uninstallation...40 update...40 update prevention state...28 V volume comparing/identity certified...29 volume update prevention

285 VSPAR...31 W wide area disaster...2, 9 277