Whitepaper Storwize V7000 replication over TCP/IP with IBM i

Transcription

1 Whitepaper Storwize V7000 replication over TCP/IP with IBM i Version 1 Jana Jamsek Storage Advanced technical skills, Europe Jens Liebetrau IT Specialist, European Storage Competence Center, Mainz Table of contents Table of contents... 1 Introduction... 2 Notices and disclaimer... 3 Equipment... 4 Local site Production site... 4 Remote site Disaster Recovery (DR) site... 4 Setting-up the Storwize environment... 5 Configuring Ethernet Ports... 5 Establishing Remote Copy partnership... 6 Establishing Metro Mirror... 7 Status of Metro Mirror relationships at initial synchronization Establishing Global Mirror with Change Volumes Create Global Mirror target volumes Create Global Mirror consistency group Adding Global Mirror Change Volumes on remote site Start Global Mirror consistency group Disaster Recovery tests with Metro Mirror of IBM i Full System Planned outage of IBM i Stop Metro Mirror Re-start Metro Mirror from Production to DR site Re-start Metro Mirror from DR site to Production site and switch Metro Mirror from Production to DR site Copyright IBM Corporation, 2014, ESCC Page 1 of 53

2 Unplanned outage of IBM i Switch to Disaster Recovery site Switch back to Production site Outage of IP links Disaster recovery tests with Global Mirror with Change Volumes of IBM i Full System29 Planned outage of IBM i Stop Global Mirror Re-start Global Mirror from Production to DR site Re-start Global Mirror from DR site to Production site and switch Global Mirror from Production to DR site Unplanned outage of IBM i Switch to Disaster Recovery site Switch back to Production site Outage of IP links Disaster Recovery tests with Metro Mirror and PowerHA for i Planned outage of IBM i Switch to Disaster Recovery site Switch-back to Production site Unplanned outage of IBM i Failover to Disaster Recovery site with reverse replication Switch back to Production site Failover to Disaster Recovery site without reverse replication Switch back to Production site Outage of IP links Introduction With Storwize V7000 release 7.2 the possibilities for Remote Copy links broadened with enablement of native IP Remote Copy. With this option, Remote replication uses TCP/IP protocol with 1Gbit or 10Gbit Ethernet connections. It supports both Metro Mirror and Global Mirror and is covered by normal Remote Copy license. Support of Storwize V7000 for IBM i was available soon after the announcement of V7000. At the beginning the IBM i connection was possible in VIOS virtual SCSI mode, NPIV and native attachment followed a little later. IBM i solutions with Storwize Remote Copy are available and implemented at the customers. Up to now the implementations were based on V7000 Remote Copy over FC links. For this document we tested and described the implementations for IBM i, using Remote Copy over TCP/IP links. The steps used in IBM i and V7000 Graphical User Interface (GUI) to implement these solutions, are the same as for Remote Copy over FC links. Copyright IBM Corporation, 2014, ESCC Page 2 of 53

3 Disaster Recovery (DR) solutions for IBM i with external storage can be implemented in two ways: a. We use storage replication of all IBM i disk capacity including disk pool 1 (ASP1), and disk pools 2 to 32 (ASPs 2 32) if they are present. Such DR solutions are usually called Full System solutions. b. We use storage replication of an Independent disk pool (IASP). This kind of solutions are managed by PowerHA SystemMirror for IBM i (PowerHA for i) We tested Metro Mirror over IP with both solutions IBM i Full System and PowerHA for i. When implementing Global Mirror over IP it is recommended to use Global Mirror with Change Volumes, therefore we tested this way of asynchrone replication. Since PowerHA for i supports standard Global Mirror but not yet the Global Mirror with Change Volumes we performed only the Full System tests using Global Mirror with Change Volumes over IP. We provide detailed instructions and screen-shots for the actions related to Storwize Remote Copy. For IBM i related functions we specify high-level instructions, the detailed instructions can be found in the following documents: IBM i and IBM Storwize Family A Practical Guide to Usage Scenarios, SG Technical document Hints and Tips: V7000 in an IBM i Environment IBM i and IBM System Storage: A Guide to Implementing External Disks on IBM i, SG PowerHA SystemMirror for IBM i Cookbook, SG Notices and disclaimer Copyright 2014 by International Business Machines Corporation. No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation. Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This information may include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or programs(s) at any time without notice.. THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided. IBM is not responsible for the performance or interoperability of any non-ibm products discussed herein. Copyright IBM Corporation, 2014, ESCC Page 3 of 53

4 The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY U.S.A. IBM, the IBM logo, Storwize V7000, POWER6 and System Storage are trademarks of International Business Machines Corporation in the United States, other countries, or both. Other company, product or service names may be trademarks or service marks of others. Equipment For our testing we used the equipment described further in this section. Local site Production site Master Storwize V One node pair Software level 7.2 Two * 1 Gb Ethernet Ports (one per each controller) connected to IP network Disk pool of 24 * 300 GB 15 K RPM disk drives in 3 * RAID-10 arrays POWER MMA Production IBM i LPAR with 2 processors and 12 GB memory IBM i uses 8 * 140 GB LUNs in the Disk pool The 8 LUNs are in ASP1 IBM i uses NPIV connection with 2 VIOS in Multipath Remote site Disaster Recovery (DR) site Auxiliary Storwize V One node pair Software level 7.2 Two * 1 Gb Ethernet Ports (one per each controller) connected to IP network POWER MMA Disaster Recovery (DR) IBM i LPAR with 2 processors and 12 GB memory Disk pool of 6 * 2 TB 7 K RPM disk drives in one RAID-6 array IBM i uses 8 * 140 GB LUNs in the Disk pool IBM i uses NPIV connection with 2 VIOS in Multipath Copyright IBM Corporation, 2014, ESCC Page 4 of 53

5 Setting-up the Storwize environment In this section we provide step by step description how we setup the Remote Copy environment over IP links in Storwize V7000. Configuring Ethernet Ports In each primary and secondary V7000 we configured Ethernet Port 2 from each node for Remote Copy. To configure each port we used the following steps: 1. In Storwize GUI click on Settings -> Network -> Ethernet Ports -> right click on the port to use for Remote Copy -> select Modify 2. In Modify port window insert the IP address, subnet mask, gateway, at Remote Copy select Group 1 -> click OK, as shown on the picture below. Copyright IBM Corporation, 2014, ESCC Page 5 of 53

6 Note: a. In our test we checked iscsi Host: Enable to enable the ports for iscsi connection, however this is not necessary for Remote Copy setup. b. One Remote Copy port group can contain maximum 2 ports from each IO group on each local and remote site. In our example we used Group 1 for all the ports for Remote Copy. The configured Ethernet Ports are shown on the picture below. Establishing Remote Copy partnership On each primary and secondary V7000 we established the Remote Copy partnership for IP connection, by using the following steps: 1. In V7000 GUI click on Copy Services -> Partnerships -> Create Partnership 2. In the Create Partnership window select IP for the Type, insert the Management IP address of the Storwize to establish the partnership, insert the estimated bandwidth of the IP link and the percentage of the maximal bandwidth 64MB to be used by the background copy rate, insert the Challenge-Handshake Authentication Protocol (CHAP) authentication secret of the partner Storwize in the field Partner's system CHAP secret. Creating the partnership in our example is shown in the picture below. Copyright IBM Corporation, 2014, ESCC Page 6 of 53

7 Note: It is recommended to configure CHAP secret based authentication for added security. You must configure the same Partner system CHAP secret when creating partnership on each Storwize system. Establishing Metro Mirror We established Metro Mirror (MM) of 8 * IBM i LUNs by using the following steps: 1. On remote V7000 we created LUNs for Metro Mirror targets. The Metro Mirror target volumes must be of the same type and capacity as the original volumes. 2. In Storwize GUI click Copy Services -> Remote Copy -> New Consistency Group 3. In the New Consistency Group window specify: the name of the Consistency group and click Next -> select that the auxiliary volumes are On another system - > select to Add relationships to the group -> select Metro Mirror type of copy -> click Next on message»no items found«-> specify master and auxiliary volumes for the relationships and add the relationships to the consistency group -> Select that the volumes are Not yet synchronized -> select to Start synchronize them -> click Close on Task completed message. The described sequence of actions in steps 2 and 3 is shown on the pictures below: Copyright IBM Corporation, 2014, ESCC Page 7 of 53

8 Copyright IBM Corporation, 2014, ESCC Page 8 of 53

9 Copyright IBM Corporation, 2014, ESCC Page 9 of 53

10 Copyright IBM Corporation, 2014, ESCC Page 10 of 53

11 Copyright IBM Corporation, 2014, ESCC Page 11 of 53

12 Status of Metro Mirror relationships at initial synchronization During initial synchronization the status of the LUNs is Inconsistent Copying, shown on the screenshot below: The progress of the synchronization can be tracked by expanding and displaying the Running Tasks in the Status Indicator in Storwize GUI, as shown on the picture below. Copyright IBM Corporation, 2014, ESCC Page 12 of 53

13 After synchronization is finished the status of Metro Mirror relationships is Consistent synchronized and can be seen on the following screenshot: Establishing Global Mirror with Change Volumes Global Mirror (GM) in Storwize V7000 can be setup in two ways: As Standard Global Mirror As Global Mirror with Change Volumes. For more information about the two Global Mirror modes refer to the Redbook Implementing the IBM Storwize V7000 V6.3, SG For the implementations with links over TCP/IP it is presently recommended to use Global Mirror with Change Volumes therefore we established this way of Global Mirror and tested it with IBM i. To establish Global Mirror with Change Volumes for the LUNs of Production IBM i we used the following actions: Create Global Mirror target volumes 1. On remote V7000 create the LUN for Global Mirror targets. The LUNs must be of the same type and capacity as the IBM i LUNs on local V7000. Create Global Mirror consistency group 2. In the GUI of local V7000 expand Copy Services -> Remote Copy -> click on the tab New Consistency Group 3. In the New Consistency Group window specify the name of the consistency group -> Specify that the auxiliary volumes are located on another system and select the V7000 for Auxiliary volumes -> select to add relationships to the group -> for Remote Copy type select Global Mirror with Change Volumes -> click Next on No items found message 4. Add the relations to the new consistency group by selecting each IBM i production LUN as Master and the corresponding LUN on remote V7000 as Copyright IBM Corporation, 2014, ESCC Page 13 of 53

14 Auxiliary -> click on Add -> select to add the master Global Mirror change volume -> select to create a new Master volume Add all the production LUNs and their remote pairs to the consistency group as described. 5. Select that the volumes are not yet synchronized -> select No, do not start copying -> Close on Task completed message This creates the Global Mirror consistency group with relations, the consistency group is not yet started, and the relations are in Inconsistent Stopped state. The master Change Volumes are now automatically created. The following screenshots show the actions of the steps 1. to 5. that are different than with described actions to establish Metro Mirror. On local V7000 create consistency group and add the relations: Copyright IBM Corporation, 2014, ESCC Page 14 of 53

15 Copyright IBM Corporation, 2014, ESCC Page 15 of 53

16 Created Master Change Volumes: Copyright IBM Corporation, 2014, ESCC Page 16 of 53

17 Adding Global Mirror Change Volumes on remote site 6. In the GUI of remote V7000 expand Copy Services -> Remote Copy -> expand the created consistency group -> right click on each relation -> select Global Mirror Change Volumes -> select Create New. This is shown on the following screenshot. Copyright IBM Corporation, 2014, ESCC Page 17 of 53

18 Start Global Mirror consistency group 7. On local V7000 start the created Global mirror consistency group by using GUI Remote Copy -> right click on the consistency group -> select Start, as shown on the following screenshot. 8. After initial synchronization is finished the relations stay in the status Consistent Copying, they don't change the status to Consistent Synchronized. Disaster Recovery tests with Metro Mirror of IBM i Full System Planned outage of IBM i In order to enable continuance of production workload on Disaster Recovery (DR) site during planned outages of IBM i, the Metro Mirror (MM) target volumes in Storwize on DR site is attached to a stand-by IBM i partition. At the planned outage we IPL the standby partition from MM target LUNs. This partition runs a clone of Production IBM i and the production workload continues in the LPAR during the outage. After the planned outage is finished we have the possibility to re-start Metro Mirror from Production to Disaster Recovery site or from DR site to Production site, depending on the type of outage and the outcome of the work during the outage. We describe both ways of fail-back: from Production to DR site, and from DR to Production site. Copyright IBM Corporation, 2014, ESCC Page 18 of 53

19 Note: a. The synchronization of Metro Mirror re-start is incremental. b. The changes done to the workload on the site which is the target of restarted Metro Mirror, will be lost. We performed the following steps for using Disaster Recovery site at planned outage of IBM i. Stop Metro Mirror 1. Power-down Production IBM i 2. In local V7000 stop consistency group by using V7000 GUI -> Copy Services -> Remote Copy -> right click on Consistency group in GUI and select Stop -> in Stop Remote-Copy Consistency group window check Allow secondary read/write access -> Close on Task completed message. The Metro Mirror relations are now in Idling status The performed actions and the new status of Metro Mirror relations, are shown on the screen-shots below: Copyright IBM Corporation, 2014, ESCC Page 19 of 53

20 3. Perform IPL of Disaster Recovery IBM i LPAR from Metro Mirror auxiliary LUNs. Setup a different IP address and different network attribute as Production system has. 4. Perform IPL of Production IBM i Re-start Metro Mirror from Production to DR site 1. Power-down Disaster Recovery IBM i 2. In local Storwize V7000 start Metro Mirror consistency group and specify Master is Primary. This can be seen on the screen-shots below: Copyright IBM Corporation, 2014, ESCC Page 20 of 53

21 During the re-synchronization the Metro Mirror relations are in Inconsistent Copying state. Note: the re-synchronization is incremental. After re-synchronization is finished the status of relationships is Consistent Synchronized in the direction from Master to Auxiliary Storwize: Copyright IBM Corporation, 2014, ESCC Page 21 of 53

22 Re-start Metro Mirror from DR site to Production site and switch Metro Mirror from Production to DR site 1. Power-down Production IBM i 2. In local Storwize start Metro Mirror consistency group and specify Auxiliary is Primary -> Close in Task completed message. This can be seen on the screenshots below: Wait that incremental re-synchronization finishes. During and after the resynchronization the direction of Metro Mirror is from Auxiliary values to Master volumes which is indicated by the icon on the screenshot below: at each relationship and can be seen Copyright IBM Corporation, 2014, ESCC Page 22 of 53

23 3. Power-down Disaster Recovery IBM i 4. In local Storwize perform Switch of the Metro Mirror consistency group by rightclick on the Consistency group -> select Switch -> confirm that you want to make the local V the primary system for our consistency group, by clicking Yes on this question -> click Close on Complete. This is shown in the following screenshots: Copyright IBM Corporation, 2014, ESCC Page 23 of 53

24 After the changes done on the Auxiliary Storwize are copied back, the status of the LUNs is Consistent Synchronized without the icon indicating switched relations. 5. Perform IPL of Production IBM i Unplanned outage of IBM i We simulate the outage in Production IBM i by disconnecting the V7000 LUNs for this IBM i LPAR. After disconnecting the LUNs Production IBM i enters DASD attention state with SRC code A Note: In this test we used the function Switch of Metro Mirror consistency group, which enables the usage of target Metro Mirror LUN and reverses the Metro Mirror direction. Depending of the unplanned outage you may want to Stop the MM consistency group, enable target volumes for reads and writes, and re-start Metro Mirror in the direction Auxiliary -> Master at a later time. We described this way of switch-over in the section Planned outage of IBM i. Switch to Disaster Recovery site 1. In the GUI of Local V7000 click on Copy Services -> Remote Copy -> rightclick on the consistency group of IBM i volumes -> select Switch -> confirm that you want to make the auxiliary V the primary cluster for our consistency group, by clicking OK on this question -> click Close on Complete message. The screenshots below show these actions: Copyright IBM Corporation, 2014, ESCC Page 24 of 53

25 After the Metro Mirror switch, the status of the LUNs in Metro Mirror relationship is marked with the icon indicating that the roles of the volumes are switched and hence the direction of Metro Mirror is switched. 2. Perform IPL of Disaster Recovery partition from auxiliary Metro Mirror LUNs. Setup a different IP address and different network attribute as Production system has. Continue the production workload in the DR partition. 3. Power-down Production IBM i if possible. Switch back to Production site 1. In our test we simulated the disaster by disconnecting the LUNs of Production LPAR; now we re-connect them to Production system. 2. Power-down Disaster recovery IBM i partition. 3. In local Storwize perform Switch of the Metro Mirror consistency group by rightclick on the Consistency group -> select Switch -> confirm that you want to make the local V the primary system for our consistency group, by clicking Yes on this question -> click Close on Complete. This is shown in the Copyright IBM Corporation, 2014, ESCC Page 25 of 53

26 screenshots in the section Re-start Metro Mirror from DR site to Production site and switch Metro Mirror from Production to DR site. After the changes done on the Auxiliary Storwize are copied back, the status of the LUNs is Consistent Synchronized without the icon indicated switched Metro Mirror direction. 4. Perform IPL of Production IBM i, and continue the workload on Production site. Outage of IP links The environment of this test is as follows: Production IBM i is running Metro Mirror from local to DR site is running The Metro Mirror relationships are in Consistent Synchronized state We simulate the outage of IP links by plugging-out the Ethernet cables from one of the Storwize systems. After this action the status of Metro Mirror relations on local V7000 is changed to Idling Disconnected and the direction of Metro Mirror is still from master to Auxiliary Storwize, as can be seen on the following screen-shot: The Metro Mirror relations on remote Storwize are in Consistent Disconnected state, indicated Metro Mirror direction is from Master to Auxiliary volumes, and Freeze Time indicates the time of failure of the links. This is shown on the screenshot below. Copyright IBM Corporation, 2014, ESCC Page 26 of 53

27 The Health Status on each V7000 is Red with alert of Remote Partnership status and can be seen on the screen-shot below: In the GUI screen Partnerships of each V7000, the state of the partner Storwize is Not Present: In the Event list in the GUI of each local and remote Storwize are two alerts for the loss of Remote Copy partner and can be seen on the screenshot below: Copyright IBM Corporation, 2014, ESCC Page 27 of 53

28 Production IBM i is running all the time; there aren't any interruptions and there aren't any messages in the QSYSOPER queue to indicate the loss of Metro Mirror links. We fix the outage of IP links by plugging the Ethernet cables in the Storwize again. After this action the status of the Metro Mirror relations in each V7000 is Consistent Stopped with indicated Freeze time, as shown on the screen-shots below: Copyright IBM Corporation, 2014, ESCC Page 28 of 53

29 In GUI of local V7000 we start Metro Mirror by right-clicking on Metro Mirror consistency group and selecting Start, confirming Start Consistency Group and Close on the Task message. This is shown on the following screen-shots: After incremental synchronization the Metro Mirror relations on each V7000 are in Consistent Synchronized state. Disaster recovery tests with Global Mirror with Change Volumes of IBM i Full System In this section we describe the Disaster Recovery tests with Global Mirror with Change Volumes of IBM i Full System. Copyright IBM Corporation, 2014, ESCC Page 29 of 53

30 Planned outage of IBM i We perform the needed actions for planned outage of IBM i the same way as we did with Metro Mirror: during the planned outage the production workload continues to run in the Stand-by IBM i partition to which the Global Mirror target LUNs are attached. After the outage is finished we have the possibility to restart Global Mirror in the direction Production -> Disaster Recovery or in the direction Disaster Recovery -> Production We performed the following steps for using Disaster Recovery site at planned outage of IBM i. Stop Global Mirror 1. Power-down Production IBM i 2. In local V7000 stop Global Mirror consistency group by using V7000 GUI Copy Services -> Remote Copy -> right click on Consistency group in GUI and select Stop -> in Stop Remote-Copy Consistency group window check Allow secondary read/write access -> Close on Task completed message. The Metro Mirror relations are now in Idling status. Stopping Global Mirrror consistency group is shown on the following screenshot: Copyright IBM Corporation, 2014, ESCC Page 30 of 53

31 3. Perform IPL of Disaster Recovery IBM i LPAR from Global Mirror auxiliary LUNs, Setup a different IP address and different network attribute as Production IBM i uses. 4. Perform IPL of Production IBM i Note: On remote site we connect to IBM i Global Mirror auxiliary LUNs, not the Change Volumes. Re-start Global Mirror from Production to DR site In this case the changes that were done to Production IBM i will be propagated to DR site and the changes that were done in Disaster Recovery IBM i will be lost. 1. Power-down Disaster Recovery IBM i 2. In local Storwize V7000 start Global Mirror consistency group and specify Master is Primary. This action is the same as with Metro Mirror, the relevant screen-shots for Metro Mirror can be seen in the section Re-start Metro Mirror from Production to DR site. After the re-synchronization the relations are in Consistent Copying status with the direction from Master to Auxiliary Storwize. Note: the re-synchronization is incremental, you can follow the progress by expanding and displaying Running Tasks in the Status Indicators in V7000 GUI. Copyright IBM Corporation, 2014, ESCC Page 31 of 53

32 Re-start Global Mirror from DR site to Production site and switch Global Mirror from Production to DR site In this case the changes that were done to Disaster Recovery IBM i will be propagated to Production partition and the changes that were done in Production IBM i will be lost. 1. Power-down Production IBM i 2. In local Storwize start Global Mirror consistency group and specify Auxiliary is Primary -> Close in Task completed message. Wait that incremental re-synchronization finishes. During and after the resynchronization the direction of Global Mirror is from Auxiliary volumes to Master Storwize which is indicated by the icon at each relationship. 3. Power-down Disaster Recovery IBM i 4. In local Storwize stop the Global Mirror consistency group by right-click on the Consistency group -> select Stop -> in Stop Remote-Copy Consistency group window check Allow secondary read/write access -> Close on Task completed message. 5. In local Storwize start the GM consistency group and specify Master is Primary - > Close in Task completed message. After the changes done on the Auxiliary Storwize are copied back, the status of the LUNs is Consistent Copying without the icon indicating switched relations. 6. Perform IPL of Production IBM i Unplanned outage of IBM i We simulate the outage in Production IBM i by disabling switch zones for connecting this IBM i LPAR. After removing zones Production IBM i enters DASD attention state with SRC code A Switch to Disaster Recovery site 1. In local V7000 stop Global Mirror consistency group by using GUI Copy Services -> Remote Copy -> right click on Consistency group for Global Mirror and select Stop -> in Stop Remote-Copy Consistency group window check Allow secondary read/write access and click Stop Consistency Group -> Close on Task completed message. 2. In local V7000 start Global Mirror consistency group and specify Auxiliary is Primary and Start Consistency group -> Close in Task completed message. After these two actions the direction of Global mirror is reversed which is denoted by the icon at Global Mirror relations, as shown in the following screenshot: Copyright IBM Corporation, 2014, ESCC Page 32 of 53

33 Note: Because the status of relations is Consistent Copying the GUI action Switch fails with the indication that the relations are not in correct status for switch. Therefore we to stop Global Mirror consistency group and start it with Global Mirror in the opposite direction. 3. Perform IPL of Disaster Recovery partition from auxiliary Metro Mirror LUNs. Setup a different IP address and different network attribute as Production system has. Continue the production workload in the DR partition. 4. Power-down Production IBM i if possible. Switch back to Production site 1. In our test we simulated the disaster by disabling zones of Production LPAR in the switches; now we enable zones to re-connect the Production system. 2. Power-down Disaster Recovery IBM i partition. 3. In local Storwize stop the Global Mirror consistency group and check Allow secondary reads/writes, as is described in Switch to Disaster Recover site, step In local Storwize start Global Mirror consistency group, and check Master is Primary. 5. After this action and incremental synchronization, the Global Mirror is running in the direction Production -> Disaster Recovery, the Global Mirror status is Consistent Copying. 6. Perform IPL of Production IBM i. Copyright IBM Corporation, 2014, ESCC Page 33 of 53

34 Outage of IP links The environment of this test is as follows: Production IBM i is running Global Mirror with Change Volumes from local to DR site is running The Global Mirror relationships are in Consistent Copying state We simulate the outage of IP links by plugging-out the Ethernet cables from one of the Storwize systems. After this action the status of Global Mirror relations on local V7000 is changed to Idling Disconnected, the direction of Global Mirror is from Master to Auxiliary Storwize, the status alert indicates that there is a problem on remote partnership. This can be seen on the following screenshot: The Event log in local Storwize lists the alert with the description Remote Copy lost synchronization and can be seen on the screen-shot below: Copyright IBM Corporation, 2014, ESCC Page 34 of 53

35 The Global Mirror relations on remote Storwize are in Consistent Disconnected state, the Global Mirror direction is from Master to Auxiliary volumes, Freeze Time indicates the time of failure of the links and status alert indicates the problem on remote partnership. This is shown on the screen-shot below. The partnerships on either local or remote V7000 indicate that the remote partner is not present. The screen-shot from local V7000 shows this: Production IBM i is running all the time, there aren't any interruptions and there aren't any messages in the QSYSOPR queue to indicate the loss of Global Mirror communication. We fix the outage of IP links by plugging the Ethernet cables in the Storwize again. After this action the Global Mirror automatically re-starts in original direction. The status of the Global Mirror relations in each V7000 is Consistent Copying, as shown on the screenshots from local V7000 below: Copyright IBM Corporation, 2014, ESCC Page 35 of 53

36 Disaster Recovery tests with Metro Mirror and PowerHA for i For these tests we setup the IBM i environment for PowerHA for I with Storwize Metro mirror. In this setup both partition Production and Disaster Recovery are running. The IASP is implemented on Production LPAR; the LUNs of the IASP are replicated from Master to Auxiliary V7000 by Metro Mirror. We used the following steps to setup PowerHA environment: 1. We created the cluster with Production LPAR and Disaster Recovery LPAR. 2. Create the device domain with both nodes 3. On Production system create an Independent Auxiliary Storage Pool (IASP). Note: In NPIV connection the IASP must be connected with different virtual FC adapters than Sysbas (ASP1 and ASPs 2 32), and the virtual FC adapters for IASP and sysbas must be mapped to different ports in adapters in VIOS. 4. On Production system generate SSH Keys to access V7000, distribute the private key to the DR partition and the public key to both Storwize systems. 5. Create the volumes for the Metro Mirror target of the IASP, connect them to DR partition and initialize them. On DR system create the description of the IASP 6. Start Metro Mirror of the IASP. 7. Create and start Cluster Resource Group (CRG) 8. Create PowerHA Copy description of both nodes 9. Start PowerHA session for Metro Mirror. The following screen-shots show creating and starting of Metro Mirror consistency group for the IASP: Copyright IBM Corporation, 2014, ESCC Page 36 of 53

37 Copyright IBM Corporation, 2014, ESCC Page 37 of 53

38 Copyright IBM Corporation, 2014, ESCC Page 38 of 53

39 Copyright IBM Corporation, 2014, ESCC Page 39 of 53

40 The following screenshots show the started PowerHA session for V7000 Metro Mirror: Copyright IBM Corporation, 2014, ESCC Page 40 of 53

41 Planned outage of IBM i Switch to Disaster Recovery site To switch to Disaster Recovery site we use command CHGCRGPRI (Change Cluster Resource Group Primary) with specified cluster name and CRG name on the Disaster Recovery partition. This command performs the following steps in our PowerHA setup: 1. It varies off the IASP on the Production site 2. It promotes the first backup node in the cluster to primary node and moves the old primary node to the last backup position. 3. It reverses the Metro Mirror direction, providing that the PowerHA session is configured with the option Switchover reverse replication = YES, which is the default value. 4. It varies on the IASP on the backup site, providing that the CRG is created with the option Configuration object online = *ONLINE After the switch is completed we can continue production workload in the Disaster Recovery LPAR. The screenshots below are taken during the execution of command CHGCRGPRI CLU(PWRHA_CLU) CRG(PWRHA_CRG). They show the status of production IASP which is varying-off, reversed Metro Mirror direction, the status of IASP on DR partition varying-on and the message about successful completion of the command. IASP on Production site varying-off: Copyright IBM Corporation, 2014, ESCC Page 41 of 53

42 Reversed direction of Metro Mirror: Varying-on of IASP on Disaster Recovery site: The command CHGCRGRI completed: Copyright IBM Corporation, 2014, ESCC Page 42 of 53

43 The status of PowerHA session for Metro Mirror after the switch to DR site is shown below. The partition DISREC on DR site SITE2 has now the source role of the DR setup, and the IASP in this partition is available. The partition PROD on Production site SITE1 has now the target role, the IASP in Production partition is not available. Copyright IBM Corporation, 2014, ESCC Page 43 of 53

44 Switch-back to Production site To switch back to Production site after the outage is finished we use command CHGCRGPRI on Production IBM i LPAR. This command performs the following steps in our PowerHA setup: 1. It varies-off the IASP on the DR site 2. It moves the current primary node (the DR node) to the backup position and moves the current backup node (on Production site) to the primary position. 3. It reverses the Metro Mirror direction, so that the direction becomes Production -> Disaster Recovery again. 4. It varies on the IASP on the Production site. The following screen-shots show varying-off of the IASP on DR site, Metro Mirror again in direction Production -> Disaster Recovery and vary-on of the Production IASP. Copyright IBM Corporation, 2014, ESCC Page 44 of 53

45 Unplanned outage of IBM i To simulate the failure of IBM i on Production site we end all subsystems in the Production partition. Failover to Disaster Recovery site with reverse replication PowerHA reacts automatically to the events that would cause cluster failover; ending all subsystems on production node is such an event. In our case PowerHA therefore automatically performs the following steps: 1. If possible it varies off the IASP on Production site 2. It promotes the first backup node in the cluster to primary node and moves the old primary node to the last backup position. 3. If the PowerHA session is configured with the option Failover reverse replication = YES PowerHA reverses the Metro Mirror direction. If the option Failover reverse replication is set to NO it does not reverse the metro mirror, 4. It varies on the IASP on the backup site, providing that the CRG is created with the option Configuration object online = *ONLINE In the setup for this test we specified Failover reverse replication -> *YES therefore PowerHA automatically reverses the Metro Mirror direction at the failover. Copyright IBM Corporation, 2014, ESCC Page 45 of 53

46 If the Cluster Resource Group (CRG) was created so that a message queue name was specified at the option Failover message queue, and the corresponding values were specified at options Failover wait time and Failover default action, then PowerHA sends a message to the specified message queue before proceeding with failover. Depending on the answer to the message and the specified default action PowerHA performs the steps to failover, or not. In our setup we specified the Failover message queue -> QSYSOPR, Failover wait time - > 1, and Failover default action -> *PROCEED. Therefore PowerHA first sends the message shown on the screen-shot below to the QSYSOPR message queue. After we answered G (Continue failover) to the message, PowerHA proceeds with failover to DR site. Following screen-shots show the performed steps by PowerHA: reversed direction of Metro Mirror and varying-on of IASP on Disaster Recovery site: Copyright IBM Corporation, 2014, ESCC Page 46 of 53

47 After failover is performed, displaying the PowerHA session shows the primary role of Disaster Recovery node, and the backup role of Production node and can be seen on the screen-shot below: Switch back to Production site After the outage on Production site is fixed we use command CHGCRGPRI on Production node to switch back to Production site. After switch back is performed the Production site has the source role in the solution, DR site has the target role. This can be seen on the screen-shot below: Copyright IBM Corporation, 2014, ESCC Page 47 of 53

48 Failover to Disaster Recovery site without reverse replication PowerHA reacts automatically to ending all subsystems and performs the following steps 1. If possible it varies off the IASP on Production site 2. It promotes the first backup node in the cluster to primary node and moves the old primary node to the last backup position. 3. The PowerHA session is configured with Failover reverse replication = NO, so PowerHA does not reverse the Metro Mirror direction. 4. It varies on the IASP on the backup site, providing that the CRG is created with the option Configuration object online = *ONLINE In our setup we specified the Failover message queue -> QSYSOPR, Failover wait time - > 1, and Failover default action -> *PROCEED. Therefore PowerHA first sends the message shown on the screen-shot below to the QSYSOPR message queue. After we answer G to proceed with failing-over to the DR site PowerHA stops Metro Mirror. Metro Mirror relations on Master V7000 and on Auxiliary V7000 are in Idling status, as is shown below: Copyright IBM Corporation, 2014, ESCC Page 48 of 53

49 The IASP on DR site is varied-on; see the screen-shot below: The DR site is now ready for taking-over the production workload in the IASP. The PowerHA session shows that the target site is now primary, and the status of Metro Mirror is Detached: We manually reverse Metro Mirror direction from Disaster Recovery site to Production site by using the GUI on Master V7000 -> Remote Copy -> right click on Metro Mirror consistency group -> Start -> specify Auxiliary is primary start click Start Consistency group. This is shown on the following screenshots: Copyright IBM Corporation, 2014, ESCC Page 49 of 53

50 Note: This action can be done only on primary V7000 (Production V7000) it can not be performed on Auxiliary V7000. The status of Metro Mirror reported by PowerHA session is now Active: Copyright IBM Corporation, 2014, ESCC Page 50 of 53

51 Switch back to Production site After the outage on Production site is fixed we use command CHGCRGPRI on Production node to switch back to Production site. After switch back is performed the Production site has the source role of the PowerHA solution and DR site has the target role. Outage of IP links During PowerHA session for Metro Mirror is running, we simulate the outage of IP links by of IP links by plugging-out the Ethernet cables from one of the Storwize systems. Before the outage PowerHA session reports Metro Mirror active as can be seen on the screenshot below, and the V7000 GUI shows Metro Mirror in Consistent Synchronized state. Copyright IBM Corporation, 2014, ESCC Page 51 of 53

52 After the failure of IP links the Metro Mirror relations on Primary V7000 are in Idling Disconnected state. See the screenshot below: The secondary V7000 reports Metro mirror relations in Consistent Disconnected state: PowerHA reports Suspended Metro Mirror status, and can be seen on the following screen-shot: The status of the partnership is Not Present, as shown on the screenshot below: Copyright IBM Corporation, 2014, ESCC Page 52 of 53

53 We fix the simulated failure by plugging-in the Ethernet cables again. After the failure is fixed Metro Mirror is automatically resumed by PowerHA for i: PowerHA health check mechanism is taking corrective action and restarts Metro Mirror. The V7000 GUI shows Metro mirror in Consistent Synchronized status and can be seen on the following screenshot: PowerHA session reports Active status of Metro Mirror:. Note: there is a difference between link outage with Metro Mirror of Full System and Metro Mirror with PowerHA. With Full System we manually re-start Metro Mirror after the IP links are back working, with PowerHA for i Metro Mirror restarts automatically Copyright IBM Corporation, 2014, ESCC Page 53 of 53