HIGH AVAILABILITY SETUP USING VERITAS CLUSTER SERVER AND NETAPP SYNCHRONOUS SNAPMIRROR Jorge Costa, NetApp June 2008
TABLE OF CONTENTS 1 PROBLEM DESCRIPTION 4 2 SOLUTION OVERVIEW 5 21 STORAGE 5 22 VCS SERVICE GROUPS6 23 VCS MAINCF 7 3 HOW DOES IT WORK? 8 31 ON NODE EBSIPS2: 8 32 ON NODE EBSIPS1: 9 33 SG_TEST_PROPERTIESFILE:9 34 SCRIPTS SUMMARY 10 35 VCS SSH FILER CREDENTIALS 10 4 INSTALATION GUIDE11 41 TASK 1: CREATE ACCOUNTS ON BOTH FILERS 11 42 TASK 2: SET SSH-PUBLIC-KEY AUTHENTICATION 12 43 TASK 3: INSTALL FCP HUK ON BOTH NODES 12 44 TASK 4: EDIT SNAPMIRRORCONF ON BOTH FILERS13 45 TASK 5: SNAPMIRROR INITIALIZE FROM CDC-FAS-PO01 13 46 TASK 6: INSTALL SCRIPTS14 47 TASK 7: EDIT VCS PREONLINE TRIGGER ON NODE EBSIPS1 14 48 TASK 8: EDIT VCS PREONLINE TRIGGER ON NODE EBSIPS2 15 49 TASK 9: EDIT VCS POSTOFFLINE TRIGGER ON NODE EBSIPS215 410 TASK 10: EDIT PROPERTIESFILE ON BOTH NODES16 411 TASK 11: EDIT SG_TEST_PRE-ONLINE_FAILOVER_TO_EBSIPS2SH ON EBSIPS2 17 412 TASK 12: EDIT SG_TEST_POST-OFFLINE_FAILBACK_TO_EBSIPS1SH ON NODE EBSIPS2 18 413 TASK 13: EDIT PRE-ONLINE_FAILBACK_TO_EBSIPS1SH ON EBSIPS1 19 414 TASK 14: EDIT SG_TEST_RESYNC_FROM_EBSIPS2_TO_EBSIPS1SH ON NODE EBSIPS2 20 5 OPERATION MANUAL22 6 LOGS AND TROUBLESHOOTING23 7 APPENDIX 124 71 PROPERTIESFILE24 72 EBSIPS1 - PREONLINE 25 73 EBSIPS1 PRE-ONLINE_FAILBACK_TO_EBSIPS1SH 26 74 EBSIPS2 RESYNC_FROM_EBSIPS2_TO_EBSIPS1SH 30 Page 2
75 EBSIPS2 POSTOFFLINE 31 76 EBSIPS2 PREONLINE 32 77 EBSIPS2 POST-OFFLINE_FAILBACK_TO_EBSIPS1SH 34 78 EBSIPS2 PRE-ONLINE_FAILOVER_TO_EBSIPS2SH35 Page 3
1 PROBLEM DESCRIPTION ACMECORPORATION has acquired a number of NetApp lers and is looking to replace some of the current storage XP infrastructure with the NetApp solution ACME has however some interesting requirements, these required NetApp Professional Services and SDT-EMEA to become involved in order to provide a solution for these requirements ACMECORPORATION is primarily a Solaris FC shop where the NetApp lers are their Tier 2 storage platform, Filesystem of choice is VxFS on top of VxVM 5,0 and using DMP as their multipath provider In terms of host and application resilience ACME uses Symantec Veritas Cluster Server 50 This document has references to two sites production (DDC) and DR (CDC); however data can be replicated in any direction and must be made easily available after a disaster or a controlled failover The requirements of replicating data in any direction and promoting one DR site online are provided by SnapMirror however ACME wanted to go a step further using VCS to control failover/failback and automate as much as possible One of their primary concerns was to have a fully synchronous replication across their production and DR site, NetApp SnapMirror Sync addresses this rst concern Especially now with the release of ONTAP 73 which brings substantial performance improvements in both SnapMirror Sync and Fibre-Channel Their other concern which required the help of NetApp PS was to integrate SnapMirror sync failover/failback with Veritas Cluster Server enabling ACME to have one-button-click capabilities to perform controlled site failovers, storage failover and node failover This paper documents the NetApp SnapMirror integration with Veritas Cluster Server Page 4
2 SOLUTION OVERVIEW 21 STORAGE The diagram above shows two sites where ACME operates: DDC and CDC (DR) Each one of these sites contains a Solaris node member of a two node VCS cluster Each node has access only to its local SAN storage The LUNs made available to one host are replicated across the WAN using SnapMirror Sync to the other storage array While this is not a real VxVM shared disk group across two nodes, SnapMirror Sync propagates every write done at the Production site to DR, so that from a VxVM perspective the DR node ebsips2 LUNs appear to be the exactly the same disks presented on the PRD node ebsips1, and part of a VxVM cluster disk group The replicated LUNs on the DR site are in an offline status and unavailable to the second host during normal operation The SnapMirror replication has to be stopped in order to online the DR LUNs and provide write access to node ebsips2 Page 5
22 VCS SERVICE GROUPS Each application congured in the cluster requires at least one VCS Service Group with one dedicated NetApp FlexVol and one dedicated LUN This Service Group is available on a single node at a single moment in time It cannot be online on two nodes simultaneously For each FlexVol congured in a VCS Service Group there is a SnapMirror relationship to another FlexVol on the DR storage array The diagram above shows how VCS is integrated with the NetApp Scripts Page 6
23 VCS MAINCF In the ACMECORPORATION setup there is a VCS SG named sg_test containing two resources: VxVM DiskGroup resource dgapp_ebsip1 and vxfs mountpoint resource mt_opt_vrtsnbu The LUNs which form DiskGroup dgapp_ebsip1 are the LUNS presented by the NetApp ler The VCS maincf is listed below: include "typescf" cluster ebsipc ( UserNames = admin = hkkmkfkhkekijhiqhj, unix_adm = allnlglilfljkijrik Administrators = admin, unix_adm ) system ebsips1 ( ) system ebsips2 ( ) group sg_test ( SystemList = ebsips1 = 0, ebsips2 = 1 AutoStart = 0 AutoFailOver = 0 AutoStartList = ebsips1, ebsips2 PreOnline = 1 OnlineRetryLimit = 3 OnlineRetryInterval = 120 ) DiskGroup dgapp_ebsp1 ( Critical = 1 DiskGroup = dgapp_ebsp1 ) Mount mt_opt_vrtsnbu ( Critical = 1 MountPoint = "/opt/vrtsnbu" BlockDevice = "/dev/vx/dsk/dgapp_ebsp1/v_opt_vrtsnbu" FSType = vxfs FsckOpt = "-y" ) mt_opt_vrtsnbu requires dgapp_ebsp1 // resource dependency tree // // group sg_test // // Mount mt_opt_vrtsnbu // // DiskGroup dgapp_ebsp1 // // Page 7
3 HOW DOES IT WORK? The integration with Veritas Cluster Server is done through a set of customized scripts(see 22) which are run by VCS while online ing or offline ing a Service Group These scripts are called by the VCS triggers preonline and postoffline during normal operation Their function is to control LUN access and invert SnapMirror replication flow 31 ON NODE EBSIPS2: Trigger preonline executes: sg_test_pre-online_failover_to_ebsips2sh Trigger postoffline executes: sg_test_post_offline_failback_to_ebsips1sh These scripts are executed by VCS when we modify some of the standard VCS triggers and enable the Service Group property PreOnline hagrp -modif sg_test PreOnline 1 The change above forces the preonline trigger to be run every time the Service Group sg_test is brought online The standard VCS preonline trigger script is modied to execute our sg_test_pre- ONLlINE_failover_to_ebsips2sh every time the service group sg_test is brought online This is done by modifying the trigger /opt/vrtsvcs/bin/triggers/preonline as listed below: put your code here if ( $ARGV[0] eq "ebsips2" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_pre-online_failover_to_ebsips2sh"); The Script POST_OFFLINE_failback_to_ebsips1sh is executed by modifying the standard VCS post-offline trigger scripts adding the following code: put your code here if ( $ARGV[0] eq "ebsips2" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_post-offline_failback_to_ebsips1sh"); The examples above are only relevant to node ebsips2 Page 8
32 ON NODE EBSIPS1: Trigger PRE-ONLINE executes: sg_test_pre-online_failback_to_ebsips1sh followed by sg_test_resync_from_ebsips2_to_ebsips1sh The script sg_test_pre-online_failback_to_ebsips1sh is run by VCS when the ServiceGroup sg_test is brought online on node ebsips1 The only required actions are to have the PreOnline attribute set to 1 and to modify the standard PreOnline trigger to call our sg_test_pre-online_failback_to_ebsips1sh That change done on /opt/vrtsvcs/bin/triggers/preonline is shown below: put your code here if ( $ARGV[0] eq "ebsips1" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_pre- ONLINE_failback_to_ebsips1sh"); 33 SG_TEST_PROPERTIESFILE: The sg_test_propertiesle le located in /home/netapp/scripts contains variables used by all other scripts These variables include the nodenames, ler hostnames, production and DR volumes, LUN paths and IGROUP names All these scripts mentioned in section 3 must be customized during install or reconguration of VCS ServiceGroups Page 9
34 SCRIPTS SUMMARY The table below contains a short summary of all the scripts and their locations on each node: node script run by ebsips2 /home/netapp/scripts/sg_test_pre-online_failover_to_ebsips2sh VCS PreOnline trigger ebsips2 /home/netapp/scripts/sg_test_post_offline_failback_to_ebsips1sh VCS PostOffline trigger ebsips1 /home/netapp/scripts/sg_test_pre-online_failback_to_ebsips1sh VCS PreOnline trigger ebsips2 /home/netapp/scripts/sg_test_resync_from_ebsips2_to_ebsips1sh VCS PreOnline trigger Both nodes /home/netapp/scripts/sg_test_propertiesle Sourced by every script Both /opt/vrtsvcs/bin/triggers/preonline VCS nodes Both nodes /opt/vrtsvcs/bin/triggers/postoffline VCS 35 VCS SSH FILER CREDENTIALS In order to execute ler commands from nodes ebsips1 and ebsips2, two users vcs_ebsips1 and vcs_ebsips2 have been dened on lers ddc-fas-p03 and cdc-fas-p01 These users enable remote execution of commands using ssh public-key autication While these users have limited access as they are part of a restricted role, their level of access allows the root user from these two hosts to perform a large set of operations such as creating, destroying, offline ing, online ing luns and volumes on each ler This may be of a security concern for ACMEClearnet, however ONTAP does not provide a mechanism to limit actions from a specic user to a set or volumes or LUNs Currently ONTAP allows us only to limit user actions to a set of commands Page 10
4 INSTALATION GUIDE This section contains the required steps to deploy or customize this solution on a new pair of nodes, or to add additional ServiceGroups to an existing VCS cluster There are several scripts that must be modied during the initial setup, those changes are related to FlexVols, LUNs and ServiceGroups names and are documented in the following pages The scripts as they ve been deployed on ebsips1 and ebsips2 are prepared to work with a single FlexVol, two LUNs and the ServiceGroup sg_test In order to use different LUNS, or just a different ServiceGroup name these scripts must be modied 41 TASK 1: CREATE ACCOUNTS ON BOTH FILERS Login into both NetApp Controllers and create the following objects: Create a role for VCS, named r_vcs with the following privileges: Name: r_vcs Info: VCS role Allowed Capabilities: login-ssh,cli-lun,cli-vol,cli-snapmirror Create a group for VCS, named g_vcs containing role r_vcs: Name: g_vcs Info: Rid: 131078 Roles: r_vcs Create users vcs_ebsips1, vcs_ebsips2 on both controllers and add these users to group g_vcs: Name: vcs_ebsips1 Info: VCS user ebsips1 Rid: 131079 Groups: g_vcs Name: vcs_ebsips2 Info: VCS user ebsips2 Rid: 131080 Groups: g_vcs Page 11
42 TASK 2: SET SSH-PUBLIC-KEY AUTHENTICATION From host ebsips1 set the ssh public-keys to user vcs_ebsips1 on both lers; From host ebsips2 set the ssh public-keys to user vcs_ebsips2 on both lers; The procedure for setting ssh public-key autication is available on the ONTAP 72 System Administration Guide -> Chapter 9: Using secureadmin -> Managing SSH for secureadmin -> Setting up public-key based autication Refer to the following link and page 191 for more information: http://nownetappcom/now/knowledge/docs/ontap/rel724_vs/pdfs/ontap/sysadminpdf From each host try to connect to the lers using ssh: [root@ebsips1] ssh vcs_ebsips1@ddc-fas-p03 snapmirror status [root@ebsips1] ssh vcs_ebsips1@cdc-fas-p01 snapmirror status [root@ebsips2] ssh vcs_ebsips2@ddc-fas-p03 snapmirror status [root@ebsips2] ssh vcs_ebsips2@cdc-fas-p01 snapmirror status 43 TASK 3: INSTALL FCP HUK ON BOTH NODES Login to the NOW website and download the FCP HUK for Solaris: http://nownetappcom/now/download/software/sanhost_sol/42/ Install the software using pkgadd on both nodes: [root@ebsips1] pkgadd d NTAPSANToolpkg [root@ebsips2] pkgadd d NTAPSANToolpkg Complete instructions are available on the FCP HUK installation guide: http://nownetappcom/now/knowledge/docs/hba/sun/relhu42/pdfs/vinstallpdf Page 12
44 TASK 4: EDIT SNAPMIRRORCONF ON BOTH FILERS Both lers should have an entry in the /etc/snapmirrorconf for every volume to be replicated across sites The example below from CDC-FAS-P01 shows volume vl_ebsips1_2 on ler DDC-FAS-P03 is replicating to a volume with the same name on ler CDC-FAS-P01 CDC-FAS-P01> rdle /etc/snapmirrorconf DDC-FAS-P03:vl_EBSIPS1_2 CDC-FAS-P01:vl_EBSIPS1_2 - sync As we need to be able to reverse replication when the ServiceGroup is active on ebsips2, an equivalent entry is required on ler DDC-FAS-P03 This enables us to replicate volume vl_ebsips1_2 from CDC-FAS-P01 to DDC-FAS-P03 ddc-fas-p03> rdle /etc/snapmirrorconf CDC-FAS-P01:vl_EBSIPS1_2 DDC-FAS-P03:vl_EBSIPS1_2 - sync If additional FlexVols are to be congured on the VCS ServiceGroup, additional entries for those FlexVols must be added to the /etc/snapmirrorconf les 45 TASK 5: SNAPMIRROR INITIALIZE FROM CDC-FAS-PO01 Connect to ler CDC-FAS-P01 and initialize SnapMirror between DDC-FAS-P03 -> CDC-FAS-P01 CDC-FAS-P01> Snapmirror initialize S DDC-FAS-P03:vl_EBSIPS1_2 w CDC-FAS- P01:vl_EBSIPS1_2 Repeat this task for every FlexVol to be SnapMirrored across sites Page 13
46 TASK 6: INSTALL SCRIPTS Create the directories /home/netapp/scripts and /home/netapp/logs on both nodes Then copy the scripts listed in Appendix 1 to the correct directories The table shows where each script should be deployed node script run by ebsips2 /home/netapp/scripts/sg_test_pre-online_failover_to_ebsips2sh VCS PreOnline trigger ebsips2 /home/netapp/scripts/sg_test_post_offline_failback_to_ebsips1sh VCS PostOffline trigger ebsips1 /home/netapp/scripts/sg_test_pre-online_failback_to_ebsips1sh VCS PreOnline trigger ebsips2 /home/netapp/scripts/sg_test_resync_from_ebsips2_to_ebsips1sh VCS PreOnline trigger Both nodes /home/netapp/scripts/sg_test_propertiesle Sourced by every script Both /opt/vrtsvcs/bin/triggers/preonline VCS nodes Both nodes /opt/vrtsvcs/bin/triggers/postoffline VCS 47 TASK 7: EDIT VCS PREONLINE TRIGGER ON NODE EBSIPS1 Each ServiceGroup running on hosts ebsips1, ebsips2 requires a set of PRE-ONLINE, POST-OFFLINE and propertiesle scripts The VCS preonline trigger contains an entry in that it executes a script if a specic ServiceGroup is bring brought online It is possible to manage several ServiceGroups by creating a new copy of the PRE- ONLINE_failback_to_ebsips1sh script with a prex for that VCS ServiceGroup and adding an entry to the VCS preonline trigger Example of different ServiceGroups: SG_test_PRE-ONLINE_failback_to_ebsips1sh SG_billing_PRE-ONLINE_failback_to_ebsips1sh SG_apps_PRE-ONLINE_failback_to_ebsips1sh Page 14
The required changes on the VCS /opt/vrtsvcs/bin/triggers/preonline trigger: put your code here if ( $ARGV[0] eq "ebsips1" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_pre-online_failback_to_ebsips1sh"); 48 TASK 8: EDIT VCS PREONLINE TRIGGER ON NODE EBSIPS2 Customize this script in the same way as explained for task 7 /opt/vrtsvcs/bin/triggers/preonline: put your code here if ( $ARGV[0] eq "ebsips2" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_pre-online_failback_to_ebsips1sh"); 49 TASK 9: EDIT VCS POSTOFFLINE TRIGGER ON NODE EBSIPS2 Customize this script in the same way as explained for task 7 /opt/vrtsvcs/bin/triggers/postoffline: put your code here if ( $ARGV[0] eq "ebsips2" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_post-offline_failback_to_ebsips1sh"); Page 15
410 TASK 10: EDIT PROPERTIESFILE ON BOTH NODES When running multiple ServiceGroups, each SG must have a dedicated propertiesle on both nodes The best method is to rename the propertiesle to <SG_NAME>_propertiesle for each ServiceGroup and include this le on the PRE-ONLINE/POST_OFFLINE scripts Open the <SG_NAME>_propertiesle and customize as needed In this le you should specify every object relevant to the VCS cluster, for example: nodeprd is the node at the production site: ebsips1; nodedr is the node at the DR site: ebsips2; PRD_VOLUME_0 is the FlexVol at the PRD site: vl_ebsips1_2 DR_VOLUME_0 is the FlexVol at the DR site: vl_ebsips1_2 PRD_LUN_0 is the rst LUN available to the PRD node: /vol/vl_ebsips1_2/qt_lun001/lun001 DR_LUN_0 is the rst LUN available to the DR node: /vol/vl_ebsips1_2/qt_lun001/lun001 PRD_LUN_0_ID is the LUN ID of the rst LUN on the PRD site: 1 DR_LUN_0_ID is the LUN ID of the rst LUN on the DR site: 1 PRD_IGROUP is the IGROUP NAME of the PRD host DR_IGROUP is the IGROUP NAME of the DR host!/usr/bin/bash export nodeprd=ebsips1 export nodedr=ebsips2 export lerprd=ddc-fas-p03 export lerdr=cdc-fas-p01 dene the volume names on PRD and DR lers export PRD_VOLUME_0="vl_EBSIPS1_2" export PRD_VOLUME_1="" export PRD_VOLUME_2="" export PRD_VOLUME_3="" export PRD_VOLUME_4="" export DR_VOLUME_0="vl_EBSIPS1_2" export DR_VOLUME_1="" export DR_VOLUME_2="" export DR_VOLUME_3="" export DR_VOLUME_4="" dene the LUNS availble on the PRD host from the PRD ler export PRD_LUN_0="/vol/vl_EBSIPS1_2/qt_LUN001/LUN001" export PRD_LUN_0_ID=1 export PRD_LUN_1="/vol/vl_EBSIPS1_2/qt_LUN002/LUN002" export PRD_LUN_1_ID=2 export PRD_LUN_2="" export PRD_LUN_2_ID= export PRD_LUN_2="" export PRD_LUN_2_ID= dene the LUNS to be made availble on the DR host from the DR ler export DR_LUN_0="/vol/vl_EBSIPS1_2/qt_LUN001/LUN001" export DR_LUN_0_ID=1 export DR_LUN_1="/vol/vl_EBSIPS1_2/qt_LUN002/LUN002" export DR_LUN_1_ID=2 export DR_LUN_2="" export DR_LUN_2_ID= What are the igroups on DR and on the PRD lers? export PRD_IGROUP=EBSIPS1 export DR_IGROUP=EBSIPS2 Page 16
411 TASK 11: EDIT SG_TEST_PRE- ONLINE_FAILOVER_TO_EBSIPS2SH ON EBSIPS2 If you have multiple ServiceGroups you need to have a dedicated PRE-ONLINE_failover_to_ebsips2sh, we recommend renaming the script to <SG_NAME>_PRE_ONLINE_failover_to_ebsips2sh Open /home/netapp/scripts/<sg_name>_pre-online_failover_to_ebsips2sh and uncomment the required entries for each LUN and FlexVol in use by the Service Group Then modify the rst entry for the propertiesle to that it reflects the name of the ServiceGroup, <SG_NAME>_propertiesle /home/netapp/scripts/sg_test_propertiesle -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd offlinelun $lerprd $PRD_LUN_0 0 tee -a $LOGFILE offlinelun $lerprd $PRD_LUN_1 0 tee -a $LOGFILE offlinelun $lerprd $PRD_LUN_2 0 tee -a $LOGFILE offlinelun $lerprd $PRD_LUN_3 0 tee -a $LOGFILE -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE flexvols IN USE BY $nodedr smbreak $lerdr $DR_VOLUME_0 smbreak $lerdr $DR_VOLUME_1 smbreak $lerdr $DR_VOLUME_2 smbreak $lerdr $DR_VOLUME_3 -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodedr unmaplun $lerdr $DR_LUN_0 $DR_IGROUP unmaplun $lerdr $DR_LUN_1 $DR_IGROUP unmaplun $lerdr $DR_LUN_2 $DR_IGROUP unmaplun $lerdr $DR_LUN_3 $DR_IGROUP -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodedr online luns onlinelun $lerdr $DR_LUN_0 1 tee -a $LOGFILE onlinelun $lerdr $DR_LUN_1 1 tee -a $LOGFILE onlinelun $lerdr $DR_LUN_2 1 tee -a $LOGFILE onlinelun $lerdr $DR_LUN_3 1 tee -a $LOGFILE map luns -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodedr maplun $lerdr $DR_LUN_0 $DR_IGROUP $DR_LUN_0_ID maplun $lerdr $DR_LUN_1 $DR_IGROUP $DR_LUN_1_ID maplun $lerdr $DR_LUN_2 $DR_IGROUP $DR_LUN_1_ID maplun $lerdr $DR_LUN_3 $DR_IGROUP $DR_LUN_2_ID Page 17
412 TASK 12: EDIT SG_TEST_POST- OFFLINE_FAILBACK_TO_EBSIPS1SH ON NODE EBSIPS2 If you have multiple ServiceGroups you need to have a dedicated POST_OFFLINE_failback_to_ebsips1sh, we recommend renaming this script to <SG_NAME>_POST_OFFLINE_failback_to_ebsips1sh Open /home/netapp/scripts/<sg_name>_post-offline_failback_to_ebsips1sh and uncomment the required entries for each LUN and FlexVol in use by the Service Group Then modify the entry for the propertiesle to that it reflects the name of the ServiceGroup, <SG_NAME>_propertiesle /home/netapp/scripts/sg_test_propertiesle offline luns on node nodedr, we don't want this node to modify any data while we are failing back to PRD -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodedr offlinelun $lerdr $DR_LUN_0 0 tee -a $LOGFILE offlinelun $lerdr $DR_LUN_1 0 tee -a $LOGFILE offlinelun $lerdr $DR_LUN_2 0 tee -a $LOGFILE offlinelun $lerdr $DR_LUN_3 0 tee -a $LOGFILE Page 18
413 TASK 13: EDIT PRE-ONLINE_FAILBACK_TO_EBSIPS1SH ON EBSIPS1 If you have multiple ServiceGroups you need to have a dedicated PRE-ONLINE_failback_to_ebsips1sh, we recommend renaming the script to <SG_NAME>_ PRE-ONLINE_failback_to_ebsips1sh Open /home/netapp/scripts/<sg_name>_pre-online_failback_to_ebsips1sh and uncomment the required entries for each LUN and FlexVol in use by the Service Group Then modify the entry for the propertiesle so that it reflects the name of the ServiceGroup, <SG_NAME>_propertiesle /home/netapp/scripts/sg_test_propertiesle -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE VOLUMES IN USE BY $nodeprd smbreak $lerprd $PRD_VOLUME_0 smbreak $lerprd $PRD_VOLUME_1 smbreak $lerprd $PRD_VOLUME_2 smbreak $lerprd $PRD_VOLUME_3 Invert the replication, we were doing DR->PRD, and now we want PRD->DR invertsync $lerprd $PRD_VOLUME_0 $lerdr $DR_VOLUME_0 invertsync $lerprd $PRD_VOLUME_1 $lerdr $DR_VOLUME_1 invertsync $lerprd $PRD_VOLUME_2 $lerdr $DR_VOLUME_2 unmap the luns to avoid errors during onlining -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd unmaplun $lerprd $DR_LUN_0 $PRD_IGROUP unmaplun $lerprd $DR_LUN_1 $PRD_IGROUP unmaplun $lerprd $DR_LUN_2 $PRD_IGROUP unmaplun $lerprd $DR_LUN_3 $PRD_IGROUP online and map the luns at the PRD site -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd onlinelun $lerprd $PRD_LUN_0 1 tee -a $LOGFILE onlinelun $lerprd $PRD_LUN_1 1 tee -a $LOGFILE onlinelun $lerprd $PRD_LUN_2 1 tee -a $LOGFILE onlinelun $lerprd $PRD_LUN_3 1 tee -a $LOGFILE map luns -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd maplun $lerprd $PRD_LUN_0 $PRD_IGROUP $PRD_LUN_0_ID maplun $lerprd $PRD_LUN_1 $PRD_IGROUP $PRD_LUN_1_ID maplun $lerprd $PRD_LUN_2 $PRD_IGROUP $PRD_LUN_1_ID maplun $lerprd $PRD_LUN_3 $PRD_IGROUP $PRD_LUN_2_ID Page 19
414 TASK 14: EDIT SG_TEST_RESYNC_FROM_EBSIPS2_TO_EBSIPS1SH ON NODE EBSIPS2 If you have multiple ServiceGroups you need to have a dedicated Resync_from_ebsips2_to_ebsips1sh, we recommend renaming the script to <SG_NAME>_ Resync_from_ebsips2_to_ebsips1sh Open /home/netapp/scripts/<sg_name>_ Resync_from_ebsips2_to_ebsips1sh and include entries for each FlexVol in use by the Service Group ssh vcs_$nodedr@$lerprd snapmirror resync -f -S $lerdr:$dr_volume_0 -w $lerprd:$prd_volume_0 tee -a $LOGFILE while [ $( ssh vcs_$nodedr@$lerprd snapmirror status grep -i ^$lerdr:$dr_volume_0 grep -i $lerprd:$prd_volume_0 grep -c "In-sync" tee -a $LOGFILE ) -lt 1 ] ssh vcs_$nodedr@$lerprd snapmirror status grep -i ^$lerdr:$dr_volume_0 grep -i $lerprd:$prd_volume_0 grep "In-sync" tee -a $LOGFILE Page 20
Task 15: Run scripts and check results 1) LUNs must be available on ebsips1 2) Conrm that SnapMirror has been initialized and is replicating to ebsips2 3) Run sg_test_pre-online_failover_to_ebsips2sh from host ebsips2 4) Run sg_test_post_offline_failback_to_ebsips1sh from host ebsips2 5) Run sg_test_resync_from_ebsips2_to_ebsips1sh from host ebsips2 6) Run sg_test_pre-online_failback_to_ebsips1sh from host ebsips1 Task 16: create VCS service Group and resource group sg_test ( SystemList = ebsips1 = 0, ebsips2 = 1 AutoStart = 0 AutoFailOver = 0 AutoStartList = ebsips1, ebsips2 PreOnline = 1 OnlineRetryLimit = 3 OnlineRetryInterval = 120 ) DiskGroup dgapp_ebsp1 ( Critical = 1 DiskGroup = dgapp_ebsp1 ) Mount mt_opt_vrtsnbu ( Critical = 1 MountPoint = "/opt/vrtsnbu" BlockDevice = "/dev/vx/dsk/dgapp_ebsp1/v_opt_vrtsnbu" FSType = vxfs FsckOpt = "-y" ) mt_opt_vrtsnbu requires dgapp_ebsp1 Task 17: Test VCS integration 1) Bring SG online on ebsips1 2) Offline SG on ebsip1 3) Conrm that SnapMirror is in-sync 4) Online SG on ebsip2 5) Offline SG on ebsip2 6) Conrm that SnapMirror is in-sync 7) Online SG on ebsip1 Page 21
5 OPERATION MANUAL Before bringing the ServiceGroup online on any node, we must check that the SnapMirror relationships are set and in-sync SnapMirror can take a few minutes to propagate the changes after a resync, these changes must be fully propagated before we bring online the ServiceGroup in any node Failure in doing so, may result in loss of data not yet propagated to the destination FlexVol We can verify the status of the SnapMirror Status by running the following commands: From ebsips1: ebsips1> ssh vcs_ebsips1@cdc-fas-p01 snapmirror status SnapMirror is on Source Destination State Lag Status ddc-fas-p03:vl_ebsips1_2 CDC-FAS-P01:vl_EBSIPS1_2 Snapmirrored - in-sync CDC-FAS-P01:vl_EBSIPS1_2 ddc-fas-p03:vl_ebsips1_2 Source 01:06:56 Idle From ebsips2: ebsips1> ssh vcs_ebsips2@ddc-fas-p03 snapmirror status SnapMirror is on Source Destination State Lag Status ddc-fas-p03:vl_ebsips1_2 CDC-FAS-P01:vl_EBSIPS1_2 Snapmirrored - in-sync CDC-FAS-P01:vl_EBSIPS1_2 ddc-fas-p03:vl_ebsips1_2 Source 01:06:56 Pending The output above shows that FlexVol vl_ebsips1_2 is being replicated from (PRD)ddc-fas-p03 to (DR)cdcfas-p01 and its current replication status is in-sync Occasionally you may found that the State is listed as Snapmirrored but there is a number on the Lag column, this indicates that SnapMirror is not yet running in Synchronous mode but propagating changes after loss of connection or a resync operation If this happens wait a few minutes and check the SnapMirror output again To bring the ServiceGroup online on ebsips1 follow these steps: ebsips1> hastatus -sum Conrm that the ServiceGroup is not running in any other node ebsips1> ssh vcs_ebsips1@cdc-fas-p01 snapmirror status and conrm that SnapMirror Status is in-sync ebsips1> hagrp online sg_test sys ebsips1 To bring the ServiceGroup offline follow these steps: ebsips1> hagrp offline sg_test any To bring the ServiceGroup online on ebsips2 follow these steps: ebsips2> hastatus -sum Conrm that the ServiceGroup is not running in any other node ebsips2> ssh vcs_ebsips2@ddc-fas-p03 snapmirror status and conrm that SnapMirror Status is in-sync ebsips2> hagrp online sg_test sys ebsips2 Page 22
6 LOGS AND TROUBLESHOOTING There are a set of logs that can be review for troubleshooting: The directory /home/netapp/logs contain logs of the different preonline/postoffline scripts ebsips2> cd /home/netapp/logs ebsips2> ls ltr grep sg_test tail -3 These commands will return the last 3 logs: 190608_121329_sg_test_PRE-ONLINE_failover_to_ebsips2log 190608_121423_sg_test_resync_from_ebsips2_to_ebsips1log 190608_122352_sg_test_POST-OFFLINE_failback_to_ebsips1log And on ebsips1: ebsips1> cd /home/netapp/logs ebsips1> ls ltr grep sg_test tail -1 190608_121329_sg_test_PRE-ONLINE_failback_to_ebsips1log Additionally the VCS logs can also be found at /var/vrtsvcs/log Page 23
7 APPENDIX 1 71 PROPERTIESFILE!/usr/bin/bash export nodeprd=ebsips1 export nodedr=ebsips2 export lerprd=ddc-fas-p03 export lerdr=cdc-fas-p01 dene the volume names on PRD and DR lers export PRD_VOLUME_0="vl_EBSIPS1_2" export PRD_VOLUME_1="" export PRD_VOLUME_2="" export PRD_VOLUME_3="" export PRD_VOLUME_4="" export DR_VOLUME_0="vl_EBSIPS1_2" export DR_VOLUME_1="" export DR_VOLUME_2="" export DR_VOLUME_3="" export DR_VOLUME_4="" dene the LUNS availble on the PRD host from the PRD ler export PRD_LUN_0="/vol/vl_EBSIPS1_2/qt_LUN001/LUN001" export PRD_LUN_0_ID=1 export PRD_LUN_1="/vol/vl_EBSIPS1_2/qt_LUN002/LUN002" export PRD_LUN_1_ID=2 export PRD_LUN_2="" export PRD_LUN_2_ID= export PRD_LUN_2="" export PRD_LUN_2_ID= dene the LUNS to be made availble on the DR host from the DR ler export DR_LUN_0="/vol/vl_EBSIPS1_2/qt_LUN001/LUN001" export DR_LUN_0_ID=1 export DR_LUN_1="/vol/vl_EBSIPS1_2/qt_LUN002/LUN002" export DR_LUN_1_ID=2 export DR_LUN_2="" export DR_LUN_2_ID= What are the igroups on DR and on the PRD lers? export PRD_IGROUP=EBSIPS1 export DR_IGROUP=EBSIPS2 Page 24
72 EBSIPS1 - PREONLINE $Id: preonline,v 22221 2006/06/23 08:00:14 vikasg Exp $ $Copyrights: Copyright (c) 2006 Symantec Corporation All rights reserved THIS SOFTWARE CONTAINS CONFIDENTIAL INFORMATION AND TRADE SECRETS OF SYMANTEC CORPORATION USE, DISCLOSURE OR REPRODUCTION IS PROHIBITED WITHOUT THE PRIOR EXPRESS WRITTEN PERMISSION OF SYMANTEC CORPORATION The Licensed Software and Documentation are deemed to be "commercial computer software" and "commercial computer software documentation" as dened in FAR Sections 12212 and DFARS Section 2277202 $ preonline preonline is invoked just before onlining the group preonline is invoked on the node where group is to be onlined A group level congurable parameter PreOnline controls whether this trigger should be invoked or not By default, PreOnline is not set PreOnline can be set in one of two ways: a) In conguration le, dene PreOnline=1 in the group description to set PreOnline to 1 for the group b) While cluster is running, and in writable state, do hagrp -modify <group> PreOnline 1 to set PreOnline to 1 for group <group> preonline gives user the control to take appropriate action depending on whether group is being manually onlined, or group is in the process of failover In failover case, for example, preonline can be used to determine whether the group can be brought online on this node in the cluster In any case, user can give control back to engine by doing: hagrp -online -nopre <group> -sys <system> This will let engine continue with online process Usage: preonline <system> <group> <whyonlining> <systemwheregroupfaulted> <system>: is the name of the system where group is to be onlined <group>: is the name of the group that is to be onlined <whyonlining>: is either "FAULT" or "MANUAL" "MANUAL" corresponds to manual online whereas "FAULT" corresponds to both failover as well as manual switch <systemwheregroupfaulted>: When preonline is invoked due to failover this argument is the name of the system where group was online before When preonline is invoked due to group online command issued with -checkpartial option, this argument is the name of system specied for this option eval 'exec $VCS_HOME:-/opt/VRTSvcs/bin/perl5 -I $VCS_HOME:-/opt/VRTSvcs/lib -S $0 $1+"$@"' if 0; $vcs_home = $ENV"VCS_HOME"; if (!dened ($vcs_home)) $vcs_home="/opt/vrtsvcs"; use ag_i18n_inc; VCSAG_SET_ENVS(); if (!dened $ARGV[0]) VCSAG_LOG_MSG ("W", "Failed to continue; undened system name", 15028); exit; elsif (!dened $ARGV[1]) VCSAG_LOG_MSG ("W", "Failed to continue; undened group name", 15031); exit; Page 25
It is a good idea to log what you're doing if (dened $ARGV[3]) VCSAG_LOG_MSG("I", "(preonline) Invoked with arg0=$argv[0], arg1=$argv[1], arg2=$argv[2], arg3=$argv[3]", msgid); else VCSAG_LOG_MSG("I", "(preonline) Invoked with arg0=$argv[0], arg1=$argv[1]", msgid); put your code here if ( $ARGV[0] eq "ebsips1" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_pre-online_failback_to_ebsips1sh"); Here is a sample code that takes into account multiple groups $group = $ARGV[1]; if ($group eq "G1") Say, G1 can not be onlined on this system We add heuristics to determine next best system to online G1 Say, sysb is such a system `$vcs_home/bin/hagrp -online G1 -sys sysb`; exit now, without sending online -nopre exit; elsif ($group eq "G2") We add heurisitics to determine if G2 can be onlined on this system Say, we determine that G2 can not be onlined anywhere Exit now without sending online -nopre exit; elsif ($group eq "G3") Say, to online G3 we want to make sure if another group P1 is online Query engine for P1's online state, using hagrp -state P1 -sys $ARGV[0] Say, that P1 is indeed online on this system Don't call exit, since online nopre is called down below Here is a sample code to notify a bunch of users @recipients=("username\@servernamecom"); $msgle="/tmp/preonline"; `echo system = $ARGV[0], group = $ARGV[1], whyonlining = $ARGV[2] > $msgle`; foreach $recipient (@recipients) Must have elm setup to run this `elm -s preonline $recipient < $msgle`; `rm $msgle`; give control back to HAD if (dened $ARGV[3]) system("$vcs_home/bin/hagrp -online -nopre $ARGV[1] -sys $ARGV[0] -checkpartial $ARGV[3]"); exit; system("$vcs_home/bin/hagrp -online -nopre $ARGV[1] -sys $ARGV[0]"); exit; 73 EBSIPS1 PRE-ONLINE_FAILBACK_TO_EBSIPS1SH!/usr/bin/bash PRE-ONLINE script: PRE-ONLINE_failback_to_ebsips1 Version 11 NetApp - 6/7/2008 This script is used by VCS preonline trigger to make the PRD luns available on node ebsips1 Page 26
In order to run this script, resync_from_ebsips2_to_ebsips1sh must have been executed already resync_from_ebsips2_to_ebsips1sh performs the sync from DR->PRD This script wiil bring PRD luns online and reverse the sync from DR->PRD to PRD->DR LOGFILE="/home/netapp/logs/`date +%d%m%y_%h%m%s`_sg_test_pre- ONLINE_failback_to_ebsips1log" /home/netapp/scripts/sg_test_propertiesle ------------------------------------------------------------------------ functions: function pingfilerup usage: pingfilerup <ler> <0 1> where <ler> is the ler we want to check if is online and <0 1> denes if the script should abort if the ler is not online <1> = critical pingfilerup () FILER=$1 CRITICAL=$2 echo "checking if ler $FILER is up and running" tee -a $LOGFILE if [ "$( ping $FILER 1 grep -c answer )" -gt 0 ] echo "ler $FILER is down" tee -a $LOGFILE if [ "$CRITICAL" -eq "1" ] echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 else echo "ler $FILER is up" tee -a $LOGFILE function offlinelun usage: offlinelun <ler> <lun> <0 1> where <ler> is the ler where we want to online the lun <lun> is the full pathname of that lun and <0 1> denes if the script should abort if the operation failed <1> = critical offlinelun () FILER=$1 LUN=$2 CRITICAL=$3 echo "offlining luns on $lerprd " tee -a $LOGFILE ssh vcs_$nodeprd@$filer lun offline $LUN tee -a $LOGFILE -> conrm that luns are offline if [ "$( ssh vcs_$nodeprd@$filer lun show $LUN grep -c online )" -eq "1" ] echo "WARNING: can't conrm that LUN $FILER:$LUN is offline" if [ "$CRITICAL" -eq "1" ] echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 function onlinelun usage: onlinelun <ler> <lun> <0 1> where <ler> is the ler where we want to online the lun <lun> is the full pathname of that lun and <0 1> denes if the script should abort if the operation failed <1> = critical onlinelun () FILER=$1 LUN=$2 CRITICAL=$3 Page 27
echo "onlining luns $LUN on $FILER " tee -a $LOGFILE ssh vcs_$nodeprd@$filer lun online $LUN tee -a $LOGFILE -> conrm that luns are online if [ "$( ssh vcs_$nodeprd@$filer lun show $LUN grep -c offline )" -eq "1" ] echo "WARNING: can't conrm that LUN $FILER:$LUN is online" if [ "$CRITICAL" -eq "1" ] echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 function smbreak usage: smbreak <ler> <volume> where <ler> is the destination ler where we want to break the SnapMirror relationship and <volume> is the volume name smbreak () FILER=$1 VOLUME=$2 snapmirror quiesce and release on the destination site echo "running snapmirror quiesce on volume $FILER:$VOLUME" tee -a $LOGFILE ssh vcs_$nodeprd@$filer snapmirror quiesce $FILER:$VOLUME tee -a $LOGFILE echo "running snapmirror break on volume $FILER:$VOLUME" tee -a $LOGFILE ssh vcs_$nodeprd@$filer snapmirror break $FILER:$VOLUME tee -a $LOGFILE -> conrm that snapmirror actions are done if [ "$( ssh vcs_$nodeprd@$filer snapmirror status $FILER:$VOLUME tee -a $LOGFILE grep -c Broken )" -lt 1 ] FATA:failed to break the SnapMirror relationship echo "FATAL: failed to break the SnapMirror relationship" tee -a $LOGFILE echo "snapmirror break $FILER:$VOLUME failed" tee -a $LOGFILE exit 255 function unmaplun usage: unmaplun <ler> <lun> <igroup> where <ler> is the destination ler where we want to unmap <lun> from the <IGROUP> unmaplun () FILER=$1 LUN=$2 IGROUP=$3 unmap luns to avoid problems during onlining echo "unmap luns $LUN from igroup $IGROUP on ler $FILER " tee -a $LOGFILE ssh vcs_$nodeprd@$filer lun unmap $LUN $IGROUP tee -a $LOGFILE function maplun usage: maplun <ler> <lun> <igroup> <lun_id> <0 1> where <ler> is the destination ler where we want to map <lun> to the <IGROUP> using <lun_id> maplun () FILER=$1 LUN=$2 IGROUP=$3 LUNID=$4 CRITICAL=$5 map luns echo "map lun $LUN to igroup $IGROUP on ler $FILER using lun id $LUNID " tee - a $LOGFILE ssh vcs_$nodeprd@$filer lun map $LUN $IGROUP $LUNID tee -a $LOGFILE check if the luns have been mapped correctly Page 28
if [ "$( ssh vcs_$nodeprd@$filer lun show $LUN tee -a $LOGFILE grep -c mapped )" -lt 1 ] if [ "$CRITICAL" -eq "1" ] echo "FATAL: failed to map lun $FILER:$LUN to $IGROUP" tee -a $LOGFILE echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 function invertsync usage: inversync <PRD ler> <PRD volume> <DR ler> <DR volume> where <PRD ler> and <PRD VOLUME> is source of the snapmirror we want to establish and <DR ler> and <DR volume> the destination invertsync () PRDFILER=$1 PRDVOLUME=$2 DRFILER=$3 DRVOLUME=$4 Invert the replication, we were doing DR->PRD, and now we want PRD->DR echo "Invert the replication: from DR->PRD to PRD->DR" tee -a $LOGFILE ssh vcs_$nodeprd@$drfiler snapmirror resync -f -S $PRDFILER:$PRDVOLUME -w $DRFILER:$DRVOLUME tee -a $LOGFILE check that the Snapmirror is replication the correct way we can't abort the PRE-ONLINE script it the replication fails to be activacted from PRD->DR as DR may not be available echo "Verify the replication: it should be from PRD->DR" tee -a $LOGFILE ssh vcs_$nodeprd@$lerdr snapmirror status grep -i ^$PRDFILER:$PRDVOLUME grep -i $DRFILER:$DRVOLUME grep Snapmirrored tee -a $LOGFILE ------------------------------------------------------ MAIN echo "starting $0 at $(date)" tee -a $LOGFILE echo tee -a $LOGFILE echo "dumping enviroment: " tee -a $LOGFILE env tee -a $LOGFILE echo tee -a $LOGFILE check if ler $lerprd is up and running pingfilerup $lerprd 1 Break the current SnapMirror We may be returning from failure scenario where DR is replicating to PRD we need to stop this DR -> PRD replication and activate the volumes on PRD -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE VOLUMES IN USE BY $nodeprd smbreak $lerprd $PRD_VOLUME_0 smbreak $lerprd $PRD_VOLUME_1 smbreak $lerprd $PRD_VOLUME_2 smbreak $lerprd $PRD_VOLUME_3 check if ler $lerdr is up and running pingfilerup $lerdr 0 if [ $? -lt 1 ] Invert the replication, we were doing DR->PRD, and now we want PRD->DR invertsync $lerprd $PRD_VOLUME_0 $lerdr $DR_VOLUME_0 invertsync $lerprd $PRD_VOLUME_1 $lerdr $DR_VOLUME_1 invertsync $lerprd $PRD_VOLUME_2 $lerdr $DR_VOLUME_2 check if ler $lerprd is up and running pingfilerup $lerprd 1 Page 29
if [ $? -lt 1 ] unmap the luns to avoid errors during onlining -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd unmaplun $lerprd $DR_LUN_0 $PRD_IGROUP unmaplun $lerprd $DR_LUN_1 $PRD_IGROUP unmaplun $lerprd $DR_LUN_2 $PRD_IGROUP unmaplun $lerprd $DR_LUN_3 $PRD_IGROUP online and map the luns at the PRD site -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd onlinelun $lerprd $PRD_LUN_0 1 tee -a $LOGFILE onlinelun $lerprd $PRD_LUN_1 1 tee -a $LOGFILE onlinelun $lerprd $PRD_LUN_2 1 tee -a $LOGFILE onlinelun $lerprd $PRD_LUN_3 1 tee -a $LOGFILE map luns -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd maplun $lerprd $PRD_LUN_0 $PRD_IGROUP $PRD_LUN_0_ID maplun $lerprd $PRD_LUN_1 $PRD_IGROUP $PRD_LUN_1_ID maplun $lerprd $PRD_LUN_2 $PRD_IGROUP $PRD_LUN_1_ID maplun $lerprd $PRD_LUN_3 $PRD_IGROUP $PRD_LUN_2_ID rescan devices echo "rescanning devices" tee -a $LOGFILE cfgadm -al tee -a $LOGFILE devfsadm -C tee -a $LOGFILE /opt/ntap/santoolkit/bin/sanlun lun show -p >/dev/null 2>&1 /opt/ntap/santoolkit/bin/sanlun lun show -p tee -a $LOGFILE 74 EBSIPS2 RESYNC_FROM_EBSIPS2_TO_EBSIPS1SH!/usr/bin/bash Resync from ebsips2 to ebsip1 LOGFILE="/home/netapp/logs/`date +%d%m%y_%h%m%s`_sg_test_resync_from_ebsips2_to_ebsips1log" /home/netapp/scripts/sg_test_propertiesle echo "starting $0 at $(date)" tee -a $LOGFILE echo tee -a $LOGFILE echo "dumping enviroment: " tee -a $LOGFILE env tee -a $LOGFILE echo tee -a $LOGFILE check if ler $lerprd is up and running echo "checking if ler $lerprd is up and running" tee -a $LOGFILE if [ "$( ping $lerprd 1 grep -c answer )" -gt 0 ] echo "ler $lerprd is down" tee -a $LOGFILE else echo "ler $lerprd is up" tee -a $LOGFILE Invert the replication and resync DR -> PRD resync DR -> PRD this command may fail as the PRD site may not be available echo "Inverting the replication, resynching DR -> PRD" tee -a $LOGFILE ssh vcs_$nodedr@$lerprd snapmirror resync -f -S $lerdr:$dr_volume_0 -w $lerprd:$prd_volume_0 tee -a $LOGFILE -> verify that snapmirror is congured from DR -> PRD Page 30
while [ $( ssh vcs_$nodedr@$lerprd snapmirror status grep -i ^$lerdr:$dr_volume_0 grep -i $lerprd:$prd_volume_0 grep -c "In-sync" tee -a $LOGFILE ) -lt 1 ] do echo "waiting for volume to be fully synched" sleep 30 done ssh vcs_$nodedr@$lerprd snapmirror status grep -i ^$lerdr:$dr_volume_0 grep -i $lerprd:$prd_volume_0 grep "In-sync" tee -a $LOGFILE 75 EBSIPS2 POSTOFFLINE $Id: postoffline,v 221 2005/10/17 12:36:59 vikasg Exp $ $Copyrights: Copyright (c) 2006 Symantec Corporation All rights reserved THIS SOFTWARE CONTAINS CONFIDENTIAL INFORMATION AND TRADE SECRETS OF SYMANTEC CORPORATION USE, DISCLOSURE OR REPRODUCTION IS PROHIBITED WITHOUT THE PRIOR EXPRESS WRITTEN PERMISSION OF SYMANTEC CORPORATION The Licensed Software and Documentation are deemed to be "commercial computer software" and "commercial computer software documentation" as dened in FAR Sections 12212 and DFARS Section 2277202 $ postoffline postoffline is invoked after a group transitions to an OFFLINE state from a non-offline state postoffline is invoked on the node where group went OFFLINE There are no congurable settings that will turn ON/OFF invoking this trigger If you don't want this trigger to be invoked, remove postoffline* les from $VCS_HOME/bin/triggers directory Usage: postoffline <system> <group> <system>: is the name of the system where group is offlined <group>: is the name of the group that is offlined eval 'exec $VCS_HOME:-/opt/VRTSvcs/bin/perl5 -I $VCS_HOME:-/opt/VRTSvcs/lib -S $0 $1+"$@"' if 0; $vcs_home = $ENV"VCS_HOME"; if (!dened ($vcs_home)) $vcs_home="/opt/vrtsvcs"; use ag_i18n_inc; VCSAG_SET_ENVS(); if (!dened $ARGV[0]) VCSAG_LOG_MSG ("W", "Failed to continue; undened system name", 15028); exit; elsif (!dened $ARGV[1]) VCSAG_LOG_MSG ("W", "Failed to continue; undened group name", 15031); exit; It is a good idea to log what you're doing VCSAG_LOG_MSG("I", "(postoffline) Invoked with arg0=$argv[0], arg1=$argv[1]", msgid); put your code here if ( $ARGV[0] eq "ebsips2" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_post-offline_failback_to_ebsips1sh"); Page 31
Here is a sample code to notify a bunch of users @recipients=("username\@servernamecom"); $msgle="/tmp/postoffline"; `echo system = $ARGV[0], group = $ARGV[1] > $msgle`; foreach $recipient (@recipients) Must have elm setup to run this `elm -s postoffline $recipient < $msgle`; `rm $msgle`; exit; 76 EBSIPS2 PREONLINE $Id: preonline,v 22221 2006/06/23 08:00:14 vikasg Exp $ $Copyrights: Copyright (c) 2006 Symantec Corporation All rights reserved THIS SOFTWARE CONTAINS CONFIDENTIAL INFORMATION AND TRADE SECRETS OF SYMANTEC CORPORATION USE, DISCLOSURE OR REPRODUCTION IS PROHIBITED WITHOUT THE PRIOR EXPRESS WRITTEN PERMISSION OF SYMANTEC CORPORATION The Licensed Software and Documentation are deemed to be "commercial computer software" and "commercial computer software documentation" as dened in FAR Sections 12212 and DFARS Section 2277202 $ preonline preonline is invoked just before onlining the group preonline is invoked on the node where group is to be onlined A group level congurable parameter PreOnline controls whether this trigger should be invoked or not By default, PreOnline is not set PreOnline can be set in one of two ways: a) In conguration le, dene PreOnline=1 in the group description to set PreOnline to 1 for the group b) While cluster is running, and in writable state, do hagrp -modify <group> PreOnline 1 to set PreOnline to 1 for group <group> preonline gives user the control to take appropriate action depending on whether group is being manually onlined, or group is in the process of failover In failover case, for example, preonline can be used to determine whether the group can be brought online on this node in the cluster In any case, user can give control back to engine by doing: hagrp -online -nopre <group> -sys <system> This will let engine continue with online process Usage: preonline <system> <group> <whyonlining> <systemwheregroupfaulted> <system>: is the name of the system where group is to be onlined <group>: is the name of the group that is to be onlined <whyonlining>: is either "FAULT" or "MANUAL" "MANUAL" corresponds to manual online whereas "FAULT" corresponds to both failover as well as manual switch <systemwheregroupfaulted>: When preonline is invoked due to failover this argument is the name of the system where group was online before When preonline is invoked due to group online command issued with -checkpartial option, this argument is the name of system specied for this option eval 'exec $VCS_HOME:-/opt/VRTSvcs/bin/perl5 -I $VCS_HOME:-/opt/VRTSvcs/lib -S $0 $1+"$@"' if 0; $vcs_home = $ENV"VCS_HOME"; if (!dened ($vcs_home)) $vcs_home="/opt/vrtsvcs"; Page 32
use ag_i18n_inc; VCSAG_SET_ENVS(); if (!dened $ARGV[0]) VCSAG_LOG_MSG ("W", "Failed to continue; undened system name", 15028); exit; elsif (!dened $ARGV[1]) VCSAG_LOG_MSG ("W", "Failed to continue; undened group name", 15031); exit; It is a good idea to log what you're doing if (dened $ARGV[3]) VCSAG_LOG_MSG("I", "(preonline) Invoked with arg0=$argv[0], arg1=$argv[1], arg2=$argv[2], arg3=$argv[3]", msgid); else VCSAG_LOG_MSG("I", "(preonline) Invoked with arg0=$argv[0], arg1=$argv[1]", msgid); put your code here if ( $ARGV[0] eq "ebsips2" && $ARGV[1] eq "sg_test" ) system("/home/netapp/scripts/sg_test_pre-online_failover_to_ebsips2sh"); Here is a sample code that takes into account multiple groups $group = $ARGV[1]; if ($group eq "G1") Say, G1 can not be onlined on this system We add heuristics to determine next best system to online G1 Say, sysb is such a system `$vcs_home/bin/hagrp -online G1 -sys sysb`; exit now, without sending online -nopre exit; elsif ($group eq "G2") We add heurisitics to determine if G2 can be onlined on this system Say, we determine that G2 can not be onlined anywhere Exit now without sending online -nopre exit; elsif ($group eq "G3") Say, to online G3 we want to make sure if another group P1 is online Query engine for P1's online state, using hagrp -state P1 -sys $ARGV[0] Say, that P1 is indeed online on this system Don't call exit, since online nopre is called down below Here is a sample code to notify a bunch of users @recipients=("username\@servernamecom"); $msgle="/tmp/preonline"; `echo system = $ARGV[0], group = $ARGV[1], whyonlining = $ARGV[2] > $msgle`; foreach $recipient (@recipients) Must have elm setup to run this `elm -s preonline $recipient < $msgle`; `rm $msgle`; give control back to HAD if (dened $ARGV[3]) system("$vcs_home/bin/hagrp -online -nopre $ARGV[1] -sys $ARGV[0] -checkpartial $ARGV[3]"); exit; system("$vcs_home/bin/hagrp -online -nopre $ARGV[1] -sys $ARGV[0]"); exit; Page 33
77 EBSIPS2 POST-OFFLINE_FAILBACK_TO_EBSIPS1SH!/usr/bin/bash POST-OFFLINE script: _POST-OFFLINE_failback_to_ebsips Version 11 NetApp - 6/7/2008 This script is used by VCS postoffline trigger to make the DR luns offline on node ebsips2 LOGFILE="/home/netapp/logs/`date +%d%m%y_%h%m%s`_sg_test_post- OFFLINE_failback_to_ebsips1log" /home/netapp/scripts/sg_test_propertiesle ------------------------------------------------------------------------ functions: function pingfilerup usage: pingfilerup <ler> <0 1> where <ler> is the ler we want to check if is online and <0 1> denes if the script should abort if the ler is not online <1> = critical pingfilerup () FILER=$1 CRITICAL=$2 echo "checking if ler $FILER is up and running" tee -a $LOGFILE if [ "$( ping $FILER 1 grep -c answer )" -gt 0 ] echo "ler $FILER is down" tee -a $LOGFILE if [ "$CRITICAL" -eq "1" ] echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 else echo "ler $FILER is up" tee -a $LOGFILE function offlinelun usage: offlinelun <ler> <lun> <0 1> where <ler> is the ler where we want to online the lun <lun> is the full pathname of that lun and <0 1> denes if the script should abort if the operation failed <1> = critical offlinelun () FILER=$1 LUN=$2 CRITICAL=$3 echo "offlining luns on $FILER " tee -a $LOGFILE ssh vcs_$nodedr@$filer lun offline $LUN tee -a $LOGFILE -> conrm that luns are offline if [ "$( ssh vcs_$nodedr@$filer lun show $LUN grep -c online )" -eq "1" ] echo "WARNING: can't conrm that LUN $FILER:$LUN is offline" if [ "$CRITICAL" -eq "1" ] echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 ------------------------------------------------------ MAIN echo "starting $0 at $(date)" tee -a $LOGFILE echo tee -a $LOGFILE echo "dumping enviroment: " tee -a $LOGFILE env tee -a $LOGFILE Page 34
echo tee -a $LOGFILE check if ler $lerdr is up and running pingfilerup $lerdr 0 if [ $? -lt 1 ] offline luns on node nodedr, we don't want this node to modify any data while we are failing back to PRD -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd offlinelun $lerdr $DR_LUN_0 0 tee -a $LOGFILE offlinelun $lerdr $DR_LUN_1 0 tee -a $LOGFILE offlinelun $lerdr $DR_LUN_2 0 tee -a $LOGFILE offlinelun $lerdr $DR_LUN_3 0 tee -a $LOGFILE rescan devices echo "rescanning devices" tee -a $LOGFILE cfgadm -al tee -a $LOGFILE devfsadm -C tee -a $LOGFILE /opt/ntap/santoolkit/bin/sanlun lun show -p >/dev/null 2>&1 /opt/ntap/santoolkit/bin/sanlun lun show -p tee -a $LOGFILE 78 EBSIPS2 PRE-ONLINE_FAILOVER_TO_EBSIPS2SH!/usr/bin/bash PRE-ONLINE script: PRE-ONLINE_failover_to_ebsips2sh Version 11 NetApp - 6/7/2008 This script is used by VCS preonline trigger to make the DR luns available on node ebsips2 LOGFILE="/home/netapp/logs/`date +%d%m%y_%h%m%s`_sg_test_pre- ONLINE_failover_to_ebsips2log" /home/netapp/scripts/sg_test_propertiesle ------------------------------------------------------------------------ functions: function pingfilerup usage: pingfilerup <ler> <0 1> where <ler> is the ler we want to check if is online and <0 1> denes if the script should abort if the ler is not online <1> = critical pingfilerup () FILER=$1 CRITICAL=$2 echo "checking if ler $FILER is up and running" tee -a $LOGFILE if [ "$( ping $FILER 1 grep -c answer )" -gt 0 ] echo "ler $FILER is down" tee -a $LOGFILE if [ "$CRITICAL" -eq "1" ] echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 else echo "ler $FILER is up" tee -a $LOGFILE function offlinelun usage: offlinelun <ler> <lun> <0 1> where <ler> is the ler where we want to online the lun <lun> is the full pathname of that lun and <0 1> denes if the script should abort if the operation failed <1> = critical offlinelun () Page 35
FILER=$1 LUN=$2 CRITICAL=$3 echo "offlining luns on $lerprd " tee -a $LOGFILE ssh vcs_$nodedr@$filer lun offline $LUN tee -a $LOGFILE -> conrm that luns are offline if [ "$( ssh vcs_$nodedr@$filer lun show $LUN grep -c online )" -eq "1" ] echo "WARNING: can't conrm that LUN $FILER:$LUN is offline" if [ "$CRITICAL" -eq "1" ] echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 function onlinelun usage: onlinelun <ler> <lun> <0 1> where <ler> is the ler where we want to online the lun <lun> is the full pathname of that lun and <0 1> denes if the script should abort if the operation failed <1> = critical onlinelun () FILER=$1 LUN=$2 CRITICAL=$3 echo "onlining luns $LUN on $FILER " tee -a $LOGFILE ssh vcs_$nodedr@$filer lun online $LUN tee -a $LOGFILE -> conrm that luns are online if [ "$( ssh vcs_$nodedr@$filer lun show $LUN grep -c offline )" -eq "1" ] echo "WARNING: can't conrm that LUN $FILER:$LUN is online" if [ "$CRITICAL" -eq "1" ] echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 function smbreak usage: smbreak <ler> <volume> where <ler> is the destination ler where we want to break the SnapMirror relationship and <volume> is the volume name smbreak () FILER=$1 VOLUME=$2 snapmirror quiesce and release on the destination site echo "running snapmirror quiesce on volume $FILER:$VOLUME" tee -a $LOGFILE ssh vcs_$nodedr@$filer snapmirror quiesce $FILER:$VOLUME tee -a $LOGFILE echo "running snapmirror break on volume $FILER:$VOLUME" tee -a $LOGFILE ssh vcs_$nodedr@$filer snapmirror break $FILER:$VOLUME tee -a $LOGFILE -> conrm that snapmirror actions are done if [ "$( ssh vcs_$nodedr@$filer snapmirror status $FILER:$VOLUME tee -a $LOGFILE grep -c Broken )" -lt 1 ] FATA:failed to break the SnapMirror relationship echo "FATAL: failed to break the SnapMirror relationship" tee -a $LOGFILE echo "snapmirror break $FILER:$VOLUME failed" tee -a $LOGFILE exit 255 function unmaplun Page 36
usage: unmaplun <ler> <lun> <igroup> where <ler> is the destination ler where we want to unmap <lun> from the <IGROUP> unmaplun () FILER=$1 LUN=$2 IGROUP=$3 unmap luns to avoid problems during onlining echo "unmap luns $LUN from igroup $IGROUP on ler $FILER " tee -a $LOGFILE ssh vcs_$nodedr@$filer lun unmap $LUN $IGROUP tee -a $LOGFILE function maplun usage: maplun <ler> <lun> <igroup> <lun_id> <0 1> where <ler> is the destination ler where we want to map <lun> to the <IGROUP> using <lun_id> maplun () FILER=$1 LUN=$2 IGROUP=$3 LUNID=$4 CRITICAL=$5 map luns echo "map lun $LUN to igroup $IGROUP on ler $FILER using lun id $LUNID " tee - a $LOGFILE ssh vcs_$nodedr@$filer lun map $LUN $IGROUP $LUNID tee -a $LOGFILE check if the luns have been mapped correctly if [ "$( ssh vcs_$nodedr@$filer lun show $LUN tee -a $LOGFILE grep -c mapped )" -lt 1 ] if [ "$CRITICAL" -eq "1" ] echo "FATAL: failed to map lun $FILER:$LUN to $IGROUP" tee -a $LOGFILE echo "FATAL: aborting PRE-ONLINE" tee -a $LOGFILE exit 255 else return 1 ------------------------------------------------------ MAIN echo "starting $0 at $(date)" tee -a $LOGFILE echo tee -a $LOGFILE echo "dumping enviroment: " tee -a $LOGFILE env tee -a $LOGFILE echo tee -a $LOGFILE check if ler $lerprd is up and running pingfilerup $lerprd 0 if [ $? -lt 1 ] offline luns on node nodeprd, we don't want this node to modify any data while we are running from DR -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodeprd offlinelun $lerprd $PRD_LUN_0 0 tee -a $LOGFILE offlinelun $lerprd $PRD_LUN_1 0 tee -a $LOGFILE offlinelun $lerprd $PRD_LUN_2 0 tee -a $LOGFILE offlinelun $lerprd $PRD_LUN_3 0 tee -a $LOGFILE check if ler $lerdr is up and running pingfilerup $lerdr 1 if [ $? -lt 1 ] snapmirror Break on the destination site Page 37
-> EDIT propertiesle AND THIS SECTION TO INCLUDE THE flexvols IN USE BY $nodedr smbreak $lerdr $DR_VOLUME_0 smbreak $lerdr $DR_VOLUME_1 smbreak $lerdr $DR_VOLUME_2 smbreak $lerdr $DR_VOLUME_3 unmap luns to avoid problems during onlining -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodedr unmaplun $lerdr $DR_LUN_0 $DR_IGROUP unmaplun $lerdr $DR_LUN_1 $DR_IGROUP unmaplun $lerdr $DR_LUN_2 $DR_IGROUP unmaplun $lerdr $DR_LUN_3 $DR_IGROUP -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodedr online luns onlinelun $lerdr $DR_LUN_0 1 tee -a $LOGFILE onlinelun $lerdr $DR_LUN_1 1 tee -a $LOGFILE onlinelun $lerdr $DR_LUN_2 1 tee -a $LOGFILE onlinelun $lerdr $DR_LUN_3 1 tee -a $LOGFILE map luns -> EDIT propertiesle AND THIS SECTION TO INCLUDE THE LUNS IN USE BY $nodedr maplun $lerdr $DR_LUN_0 $DR_IGROUP $DR_LUN_0_ID maplun $lerdr $DR_LUN_1 $DR_IGROUP $DR_LUN_1_ID maplun $lerdr $DR_LUN_2 $DR_IGROUP $DR_LUN_1_ID maplun $lerdr $DR_LUN_3 $DR_IGROUP $DR_LUN_2_ID rescan luns on the host echo "rescan devices " tee -a $LOGFILE cfgadm -al tee -a $LOGFILE devfsadm -C tee -a $LOGFILE /opt/ntap/santoolkit/bin/sanlun lun show -p >/dev/null 2>&1 /opt/ntap/santoolkit/bin/sanlun lun show -p tee -a $LOGFILE last minute change requested from ACME, they would like the failback replication to be set and initiated automatically This hasn t been tested! /home/netapp/scripts/sg_test_resync_from_ebsips2_to_ebsips1sh Page 38