MetroCluster Systems Running Clustered Data ONTAP MetroCluster Service Guide

Transcription

1 MetroCluster Systems Running Clustered Data ONTAP MetroCluster Service Guide June _D0

2

3 Table of Contents 3 Contents Where to find MetroCluster documentation... 5 Where to find procedures for MetroCluster maintenance tasks... 7 Performing FC-to-SAS bridge maintenance... 9 FibreBridge 7500N bridges in MetroCluster configurations... 9 Updating the firmware on a FibreBridge bridge Replacing a single FC-to-SAS bridge Verifying storage connectivity Hot-swapping the bridge with a replacement bridge of the same model Hot-swapping a FibreBridge 6500N bridge with a FibreBridge 7500N bridge Replacing a pair of FibreBridge 6500N bridges with 7500N bridges Verifying storage connectivity Hot-swapping FibreBridge 6500N bridges to create a pair of FibreBridge 7500N bridges Cabling the bridge SAS ports when consolidating storage behind FibreBridge 7500N bridges Updating zoning when adding FibreBridge 7500N bridges to a configuration Cabling the second bridge FC port when adding FibreBridge 7500N bridges to a configuration Disabling unused FC-to-SAS bridge ports Replacing SAS copper cables Requirements for using other interfaces to configure and manage FibreBridge bridges Hot-replacing a failed power supply module Zoning for an eight- or a four-node MetroCluster configuration using FibreBridge 7500N bridges Zoning for FibreBridge 6500N bridges in a four-node MetroCluster configuration FC switch maintenance and replacement Upgrading firmware on a Brocade FC switch Upgrading firmware on a Cisco FC switch Replacing a Brocade FC switch Replacing a Cisco FC switch Nondisruptively replacing SAS cables Hot-adding storage to the MetroCluster configuration Hot-adding a SAS disk shelf in a direct-attached MetroCluster configuration using SAS optical cables Installing a disk shelf for a hot-add Cabling the hot-added disk shelf Verifying the disk drive and disk shelf firmware versions... 80

4 4 MetroCluster Service Guide Hot-adding SAS storage to a bridge-attached MetroCluster configuration Hot-adding a stack of SAS disk shelves to an existing pair of FibreBridge 7500N bridges Hot-adding a stack of SAS disk shelves and bridges to a MetroCluster system Hot-adding a SAS disk shelf to a stack of SAS disk shelves Hot-removing storage from the MetroCluster configuration Hot-removing a direct-attached disk shelf Hot-removing a disk shelf that is attached to an FC-to-SAS bridge Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration Verifying the state of the MetroCluster configuration Zoning for the new controller ports when adding a controller module in a fabric-attached MetroCluster configuration Adding a new controller module to each cluster Preparing for the upgrade Preparing cluster ports on the existing controller module Preparing the netboot server Setting the HA mode on the existing controller module Shutting down the existing controller Installing and cabling the new controller Verifying and setting the HA state of the controller module and chassis Assigning disks to the new controller Netbooting and setting up ONTAP on the new controller module Mirroring the root aggregate on the new controller Configuring intercluster LIFs Creating a mirrored data aggregate on each node Ensuring redundant protection for the CRS metadata volume Installing licenses for the new controller module Creating unmirrored data aggregates Verifying storage failover on both controller modules and setting cluster HA Installing the firmware after adding a controller module Refreshing the MetroCluster configuration with new controllers Checking the MetroCluster configuration Verifying the configuration with the Config Advisor Verifying switchover, healing, and switchback Glossary of MetroCluster terms Copyright information Trademark information How to send comments about documentation and receive update notifications Index

5 5 Where to find MetroCluster documentation There are a number of documents that can help you configure, operate, and monitor a MetroCluster configuration. MetroCluster and ONTAP libraries Library NetApp Documentation: MetroCluster in clustered Data ONTAP NetApp Documentation: Clustered Data ONTAP Express Guides NetApp Documentation: ONTAP 9 Content All MetroCluster guides All ONTAP express guides All ONTAP guides MetroCluster and miscellaneous guides Guide NetApp Technical Report 4375: MetroCluster for Data ONTAP Version 8.3 Overview and Best Practices ONTAP 9.0 Fabric-attached MetroCluster Installation and Configuration Guide Content A technical overview of the MetroCluster configuration and operation. Best practices for MetroCluster configuration. Fabric-attached MetroCluster architecture Cabling the configuration Configuring the FC-to-SAS bridges Configuring the FC switches Configuring the MetroCluster in ONTAP ONTAP 9.0 Stretch MetroCluster Installation and Configuration Guide Stretch MetroCluster architecture Cabling the configuration Configuring the FC-to-SAS bridges Configuring the MetroCluster in ONTAP ONTAP 9.0 MetroCluster Management and Disaster Recovery Guide Understanding the MetroCluster configuration Switchover, healing and switchback Disaster recovery

6 6 MetroCluster Service Guide Guide ONTAP 9.0 Data Protection Guide Using SnapMirror and SnapVault Technology Content How mirrored aggregates work SyncMirror SnapMirror SnapVault NetApp Documentation: OnCommand Unified Manager Core Package (current releases) NetApp Documentation: OnCommand Performance Manager for Clustered Data ONTAP MetroCluster Tiebreaker Software Installation and Configuration Guide 7-Mode Transition Tool 2.3 Copy-Based Transition Guide Monitoring the MetroCluster configuration Monitoring MetroCluster performance Monitoring the MetroCluster configuration with the MetroCluster Tiebreaker software Transitioning data from 7-Mode storage systems to clustered storage systems

7 7 Where to find procedures for MetroCluster maintenance tasks You should be sure you select the correct procedure when you perform MetroCluster hardware maintenance tasks. Component Task Procedure Storage controller Disk shelf Data ONTAP software upgrade Hardware upgrade FRU replacement (including controllers, PCIe cards, FC- VI card, and so on) Moving a storage controller module or NVRAM card among the MetroCluster storage systems is not supported. Shelf addition (stack with bridges or individual shelf) Shelf removal Disk drive replacement Shelf replacement Firmware upgrade ONTAP 9.0 Upgrade and Revert/Downgrade Guide NetApp Documentation: FAS8000 Series NetApp Documentation: FAS6200 Series NetApp Documentation: FAS3200 Series NetApp Documentation: FAS8000 Series NetApp Documentation: FAS6200 Series NetApp Documentation: FAS3200 Series Hot-adding a stack of SAS disk shelves to an existing pair of FibreBridge 7500N bridges on page 82 Hot-adding a stack of SAS disk shelves and bridges to a MetroCluster system on page 88 Hot-adding a SAS disk shelf to a stack of SAS disk shelves on page 96 Hot-removing storage from the MetroCluster configuration on page 100 NetApp Documentation: Disk Shelves NetApp KB Article : How to Nondisruptively Power Cycle or Replace a Disk Shelf in a Metrocluster ONTAP 9.0 Upgrade and Revert/Downgrade Guide

8 8 MetroCluster Service Guide Component Task Procedure FC-to-SAS bridge Replacing a single bridge This procedure is for replacing a single failed bridge. Replacing FibreBridge 6500N bridges with a pair of FibreBridge 7500N bridges This procedure is for upgrading a bridge pair to FibreBridge 7500N bridges. The existing bridges can both be FibreBridge 6500N bridges or one FibreBridge 6500 bridge and one FibreBridge 7500N bridge. Firmware upgrade Replacing a failed power supply module Replacing a single FC-to-SAS bridge on page 11 Replacing a pair of FibreBridge 6500N bridges with 7500N bridges on page 21 Updating the firmware on a FibreBridge on page 10 Hot-replacing a failed power supply on page 35 FC switch Replacement Replacing a Brocade FC switch (MetroCluster) on page 63 Replacing a Cisco FC switch on page 67 Firmware upgrade (FC switch) Updating firmware on a Brocade FC switch on page 59 Updating firmware on a Cisco FC switch on page 61 SAS optical cables Replacement Nondisruptively replacing SAS cables on page 74 Cluster interconnect switches Replacement Firmware upgrade NetApp Documentation: Cluster Management and Interconnect Switches NetApp Documentation: Cluster Management and Interconnect Switches

9 9 Performing FC-to-SAS bridge maintenance If necessary, you can nondisruptively replace the FC-to-SAS bridges or upgrade their firmware in the MetroCluster configuration. Choices FibreBridge 7500N bridges in MetroCluster configurations on page 9 Updating the firmware on a FibreBridge bridge on page 10 Replacing a single FC-to-SAS bridge on page 11 Replacing a pair of FibreBridge 6500N bridges with 7500N bridges on page 21 Replacing SAS copper cables on page 39 Requirements for using other interfaces to configure and manage FibreBridge bridges on page 40 Hot-replacing a failed power supply module on page 41 Zoning for an eight- or a four-node MetroCluster configuration using FibreBridge 7500N bridges on page 41 Zoning for FibreBridge 6500N bridges in a four-node MetroCluster configuration on page 54 FibreBridge 7500N bridges in MetroCluster configurations The FibreBridge 7500N bridge is supported as a replacement for the FibreBridge 6500N bridge or for when adding new storage to the MetroCluster configuration. The supported configurations have zoning requirements and restrictions regarding use of the bridge's FC ports and stack and storage shelf limits. Use case Zoning changes needed? Restrictions Procedure Replacing a single 6500N bridge with a single 7500N bridge No The 7500N bridge must be configured exactly the same as the 6500N bridge, using a single FC port and attaching to a single stack. The second FC port on the 7500N must not be used. Hot-swapping a FibreBridge 6500N bridge with a FibreBridge 7500N bridge on page 17 Consolidating multiple stacks by replacing multiple pairs of 6500N bridges with a single pair of 7500N bridges Yes In this case, you take the 6500N bridges out of service and replace them with a single pair of 7500N bridges. Each pair of FibreBridge 7500N bridges can support up to four stacks and a maximum of forty storage shelves. Replacing a pair of FibreBridge 6500N bridges with 7500N bridges on page 21 At the end of the procedure, both the top and bottom of the stacks must be connected to corresponding ports on the 7500N bridges.

10 10 MetroCluster Service Guide Use case Zoning changes needed? Restrictions Procedure Adding new storage through adding a new pair of 7500N bridges Yes You must add storage zones for each of the FC ports of the new bridges. You must have available ports on the FC switch fabric (in a fabric-attached MetroCluster configuration) or on the storage controllers (in a stretch MetroCluster configuration). Each pair of FibreBridge 7500N bridges can support up to four stacks and a maximum of forty storage shelves. Hot-adding a stack of SAS disk shelves and bridges to a MetroCluster system on page 88 Updating the firmware on a FibreBridge bridge You might need to update the firmware on your model of the FibreBridge bridge to help you have the latest features and resolve possible issues. Before you begin The MetroCluster configuration must be operating normally. All FibreBridge bridges in the MetroCluster configuration must be up and operating. All storage paths must be available. About this task This task must be performed on each FibreBridge bridge in the MetroCluster configuration, so that all FibreBridge bridges are running the same firmware version. Steps 1. Go to the ATTO FibreBridge Description page. If you are using... FibreBridge 7500N Then.. NetApp Downloads: ATTO FibreBridge 7500N a. Go to b. Select ATTO FibreBridge 7500N Platform, and then click Go!. FibreBridge 6500N NetApp Downloads: ATTO FibreBridge 6500N 2. Using the link on the ATTO FibreBridge Description page, access the ATTO web site. If you are using... FibreBridge 7500N Then download... ATTO FibreBridge 7500N Installation and Operation Manual ATTO QuickNAV utility (to the computer you are using for setup) FibreBridge 6500N ATTO FibreBridge 6500N Installation and Operation Manual ATTO QuickNAV utility (to the computer you are using for setup)

11 Performing FC-to-SAS bridge maintenance Go to the ATTO FibreBridge Firmware Download page for your bridge model and do the following: a. Navigate to the ATTO FibreBridge Firmware Download page by clicking Continue at the end of the ATTO FibreBridge Description page. b. Download the bridge firmware file using Steps 1 through 3 of the procedure on that page. c. Make a copy of the ATTO FibreBridge Firmware Download page and release notes for reference when you are instructed to update the firmware on each bridge. 4. Update the bridges: a. Select a bridge and update the firmware to the latest version by following the instructions starting with Step 4 on the FibreBridge Download page. You must follow the steps to confirm that the bridge has successfully restarted and that the firmware has been updated. b. Repeat the procedure on the next bridge, until all bridges in the MetroCluster configuration have been updated. Replacing a single FC-to-SAS bridge You can replace a bridge with a same model bridge or with a new model bridge. This procedure can be done without disrupting the storage system when the system is cabled correctly for Multipath HA. About this task This procedure is written with the assumption that you are using the recommended bridge management interfaces: the ATTO ExpressNAV GUI and ATTO QuickNAV utility. You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge firmware. You use the ATTO QuickNAV utility to configure the bridge Ethernet management 1 port. You can use other management interfaces instead, if needed, such as a serial port or Telnet to configure and manage a bridge and to configure the Ethernet management 1 port, and FTP to update the bridge firmware. If you choose any of these management interfaces, your system must meet the applicable requirements in the Other FibreBridge management interfaces section. Choices Verifying storage connectivity on page 11 Hot-swapping the bridge with a replacement bridge of the same model on page 12 Hot-swapping a FibreBridge 6500N bridge with a FibreBridge 7500N bridge on page 17 Related tasks Replacing a pair of FibreBridge 6500N bridges with 7500N bridges on page 21 Verifying storage connectivity Before replacing bridges, you should verify bridge and storage connectivity. Familiarizing yourself with the command output enables you to subsequently confirm connectivity after making configuration changes. About this task You can issue these commands from the admin prompt of any of the controller modules in the MetroCluster configuration at the site undergoing maintenance.

12 12 MetroCluster Service Guide Step 1. Confirm connectivity to the bridge-connected disks: run local sysconfig -v The output shows the disks attached to the initiator ports on the controller, and identifies the shelves connected to the FC-to-SAS bridges: node_a_1> run local sysconfig -v NetApp Release 8.3.2X18: Sun Dec 13 01:23:24 PST 2015 System ID: (node_a_1); partner ID: (node_b_1) System Serial Number: (node_a_1) System Rev: 70 System Storage Configuration: Multi-Path HA<=== Configuration should be multi-path HA... slot 0: FC Host Adapter 0g (QLogic 8324 rev. 2, N-port, <UP>)<=== Initiator port Firmware rev: Flash rev: Host Port Id: 0x60100 FC Node Name: 5:00a:098201:bae312 FC Port Name: 5:00a:098201:bae312 SFP Vendor: FINISAR CORP. SFP Part Number: FTLF8529P3BCVAN1 SFP Serial Number: URQ0R1R SFP Capabilities: 4, 8 or 16 Gbit Link Data Rate: 16 Gbit Switch Port: brcd6505-fcs40:1 <List of disks visible to port> ID Vendor Model FW Size brcd6505-fcs40:12.126l1527 : NETAPP X302_HJUPI01TSSM NA GB ( B/sect) brcd6505-fcs40:12.126l1528 : NETAPP X302_HJUPI01TSSA NA GB ( B/sect)... <List of FC-to-SAS bridges visible to port> FC-to-SAS Bridge: brcd6505-fcs40:12.126l0 : ATTO FibreBridge6500N 1.61 FB6500N brcd6505-fcs42:13.126l0 : ATTO FibreBridge6500N 1.61 FB6500N brcd6505-fcs42:6.126l0 : ATTO FibreBridge6500N 1.61 FB6500N brcd6505-fcs42:7.126l0 : ATTO FibreBridge6500N 1.61 FB6500N <List of storage shelves visible to port> brcd6505-fcs40:12.shelf6: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: 0200 brcd6505-fcs40:12.shelf8: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: Hot-swapping the bridge with a replacement bridge of the same model You can hot-swap a failed bridge with the same model bridge. Steps 1. If the bridge is in a fabric-attached MetroCluster configuration, disable the switch port or ports that connects to the bridge FC port or ports. 2. Properly ground yourself. 3. Power off the failed bridge and disconnect it from the power source. 4. Disconnect the cables connected to the failed bridge. You should make note of the port that each cable was connected to. 5. Remove the failed bridge from the rack.

13 Performing FC-to-SAS bridge maintenance Install the new bridge into the rack. 7. Reconnect the power cord and shielded Ethernet cable. Attention: Do not reconnect the SAS or FC cables at this time. 8. Connect the bridge to a power source and turn it on. The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-on self test sequence. 9. Configure the Ethernet management 1 port for each bridge. If you are using... FibreBridge 7500N FibreBridge 6500N Then... Follow the procedure in the ATTO FibreBridge 7500N Installation and Operation Manual, section 2.0. Follow the procedure in the ATTO FibreBridge 6500N Installation and Operation Manual, section 2.0. Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that is connected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2 port, you would need to connect the shielded Ethernet cable to port 2 and run QuickNAV. 10. Configure the bridges. Be sure to make note of the user name and password that you designate. The ATTO FibreBridge Installation and Operation Manual for your bridge model has the most current information on available commands and how to use them. a. Configure the IP settings of the bridge. To set the IP address without the Quicknav utility, you need to have a serial connection to the FibreBridge. If using the CLI, issue the following commands: set ipaddress mp1 ip-address set ipsubnetmask mp1 subnet-mask set ipgateway mp1 x.x.x.x set ipdhcp mp1 b. Configure the bridge name. The bridges should each have a unique name within the MetroCluster configuration. bridge names for one stack group on each site: bridge_a_1a bridge_a_1b bridge_b_1a bridge_b_1b If using the CLI, issue the following command: set bridgename bridgename

14 14 MetroCluster Service Guide c. Enable SNMP on the bridge: set SNMP enabled 11. Configure the bridge FC ports. a. Configure the data rate/speed of the bridge FC ports. Note: Set the FCDataRate speed to the maximum speed supported by the FC port of the FC switch or the controller module to which the bridge port connects. The most current information on supported distance can be found in the Interoperability Matrix. NetApp Interoperability Matrix Tool After opening the Interoperability Matrix, you can use the Storage Solution field to select your MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray). You use the Component Explorer to select the components and ONTAP version to refine your search. You can click Show Results to display the list of supported configurations that match the criteria. If using the CLI, issue the following command: set FCDataRate port-number 16Gb b. Configure the connection mode that the port uses to communicate across the FC network. If you have a fabric-attached MetroCluster system, you must set the bridge connection mode to ptp (point-to-point). If you have a stretch MetroCluster system, you must set the bridge connection mode depending on the adapter that the bridge connects to: For 16-Gbps capable adapters, set the connection mode to ptp, even if it is operating at a lower speed. For 8-Gbps and 4-Gbps capable adapters, set the connection mode to loop. If using the CLI, issue the following command: set FCConnMode port-number ptp-or-loop c. If you are configuring an ATTO 7500N bridge and using the second port, repeat the previous substeps for the FC2 port. 12. Save the bridge's configuration. If using the CLI, issue the following command: SaveConfiguration Restart You are prompted to restart the bridge. 13. After completing Steps 1 through 3, update the firmware on the bridge to the latest version by following the instructions starting with Step 4 on the FibreBridge Download page for your bridge model. 14. Reconnect the SAS and FC cables to the same ports on the new bridge. When replacing a FibreBridge 7500N bridge, SAS ports B, C, and D are enabled by default. You must disable the SAS ports that are not being used.

15 Performing FC-to-SAS bridge maintenance 15 After checking on the SAS ports, verify that the FC2 port is disabled in the FibreBridge 7500N bridge. Having the FC2 port enabled displays an error in the health monitoring setup. Note: Do not force a connector into a port. Wait at least 10 seconds before connecting the port. The SAS cable connectors are keyed; when oriented correctly into a SAS port, the connector clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector). For controllers, the orientation of SAS ports can vary depending on the platform model; therefore, the correct orientation of the SAS cable connector varies. 15. Verify that each bridge can see all disk drives and disk shelves it is connected to. If you are using the... ATTO ExpressNAV GUI Then... a. In a supported web browser, enter the IP address of the bridge in the browser box. You are brought to the ATTO FibreBridge homepage, which has a link. b. Click the link, and then enter your user name and the password that you designated when you configured the bridge. The ATTO FibreBridge status page appears with a menu to the left. c. Click Advanced in the menu. d. Enter the following command, and then click Submit to implement the changes: sastargets Serial port connection Enter the following command: sastargets The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentially numbered so that you can quickly count the devices. For example, the following output shows that 10 disks are connected: Tgt VendorID ProductID Type SerialNumber 0 NETAPP X410_S15K6288A15 DISK 3QP1CLE UHJV 1 NETAPP X410_S15K6288A15 DISK 3QP1ELF V1BV 2 NETAPP X410_S15K6288A15 DISK 3QP1G3EW U2M0 3 NETAPP X410_S15K6288A15 DISK 3QP1EWMP U1X5 4 NETAPP X410_S15K6288A15 DISK 3QP1FZLE G8YU 5 NETAPP X410_S15K6288A15 DISK 3QP1FZLF TZKZ 6 NETAPP X410_S15K6288A15 DISK 3QP1CEB MGXL 7 NETAPP X410_S15K6288A15 DISK 3QP1G7A FNTT 8 NETAPP X410_S15K6288A15 DISK 3QP1FY0T G8PA 9 NETAPP X410_S15K6288A15 DISK 3QP1FXW VERQ Note: If the text response truncated appears at the beginning of the output, you can use Telnet to access the bridge and enter the same command to see all of the output. 16. Verify that the command output shows that the bridge is connected to all of the necessary disks and disk shelves in the stack. If the output is... Correct Then... Repeat Step 12 for each remaining bridge.

16 16 MetroCluster Service Guide If the output is... Not correct Then... a. Check for loose SAS cables or correct the SAS cabling. b. Repeat Step If you are using a FibreBridge 6500N or a FibreBridge 7500N bridge, from the FibreBridge serial port console interface: a. Disable the SNMP setting: set SNMP disabled b. Save the configuration and restart the bridge: SaveConfiguration restart 18. Reenable the FC switch port you disabled in Step 1 on page 12that connects to the bridge. 19. Verify that all controllers have access through the new bridge to the disk shelves (that is, that the system is cabled for Multipath HA) from the system console of both controllers: run local sysconfig Note: It might take up to a minute for the system to complete discovery. For example, the following output states that the system is cabled for Multipath HA: NetApp Release 8.3.2: Tue Jan 26 01:41:49 PDT 2016 System ID: (node_a_1); partner ID: (node_a_2) System Serial Number: (node_a_1); partner Serial Number: (node_a_2) System Rev: B0 System Storage Configuration: Multi-Path HA System ACP Connectivity: NA If the output does note indicate Multipath HA, ou must correct the SAS and FC cabling because not all disk drives are accessible through the new bridge. Attention: When the system is not cabled as Multipath HA, restarting a bridge might cause loss of access to the disk drives and result in a multi-disk panic. 20. Verify that the bridge is configured for SNMP in the Remote Management page of the ExpressNAV interface. 21. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/.

17 Performing FC-to-SAS bridge maintenance 17 After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered. Hot-swapping a FibreBridge 6500N bridge with a FibreBridge 7500N bridge You can hot-swap a FibreBridge 6500N bridge with a FibreBridge 7500N bridge to replace a failed bridge or upgrade your bridge in a fabric-attached or a bridge-attached MetroCluster configuration. About this task This procedure is for hot-swapping a single FibreBridge 6500N bridge with single FibreBridge 7500N bridge. When you hot-swap a FibreBridge 6500N bridge with a FibreBridge 7500N bridge, you can use 1 FC port and 1 SAS port. By replacing both FibreBridge 6500N bridges on the bridge, you can take advantage of the additional ports on the FibreBridge 7500N bridge. Attention: If you are hot-swapping both FibreBridge 6500N bridges in a pair you must use the Consolidate Multiple Storage Stacks on page 21 procedure for zoning instructions. Steps 1. Do one of the following: If the bridge is in a fabric-attached MetroCluster configuration, disable the switch port or ports that connects to the bridge FC port or ports. If the bridge is in a stretch Metrocluster configuration, use either one of the available FC ports. 2. Properly ground yourself. 3. Power off the FibreBridge 6500N bridge. 4. Disconnect the cables connected from the shelf to the FibreBridge 6500N bridge ports and power cables. You should make note of the ports that each cable was connected to. 5. Remove the FibreBridge 6500N bridge that you need to replace from the rack. 6. Install the new FibreBridge 7500N bridge into the rack. 7. Reconnect the power cord and Ethernet cable. Attention: Do not reconnect the SAS or FC cables at this time. 8. Connect the bridge to a power source and turn it on. The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-on self test sequence. 9. Configure the Ethernet management 1 port for each bridge by following the procedure in the ATTO FibreBridge 7500N Installation and Operation Manual, section 2.0. Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that is connected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2 port, you would need to connect the Ethernet cable to port 2 and run QuickNAV. 10. Configure the bridges. Be sure to make note of the user name and password that you designate.

18 18 MetroCluster Service Guide The ATTO FibreBridge Installation and Operation Manual for your bridge model has the most current information on available commands and how to use them. a. Configure the IP settings of the bridge. To set the IP address without the Quicknav utility, you need to have a serial connection to the FibreBridge. If using the CLI, issue the following commands: set ipaddress mp1 ip-address set ipsubnetmask mp1 subnet-mask set ipgateway mp1 x.x.x.x set ipdhcp mp1 b. Configure the bridge name. The bridges should each have a unique name within the MetroCluster configuration. bridge names for one stack group on each site: bridge_a_1a bridge_a_1b bridge_b_1a bridge_b_1b If using the CLI, issue the following command: set bridgename bridgename c. Enable SNMP on the bridge: set SNMP enabled 11. Configure the bridge FC ports. a. Configure the data rate/speed of the bridge FC ports. Note: Set the FCDataRate speed to the maximum speed supported by the FC port of the FC switch or the controller module to which the bridge port connects. The most current information on supported distance can be found in the Interoperability Matrix. NetApp Interoperability Matrix Tool After opening the Interoperability Matrix, you can use the Storage Solution field to select your MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray). You use the Component Explorer to select the components and ONTAP version to refine your search. You can click Show Results to display the list of supported configurations that match the criteria. If using the CLI, issue the following command: set FCDataRate port-number 16Gb b. Configure the connection mode that the port uses to communicate across the FC network.

19 Performing FC-to-SAS bridge maintenance 19 If you have a fabric-attached MetroCluster system, you must set the bridge connection mode to ptp (point-to-point). If you have a stretch MetroCluster system, you must set the bridge connection mode depending on the adapter that the bridge connects to: For 16-Gbps capable adapters, set the connection mode to ptp, even if it is operating at a lower speed. For 8-Gbps and 4-Gbps capable adapters, set the connection mode to loop. If using the CLI, issue the following command: set FCConnMode port-number ptp-or-loop c. If you are configuring an ATTO 7500N bridge and using the second port, repeat the previous substeps for the FC2 port. 12. Save the bridge's configuration. If using the CLI, issue the following command: SaveConfiguration Restart You are prompted to restart the bridge. 13. Turn on the Health Monitoring for FibreBridge 7500N bridges. Note: If you have both FibreBridge 6500N bridges and FibreBridge 7500N bridges, health monitoring should not be configured for the FibreBridge 6500N bridges. 14. Update the firmware on the bridge to the latest version by following the instructions starting with Step 4 on the FibreBridge Download page. 15. Reconnect the SAS and FC cables to the same ports on the new bridge. Note: Do not force a connector into a port. The mini-sas cables are keyed; when oriented correctly into a SAS port, the SAS cable clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector). For controllers, the orientation of SAS ports can vary depending on the platform model; therefore, the correct orientation of the SAS cable connector varies. a. Cable FC port 1 of the bridge to the same speed FC port on the controller at cluster_a. Note: The Brocade and Cisco switches use different port numbering: On Brocade switches, the first port is numbered 0. On Cisco switches, the first port is numbered 1. This is reflected in the following tables: Cable port FC 1 on this bridge... To this port on FC_switch_x_1... Brocade 6505 Brocade 6510 Cisco bridge_a_1_port number bridge_a_1_port number 7 7 8

20 20 MetroCluster Service Guide Cable port FC 1 on this bridge... To this port on FC_switch_x_1... Brocade 6505 Brocade 6510 Cisco bridge_a_1_port number bridge_a_1_port number b. Repeat the previous step on the other bridges until all bridges have been cabled. 16. Verify that each bridge can see all disk drives and disk shelves it is connected to. If you are using the... ATTO ExpressNAV GUI Then... a. In a supported web browser, enter the IP address of the bridge in the browser box. You are brought to the ATTO FibreBridge homepage, which has a link. b. Click the link, and then enter your user name and the password that you designated when you configured the bridge. The ATTO FibreBridge status page appears with a menu to the left. c. Click Advanced in the menu. d. Enter the following command and then click Submit to see the list of disks visible to the bridge: sastargets Serial port connection Display the list of disks visible to the bridge: sastargets The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentially numbered so that you can quickly count the devices. For example, the following output shows that 10 disks are connected: Tgt VendorID ProductID Type SerialNumber 0 NETAPP X410_S15K6288A15 DISK 3QP1CLE UHJV 1 NETAPP X410_S15K6288A15 DISK 3QP1ELF V1BV 2 NETAPP X410_S15K6288A15 DISK 3QP1G3EW U2M0 3 NETAPP X410_S15K6288A15 DISK 3QP1EWMP U1X5 4 NETAPP X410_S15K6288A15 DISK 3QP1FZLE G8YU 5 NETAPP X410_S15K6288A15 DISK 3QP1FZLF TZKZ 6 NETAPP X410_S15K6288A15 DISK 3QP1CEB MGXL 7 NETAPP X410_S15K6288A15 DISK 3QP1G7A FNTT 8 NETAPP X410_S15K6288A15 DISK 3QP1FY0T G8PA 9 NETAPP X410_S15K6288A15 DISK 3QP1FXW VERQ Note: If the text response truncated appears at the beginning of the output, you can use Telnet to access the bridge and enter the same command to see all of the output. 17. Verify that the command output shows that the bridge is connected to all of the necessary disks and disk shelves in the stack. If the output is... Correct Then... Repeat Step 11 on page 18 for each remaining bridge.

21 Performing FC-to-SAS bridge maintenance 21 If the output is... Not correct Then... a. Check for loose SAS cables or correct the SAS cabling by repeating Step 10 on page 17. b. Repeat Step 11 on page Reenable the FC switch port that connects to the bridge. 19. Verify that all controllers have access through the new bridge to the disk shelves (that the system is cabled for Multipath HA), by entering the following command at the system console of both controllers: run local sysconfig Note: It might take up to a minute for the system to complete discovery. For example, the following output states that the system is cabled for Multipath HA: NetApp Release 8.3.2: Tue Jan 26 01:23:24 PST 2016 System ID: (node_a_1); partner ID: (node_a_2) System Serial Number: (node_a_1); partner Serial Number: (node_a_2) System Rev: B0 System Storage Configuration: Multi-Path HA System ACP Connectivity: NA If the command output indicates that the configuration is mixed-path or single-path HA, you must correct the SAS and FC cabling because not all disk drives are accessible through the new bridge. Attention: When the system is not cabled as Multipath HA, restarting a bridge might cause loss of access to the disk drives and result in a multi-disk panic. 20. Return the failed part to NetApp as described in the RMA instructions shipped with the kit. Contact technical support at NetApp Support, (North America), (Europe), or (Asia/Pacific) if you need the RMA number or additional help with the replacement procedure. Replacing a pair of FibreBridge 6500N bridges with 7500N bridges To take advantage of the additional FC2 port on the FibreBridge 7500N bridges and reduce rack utilization, you can replace 6500N bridges and consolidate up to four storage stacks behind a single pair of FibreBridge 7500N bridges. FibreBridge 6500N bridges supported only a single stack per pair of bridges. This replacement and consolidation can be performed non-disruptively. About this task You should use this procedure if: You are replacing a pair of FibreBridge 6500N bridges with FibreBridge 7500N bridges. You previously replaced a single FibreBridge 6500N bridge with a 7500N bridge and are now replacing the second bridge in the pair. You have a pair of FibreBridge 7500N bridges with available SAS ports and you are consolidating SAS storage stacks that are currently connected using FibreBridge 6500N bridges.

22 22 MetroCluster Service Guide Steps 1. Verifying storage connectivity on page Hot-swapping FibreBridge 6500N bridges to create a pair of FibreBridge 7500N bridges on page Cabling the bridge SAS ports when consolidating storage behind FibreBridge 7500N bridges on page Updating zoning when adding FibreBridge 7500N bridges to a configuration on page Cabling the second bridge FC port when adding FibreBridge 7500N bridges to a configuration on page Disabling unused FC-to-SAS bridge ports on page 38 Related tasks Replacing a single FC-to-SAS bridge on page 11 Verifying storage connectivity Before replacing bridges, you should verify bridge and storage connectivity. Familiarizing yourself with the command output enables you to subsequently confirm connectivity after making configuration changes. About this task You can issue these commands from the admin prompt of any of the controller modules in the MetroCluster configuration at the site undergoing maintenance. Step 1. Confirm connectivity to the bridge-connected disks: run local sysconfig -v The output shows the disks attached to the initiator ports on the controller, and identifies the shelves connected to the FC-to-SAS bridges: node_a_1> run local sysconfig -v NetApp Release 8.3.2X18: Sun Dec 13 01:23:24 PST 2015 System ID: (node_a_1); partner ID: (node_b_1) System Serial Number: (node_a_1) System Rev: 70 System Storage Configuration: Multi-Path HA<=== Configuration should be multi-path HA... slot 0: FC Host Adapter 0g (QLogic 8324 rev. 2, N-port, <UP>)<=== Initiator port Firmware rev: Flash rev: Host Port Id: 0x60100 FC Node Name: 5:00a:098201:bae312 FC Port Name: 5:00a:098201:bae312 SFP Vendor: FINISAR CORP. SFP Part Number: FTLF8529P3BCVAN1 SFP Serial Number: URQ0R1R SFP Capabilities: 4, 8 or 16 Gbit Link Data Rate: 16 Gbit Switch Port: brcd6505-fcs40:1 <List of disks visible to port> ID Vendor Model FW Size brcd6505-fcs40:12.126l1527 : NETAPP X302_HJUPI01TSSM NA GB ( B/sect) brcd6505-fcs40:12.126l1528 : NETAPP X302_HJUPI01TSSA NA GB ( B/sect)... <List of FC-to-SAS bridges visible to port> FC-to-SAS Bridge:

23 Performing FC-to-SAS bridge maintenance 23 brcd6505-fcs40:12.126l0 : ATTO FibreBridge6500N 1.61 FB6500N brcd6505-fcs42:13.126l0 : ATTO FibreBridge6500N 1.61 FB6500N brcd6505-fcs42:6.126l0 : ATTO FibreBridge6500N 1.61 FB6500N brcd6505-fcs42:7.126l0 : ATTO FibreBridge6500N 1.61 FB6500N <List of storage shelves visible to port> brcd6505-fcs40:12.shelf6: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: 0200 brcd6505-fcs40:12.shelf8: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: Hot-swapping FibreBridge 6500N bridges to create a pair of FibreBridge 7500N bridges To hot-swap one or two FibreBridge 6500N bridges to create a configuration with a pair of FibreBridge 7500N bridges, you must replace the bridges one at a time and follow the correct cabling. The new cabling is different from the original cabling. About this task You can also use this procedure if: You are replacing a pair of FibreBridge 6500N bridges that are both connected to the same stack of SAS storage. You previously replaced one FibreBridge 6500N bridge in the pair, and your storage stack is configured with one FibreBridge 6500N bridge and one FibreBridge 7500N bridge. In this case, you should begin with Step 3 on page 26. The following diagram shows an example of the initial configuration, in which four FibreBridge 6500N bridges are connecting two SAS storage stacks:

24 24 MetroCluster Service Guide site_a FC_switch_A_1 site_b FC_switch_A_2 FC1 bridge_a_1a (6500N) FC1 bridge_a_3a (6500N) SAS A SAS A SAS stack SAS stack SAS A SAS A FC1 bridge_a_1b (6500N) FC1 bridge_a_3b (6500N) Steps 1. Using the following guidelines, hot-swap the top FibreBridge 6500N bridge with a FibreBridge 7500N bridge using the procedure in Hot-swapping a FibreBridge 6500N bridge with a FibreBridge 7500N bridge on page 17: Connect the FibreBridge 7500N bridge FC1 port to the switch or controller. This is the same connection that was made to the FibreBridge 6500N bridge FC1 port. Do not connect the FibreBridge 7500N bridge FC2 port at this time. The following diagram shows that bridge_a_1a has been replaced and is now a FibreBridge 7500N bridge:

25 Performing FC-to-SAS bridge maintenance 25 site_a FC_switch_A_1 site_b FC_switch_A_2 FC1 FC2 bridge_a_1a (7500N) FC1 bridge_a_3a (6500N) SAS A SAS B SAS C SAS D SAS A SAS stack SAS stack SAS A SAS A FC1 bridge_a_1b (6500N) FC1 bridge_a_3b (6500N) 2. Confirm connectivity to the bridge-connected disks and that the new FibreBridge 7500N is visible in the confiugration: run local sysconfig -v node_a_1> run local sysconfig -v NetApp Release 8.3.2X18: Sun Dec 13 01:23:24 PST 2015 System ID: (node_a_1); partner ID: (node_b_1) System Serial Number: (node_a_1) System Rev: 70 System Storage Configuration: Multi-Path HA<=== Configuration should be multi-path HA... slot 0: FC Host Adapter 0g (QLogic 8324 rev. 2, N-port, <UP>)<=== Initiator port Firmware rev: Flash rev: Host Port Id: 0x60100

26 26 MetroCluster Service Guide FC Node Name: 5:00a:098201:bae312 FC Port Name: 5:00a:098201:bae312 SFP Vendor: FINISAR CORP. SFP Part Number: FTLF8529P3BCVAN1 SFP Serial Number: URQ0R1R SFP Capabilities: 4, 8 or 16 Gbit Link Data Rate: 16 Gbit Switch Port: brcd6505-fcs40:1 <List of disks visible to port> ID Vendor Model FW Size brcd6505-fcs40:12.126l1527 : NETAPP X302_HJUPI01TSSM NA GB ( B/sect) brcd6505-fcs40:12.126l1528 : NETAPP X302_HJUPI01TSSA NA GB ( B/sect)... <List of FC-to-SAS bridges visible to port> FC-to-SAS Bridge: brcd6505-fcs40:12.126l0 : ATTO FibreBridge7500N A30H FB7500N100104<=== brcd6505-fcs42:13.126l0 : ATTO FibreBridge6500N 1.61 FB6500N brcd6505-fcs42:6.126l0 : ATTO FibreBridge6500N 1.61 FB6500N brcd6505-fcs42:7.126l0 : ATTO FibreBridge6500N 1.61 FB6500N <List of storage shelves visible to port> brcd6505-fcs40:12.shelf6: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: 0200 brcd6505-fcs40:12.shelf8: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: Using the following guidelines, hot-swap the bottom FibreBridge 6500N bridge with a FibreBridge 7500N bridge using the procedure in Hot-swapping a FibreBridge 6500N bridge with a FibreBridge 7500N bridge on page 17: Connect the FibreBridge 7500N bridge FC2 port to the switch or controller. This is the same connection that was made to the FibreBridge 6500N bridge FC1 port. Do not connect the FibreBridge 7500N bridge FC1 port at this time.

27 Performing FC-to-SAS bridge maintenance 27 site_a FC_switch_A_1 site_b FC_switch_A_2 FC1 FC2 bridge_a_1a (7500N) FC1 bridge_a_3a (6500N) SAS A SAS B SAS C SAS D SAS A SAS stack SAS stack SAS A SAS B SAS C SAS D SAS A FC1 FC2 bridge_a_1b (7500N) FC1 bridge_a_3b (6500N) 4. Confirm connectivity to the bridge-connected disks: run local sysconfig -v The output shows the disks attached to the initiator ports on the controller, and identifies the shelves connected to the FC-to-SAS bridges: node_a_1> run local sysconfig -v NetApp Release 8.3.2X18: Sun Dec 13 01:23:24 PST 2015 System ID: (node_a_1); partner ID: (node_b_1) System Serial Number: (node_a_1) System Rev: 70 System Storage Configuration: Multi-Path HA<=== Configuration should be multi-path HA... slot 0: FC Host Adapter 0g (QLogic 8324 rev. 2, N-port, <UP>)<=== Initiator port Firmware rev: Flash rev: Host Port Id: 0x60100

28 28 MetroCluster Service Guide FC Node Name: 5:00a:098201:bae312 FC Port Name: 5:00a:098201:bae312 SFP Vendor: FINISAR CORP. SFP Part Number: FTLF8529P3BCVAN1 SFP Serial Number: URQ0R1R SFP Capabilities: 4, 8 or 16 Gbit Link Data Rate: 16 Gbit Switch Port: brcd6505-fcs40:1 <List of disks visible to port> ID Vendor Model FW Size brcd6505-fcs40:12.126l1527 : NETAPP X302_HJUPI01TSSM NA GB ( B/sect) brcd6505-fcs40:12.126l1528 : NETAPP X302_HJUPI01TSSA NA GB ( B/sect)... <List of FC-to-SAS bridges visible to port> FC-to-SAS Bridge: brcd6505-fcs40:12.126l0 : ATTO FibreBridge7500N A30H FB7500N brcd6505-fcs42:13.126l0 : ATTO FibreBridge7500N A30H FB7500N <List of storage shelves visible to port> brcd6505-fcs40:12.shelf6: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: 0200 brcd6505-fcs40:12.shelf8: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: Cabling the bridge SAS ports when consolidating storage behind FibreBridge 7500N bridges When consolidating multiple SAS storage stacks behind a single pair of FibreBridge 7500N bridges with available SAS ports, you must move the top and bottom SAS cables to the new bridges. About this task The FibreBridge 6500N bridge SAS ports use QSFP connectors. The FibreBridge 7500N bridge SAS ports use mini-sas connectors. Attention: When you remove a cable from a SAS port, you must wait at least 70 seconds before plugging the cable into a different SAS port. If you fail to do so, the system will not recognize that the cable has been moved to another port. Note: Do not force a connector into a port. Wait at least 10 seconds before connecting the port. The SAS cable connectors are keyed; when oriented correctly into a SAS port, the connector clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector). Steps 1. Remove the cable that connects the SAS A port of the top FibreBridge 6500N bridge to the top SAS shelf, being sure to note the SAS port on the storage shelf to which it connects. This cable is shown in blue in the following example:

29 Performing FC-to-SAS bridge maintenance 29 FC1 FC2 bridge_a_1a (7500N) FC1 bridge_a_3a (6500N) SAS A SAS B SAS C SAS D SAS A SAS stack SAS stack SAS A SAS B SAS C SAS D SAS A FC1 FC2 bridge_a_1b (7500N) FC1 bridge_a_3b (6500N) 2. Using a cable with a mini-sas connector, connect the same SAS port on the storage shelf to the SAS B port of the top FibreBridge 7500N bridge. This cable is shown in blue in the following example: FC1 FC2 bridge_a_1a (7500N) FC1 bridge_a_3a (6500N) SAS A SAS B SAS C SAS D SAS A SAS stack SAS stack SAS A SAS B SAS C SAS D SAS A FC1 FC2 bridge_a_1b (7500N) FC1 bridge_a_3b (6500N) 3. Remove the cable that connects the SAS A port of the bottom FibreBridge 6500N bridge to the top SAS shelf, being sure to note the SAS port on the storage shelf to which it connects. This cable is shown in green in the following example:

30 30 MetroCluster Service Guide FC1 FC2 bridge_a_1a (7500N) FC1 bridge_a_3a (6500N) SAS A SAS B SAS C SAS D SAS A SAS stack SAS stack SAS A SAS B SAS C SAS D SAS A FC1 FC2 bridge_a_1b (7500N) FC1 bridge_a_3b (6500N) 4. Using a cable with a mini-sas connector, connect the same SAS port on the storage shelf to the SAS B port of the bottom FibreBridge 7500N bridge. This cable is shown in green in the following example: FC1 FC2 bridge_a_1a (7500N) FC1 bridge_a_3a (6500N) SAS A SAS B SAS C SAS D SAS A SAS stack SAS stack SAS A SAS B SAS C SAS D SAS A FC1 FC2 bridge_a_1b (7500N) FC1 bridge_a_3b (6500N) 5. Confirm connectivity to the bridge-connected disks:

31 Performing FC-to-SAS bridge maintenance 31 run local sysconfig -v The output shows the disks attached to the initiator ports on the controller, and identifies the shelves connected to the FC-to-SAS bridges: node_a_1> run local sysconfig -v NetApp Release 8.3.2X18: Sun Dec 13 01:23:24 PST 2015 System ID: (node_a_1); partner ID: (node_b_1) System Serial Number: (node_a_1) System Rev: 70 System Storage Configuration: Multi-Path HA<=== Configuration should be multi-path HA... slot 0: FC Host Adapter 0g (QLogic 8324 rev. 2, N-port, <UP>)<=== Initiator port Firmware rev: Flash rev: Host Port Id: 0x60100 FC Node Name: 5:00a:098201:bae312 FC Port Name: 5:00a:098201:bae312 SFP Vendor: FINISAR CORP. SFP Part Number: FTLF8529P3BCVAN1 SFP Serial Number: URQ0R1R SFP Capabilities: 4, 8 or 16 Gbit Link Data Rate: 16 Gbit Switch Port: brcd6505-fcs40:1 <List of disks visible to port> ID Vendor Model FW Size brcd6505-fcs40:12.126l1527 : NETAPP X302_HJUPI01TSSM NA GB ( B/sect) brcd6505-fcs40:12.126l1528 : NETAPP X302_HJUPI01TSSA NA GB ( B/sect)... <List of FC-to-SAS bridges visible to port> FC-to-SAS Bridge: brcd6505-fcs40:12.126l0 : ATTO FibreBridge7500N A30H FB7500N brcd6505-fcs42:13.126l0 : ATTO FibreBridge7500N A30H FB7500N <List of storage shelves visible to port> brcd6505-fcs40:12.shelf6: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: 0200 brcd6505-fcs40:12.shelf8: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: Remove the old FibreBridge 6500N bridges that are no longer connected to the SAS storage. 7. If necessary, repeat the preceding steps to move up to two additional SAS stacks behind the new FibreBridge 7500N bridges, using SAS ports C and then D. Each SAS stack must be connected to the same SAS port on the top and bottom bridge. For example, if the top connection of the stack is connected to the top bridge SAS B port, the bottom connection must be connected to the SAS B port of the bottom bridge.

32 32 MetroCluster Service Guide FC1 FC2 bridge_a_1a (7500N) SAS A SAS B SAS C SAS D SAS stack SAS stack SAS stack SAS stack SAS A SAS B SAS C SAS D FC1 FC2 bridge_a_1b (7500N) Updating zoning when adding FibreBridge 7500N bridges to a configuration The zoning must be changed when you are replacing FibreBridge 6500N bridges with FibreBridge 7500N bridges and using both FC ports on the FibreBridge 7500N bridges. Each zone can have no more than four initiator ports. About this task The zoning changes are required to avoid issues with Data ONTAP, which requires that no more than four FC initiator ports can have a path to a disk. After recabling to consolidate the shelves, the existing zoning would result in each disk being reachable by eight FC ports. You must change the zoning to reduce the initiator ports in each zone to four. The following diagram shows the zoning on site_a before the changes:

33 Performing FC-to-SAS bridge maintenance 33 site_a fc-vi a fc-vi b fc-vi a fc-vi b controller_a_1 controller_a_2 0a 0b 0c 0d 0a 0b 0c 0d FC_switch_A_1 site_b FC_switch_A_2 FC1 bridge_a_1a (6500N) FC1 bridge_a_3a (6500N) SAS A SAS A SAS stack SAS stack SAS A SAS A FC1 bridge_a_1b (6500N) FC1 bridge_a_3b (6500N)

34 34 MetroCluster Service Guide Step 1. Update the storage zones for the FC switches by removing half of the initiator ports from each existing zone and creating new zones for the FibreBridge 7500N FC2 ports. The zones for the new FC2 ports will contain the initiator ports removed from the existing zones. In the diagrams, these zones are shown with dashed lines. For details about the zoning commands, see the FC switch sections of the ONTAP 9.0 Fabricattached MetroCluster Installation and Configuration Guide or ONTAP 9.0 Stretch MetroCluster Installation and Configuration Guide. The following examples show the storage zones and the ports in each zone before and after the consolidation. The ports are identified by domain, port pairs. Domain 5 consists of switch FC_switch_A_1. Domain 6 consists of switch FC_switch_A_2. Domain 7 consists of switch FC_switch_B_1. Domain 8 consists of switch FC_switch_B_2. Before or after consolidation Zone Domains and ports Colors in diagram* Zones before the consolidation. STOR_A_1a- FC1 5,1; 5,2; 5,4; 5,5; 7,1; 7,2; 7,4; 7,5; 5,6 Purple + dashed purple + blue There is a zone for each FC port on the four FibreBridge 6500N bridges. STOR_A_1b- FC1 STOR_A_2a- FC1 6,1; 6,2; 6,4; 6,5; 8,1; 8,2; 8,4; 8,5; 6,6 5,1; 5,2; 5,4; 5,5; 7,1; 7,2; 7,4; 7,5; 5,7 Brown + dashed brown + green Purple + dashed purple + red STOR_A_2b- FC1 6,1; 6,2; 6,4; 6,5; 8,1; 8,2; 8,4; 8,5; 6,7 Brown + dashed brown + orange Zones after the consolidation. There is a zone for each FC port on the two FibreBridge 7500N bridges. STOR_A_1a- FC1 STOR_A_1b- FC1 STOR_A_1a- FC2 STOR_A_1b- FC2 5,1; 5,4; 5,6; 7,1; 7,4 Purple + blue 5,2; 5,5; 5,7; 7,2; 7,5 Dashed purple + red 6,1; 6,4; 6,6; 8,1; 8,4 Brown + green 6,2; 6,5; 6,7; 8,2; 8,5 Dashed brown + orange * The diagrams only show site_a. The following diagram shows zoning at site_a after the consolidation:

35 Performing FC-to-SAS bridge maintenance 35 site_a fc-vi a fc-vi b fc-vi a fc-vi b controller_a_1 controller_a_2 0a 0b 0c 0d 0a 0b 0c 0d FC_switch_A_1 site_b FC_switch_A_2 FC1 FC2 bridge_a_1a SAS storage FC1 FC2 bridge_a_1b

36 36 MetroCluster Service Guide Cabling the second bridge FC port when adding FibreBridge 7500N bridges to a configuration To provide multiple paths to the storage stacks, you can cable the second FC port on each FibreBridge 7500N bridge when you have added the FibreBridge 7500N bridge to your configuration. Before you begin The zoning must have been adjusted to provide zones for the second FC ports. Steps 1. Cable the FC2 port of the top bridge to the correct port on FC_switch_A_2. site_a FC_switch_A_1 site_b FC_switch_A_2 FC1 FC2 bridge_a_1a (7500N) SAS A SAS B SAS C SAS D SAS stack SAS stack SAS A SAS B SAS C SAS D FC1 FC2 bridge_a_1b (7500N)

37 Performing FC-to-SAS bridge maintenance Cable the FC1 port of the bottom bridge to the correct port on FC_switch_A_1. site_a FC_switch_A_1 site_b FC_switch_A_2 FC1 FC2 bridge_a_1a (7500N) SAS A SAS B SAS C SAS D SAS stack SAS stack SAS A SAS B SAS C SAS D FC1 FC2 bridge_a_1b (7500N) 3. Confirm connectivity to the bridge-connected disks: run local sysconfig -v The output shows the disks attached to the initiator ports on the controller, and identifies the shelves connected to the FC-to-SAS bridges: node_a_1> run local sysconfig -v NetApp Release 8.3.2X18: Sun Dec 13 01:23:24 PST 2015 System ID: (node_a_1); partner ID: (node_b_1) System Serial Number: (node_a_1) System Rev: 70 System Storage Configuration: Multi-Path HA<=== Configuration should be multi-path HA..

38 38 MetroCluster Service Guide. slot 0: FC Host Adapter 0g (QLogic 8324 rev. 2, N-port, <UP>)<=== Initiator port Firmware rev: Flash rev: Host Port Id: 0x60100 FC Node Name: 5:00a:098201:bae312 FC Port Name: 5:00a:098201:bae312 SFP Vendor: FINISAR CORP. SFP Part Number: FTLF8529P3BCVAN1 SFP Serial Number: URQ0R1R SFP Capabilities: 4, 8 or 16 Gbit Link Data Rate: 16 Gbit Switch Port: brcd6505-fcs40:1 <List of disks visible to port> ID Vendor Model FW Size brcd6505-fcs40:12.126l1527 : NETAPP X302_HJUPI01TSSM NA GB ( B/sect) brcd6505-fcs40:12.126l1528 : NETAPP X302_HJUPI01TSSA NA GB ( B/sect)... <List of FC-to-SAS bridges visible to port> FC-to-SAS Bridge: brcd6505-fcs40:12.126l0 : ATTO FibreBridge7500N A30H FB7500N brcd6505-fcs42:13.126l0 : ATTO FibreBridge7500N A30H FB7500N <List of storage shelves visible to port> brcd6505-fcs40:12.shelf6: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: 0200 brcd6505-fcs40:12.shelf8: DS4243 Firmware rev. IOM3 A: 0200 IOM3 B: Disabling unused FC-to-SAS bridge ports After making cabling changes to the bridge, you should disable any unused FC or SAS ports on the bridge to avoid health monitor alerts related to the unused ports. Steps 1. Disable unused ports on the top FC-to-SAS bridge: a. Log in to the bridge CLI. b. Disable any unused FC ports: FCPortDisable port-number The following example shows the disabling of FC port 1: FCPortDisable 1 Fibre Channel Port 1 has been disabled. c. Disable any unused the SAS ports: SASPortDisable port-letter The following example shows disabling of SAS ports C and D: Ready. * SASPortDisable C SAS Port C has been disabled. Ready. * SASPortDisable D

39 Performing FC-to-SAS bridge maintenance 39 SAS Port D has been disabled. Ready. * d. Save the bridge configuration: SaveConfiguration The following example shows disabling of SAS ports C and D. Note that the asterisk no longer appears, indicating that the configuration has been saved. Ready. * SaveConfiguration Ready. 2. Repeat the previous step on the bottom FC-to-SAS bridge. Replacing SAS copper cables You can replace SAS copper cables with other SAS copper cables nondisruptively in a multipath HA configuration when a cable has failed or you need a different-length cable. About this task Replacing a SAS copper cable means that you are replacing a cable using the same ports for bridge-to-shelf or shelf-to-shelf connections. You cannot change any disks, disk shelves, or components of a controller module as part of this procedure. Steps 1. Verify that the system is cabled for Multipath HA by entering the following command at the system console of both controllers: run local sysconfig Note: It might take up to a minute for the system to complete discovery. For example, the following output states that the system is cabled for Multipath HA: NetApp Release 8.3.2: Tue Jan 26 01:41:49 PST 2016 System ID: (node_a_1); partner ID: (node_a_2) System Serial Number: (node_a_1); partner Serial Number: (node_a_2) System Rev: B0 System Storage Configuration: Multi-Path HA System ACP Connectivity: NA 2. Replace SAS cables by completing the following substeps: You can ignore cabling messages that might appear on the console. Note: When replacing a SAS cable, wait a minimum of 10 seconds before plugging in the new cable so that the system can detect the cable change. Note: Do not force a connector into a port. Wait at least 10 seconds before connecting the port. The SAS cable connectors are keyed; when oriented correctly into a SAS port, the connector clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves,

40 40 MetroCluster Service Guide you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector). a. Replace cables on Side A one cable at a time. The Side A cables are the cables connected to IOM A of each disk shelf. b. Verify that you have correctly replaced the SAS cables by entering the following command at the system console: sysconfig You can run the command from either controller. Note: It might take up to a minute for the system to complete discovery. The output should be the same as Step 1: the system should be Multi-Path HA, and the SAS port and attached disk shelf information should be the same. If the output is something other than Multi-Path HA, you must identify the cabling error, correct it, and run the sysconfig command again. c. Repeat substeps a and b for Side B. The Side B cables are the cables connected to IOM B of each disk shelf. Requirements for using other interfaces to configure and manage FibreBridge bridges You can use the combination of a serial port, Telnet, and FTP to manage the FibreBridge bridges instead of the recommended management interfaces. Your system must meet the requirements for the applicable interface before you install the bridges. You can use a serial port or Telnet to configure the bridge and Ethernet management 1 port, and to manage the bridge. You can use FTP to update the bridge firmware. Note: The ATTO FibreBridge 6500N Installation and Operation Manual has more information about management interfaces. You can access this document on the ATTO web site by using the link provided on the ATTO FibreBridge 6500N Description page. NetApp Downloads: ATTO FibreBridge 6500N Serial port When using the serial port to configure and manage a bridge, and to configure the Ethernet management 1 port, your system must meet the following requirements: A serial cable (which connects from the bridge serial port to a serial (COM) port on the computer you are using for setup) The bridge serial port is RJ-45 and has the same pin-out as the controllers. A terminal emulation program such as Hyperterminal, Teraterm, or PuTTY to access the console The terminal program should be capable of logging screen output to a file. Telnet When using Telnet to configure and manage a bridge, your system must meet the following requirements:

41 Performing FC-to-SAS bridge maintenance 41 A serial cable (which connects from the bridge serial port to a serial (COM) port on the computer you are using for setup) The bridge serial port is RJ-45 and has the same pin-out as the controllers. (Recommended) A non-default user name and password (for accessing the bridge) A terminal emulation program such as Hyperterminal, Teraterm, or PuTTY to access the console The terminal program should be capable of logging screen output to a file. An IP address, subnet mask, and gateway information for the Ethernet management 1 port on each bridge FTP When using FTP to update bridge firmware, your system must meet the following requirements: A standard Ethernet cable (which connects from the bridge Ethernet management 1 port to your network) (Recommended) A non-default user name and password (for accessing the bridge) Hot-replacing a failed power supply module When there is a change in status of a power supply module to the bridge, you can remove and install the power supply module. You can view the change in status of a power supply module through the LEDs on the bridge. You can also view the status of power supply modules via ExpressNAV GUI and the bridge CLI, via serial port, or via Telnet. Note: The ATTO FibreBridge 7500N Installation and Operation Manual has the procedure to install and replace a power supply module. You can access this document on the ATTO website by using the link provided on the ATTO FibreBridge Description page. Zoning for an eight- or a four-node MetroCluster configuration using FibreBridge 7500N bridges You should understand the zones and associated ports for each zone in the FC switches when using FibreBridge 7500N bridges in an eight- or a four-node fabric MetroCluster configuration. Required zones You must configure the following zones: FC-VI zones Two zones for each DR group in the MetroCluster configuration. These zones contain the FC-VI connections that allow controller-to-controller traffic. Each zone contains four ports, one for each FC-VI cable from each controller in the DR group. These zones are configured for high priority. Storage zones One zone for each of the FC-to-SAS bridge FC ports that allows traffic between initiators on each controller modules and that FC-to-SAS bridge. Each storage zone contains five ports:

42 42 MetroCluster Service Guide Four HBA initiator ports of the same fabric (two connections for each controller). One port connecting to an FC-to-SAS bridge FC port. The storage zones use standard zoning. The examples show one pair of bridges connecting a single stack group at each site. When connecting to FC1 and FC2 ports, there is a total of eight storage zones. Zoning differences between eight-node and four-node MetroCluster configurations Because switch port usage is different in eight-node and four-node MetroCluster configurations, the zoning for the configuration is different. The following sections show details for a four-node configuration and a summary of the zones for an eight-node configuration. Note: Because the existing cabling changes when expanding a four-node MetroCluster configuration to an eight-node configuration, you must reconfigure all zones when doing the expansion. GUID-8A7F7A3F-34C8-4A82-BEF5-5EC8BD61AA6F#GUID-8A7F7A3F-34C8-4A82- BEF5-5EC8BD61AA6F Port assignments for FC switches in a four-node configuration on page 122 Bridge naming The bridges use the following example naming: bridge_site_stack grouplocation in pair This portion of the name... Identifies the... Possible values... site stack group location in pair Site on which the bridge pair physically resides. Number of stack group to which the bridge pair connects. FibreBridge 7500N bridges support up to four stacks in the stack group. The stack group can contain no more than forty storage shelves. FibreBridge 6500N bridges support only a single stack in the stack group. The bridge within the bridge pair. A pair of bridges connect to a specific stack group. A or B 1, 2, etc. a or b bridge names for one stack group on each site: bridge_a_1a bridge_a_1b bridge_b_1a bridge_b_1b

43 Performing FC-to-SAS bridge maintenance 43 Storage zone naming Because there is a storage zone for each bridge FC port, the zone names correspond to each specific bridge FC port. The names have the following format: MC1-or-2_INIT_GRP_1-or-2_SITE_a-or-b_STK_GRP_n_bridge-location_FC-portnumber MC1-or-2 Identifies the DR group the port belongs to. In an eight-node MetroCluster configuration, there are two DR groups, each with its own zones In a four-node MetroCluster configuration, there is one DR group with one set of zones INIT_GRP_1-or-2 Identifies the initiator port group on the controller module. Typically there are four initiator ports divided into two groups, 0a and 0c in MC1_INIT_GRP_1 and 0b and 0d in MC1_INIT_GRP_2. SITE_a-or-b Identifies the site on which the bridge resides, either SITE_A or SITE_B. STK_GRP_n Identifies the stack group to which the bridge is connected. If there is one stack at each site, the values are STK_GRP_1 (at site_a) and STK_GRP_2 (at site_b). bridge-location Identifies the bridge in relation to the stack group to which the bridge is connected. Each stack group has a pair of bridges, identified as either TOP or BOTTOM. FC-port-number Identifies the FC port on the bridge, FC1 or FC2. Summary of zones for a four-node MetroCluster configuration (single DR group) The following table shows each of the zones and the ports in each zone. The ports are identified by domain, port pairs. Note: These examples show the port usage for a Brocade switch. If you are using Cisco switches, the port numbering is one higher than that of Brocade. For example Brocade switch port 1 corresponds to Cisco port 2. Zone Domains and ports Color in illustration QOSH1_FCVI_a 5,0; 5,3; 7,0; 7,3 Black QOSH1_FCVI_b 6,0; 6,3; 8,0; 8,3 Black MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TO P_FC1 MC1_INIT_GRP_2_SITE_A_STK_GRP_1_BO T_FC1 MC1_INIT_GRP_1_SITE_B_STK_GRP_2_TO P_FC1 MC1_INIT_GRP_2_SITE_B_STK_GRP_2_BO T_FC1 5,1; 5,4; 7,1; 7,4; 5,6 Blue 5,2; 5,5; 7,2; 7,5; 5,7 Red 5,1; 5,4; 7,1; 7,4; 7,6 Blue 5,2; 5,5; 7,2; 7,5; 7,7 Red

44 44 MetroCluster Service Guide Zone Domains and ports Color in illustration MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TO P_FC2 MC1_INIT_GRP_2_SITE_A_STK_GRP_1_BO T_FC2 MC1_INIT_GRP_1_SITE_B_STK_GRP_2_TO P_FC2 MC1_INIT_GRP_2_SITE_B_STK_GRP_2_BO T_FC2 6,1; 6,4; 8,1; 8,4; 6,6 Green 6,2; 6,5; 8,2; 8,5; 6,7 Yellow 6,1; 6,4; 8,1; 8,4; 8,6 Green 6,2; 6,5; 8,2; 8,5; 8,7 Yellow The following illustration shows connections for all zones on site_a. The illustration shows Brocade switches. Cisco switch port numbering begins with one instead of zero. Note: For simplicity, the zones for the corresponding bridge FC ports on each site are the same color. For example, MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC1 and MC1_INIT_GRP_1_SITE_B_STK_GRP_2_TOP_FC1 are both blue. These two groups include the same initiator ports, but not the same bridge FC port: Bridge port bridge_a_1a-fc1 is in MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC1 only. Bridge port bridge_b_1a-fc1 is in MC1_INIT_GRP_1_SITE_B_STK_GRP_2_TOP_FC1 only. Note: These examples show the port usage for a Brocade switch. If you are using Cisco switches, the port numbering is one higher than that of Brocade. For example Brocade switch port 1 corresponds to Cisco port 2.

45 Performing FC-to-SAS bridge maintenance 45 site_a fc-vi a fc-vi b fc-vi a fc-vi b controller_a_1 controller_a_2 0a 0b 0c 0d 0a 0b 0c 0d FC_switch_A_1 site_b FC_switch_A_2 FC1 FC2 bridge_a_1a SAS storage FC1 FC2 bridge_a_1b The following illustration shows the connections for all zones on site_b.

46 46 MetroCluster Service Guide site_b fc-vi a fc-vi b fc-vi a fc-vi b controller_b_1 controller_b_2 0a 0b 0c 0d 0a 0b 0c 0d FC_switch_B_1 site_a FC_switch_B_2 FC1 FC2 bridge_b_1a SAS storage FC1 FC2 bridge_b_1b The color codes identify which ports belong to each zone. For example, the following illustrations show a single zone, MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC1, which is shown in blue:

47 Performing FC-to-SAS bridge maintenance 47 site_a fc-vi a fc-vi b fc-vi a fc-vi b controller_a_1 controller_a_2 0a 0b 0c 0d 0a 0b 0c 0d FC_switch_A_1 site_b FC_switch_A_2 FC1 FC2 bridge_a_1a SAS storage FC1 FC2 bridge_a_1b

48 48 MetroCluster Service Guide site_b fc-vi a fc-vi b fc-vi a fc-vi b controller_b_1 controller_b_2 0a 0b 0c 0d 0a 0b 0c 0d FC_switch_B_1 site_a FC_switch_B_2 FC1 FC2 bridge_b_1a SAS storage FC1 FC2 bridge_b_1b

49 Performing FC-to-SAS bridge maintenance 49 FC-VI zones for a four-node MetroCluster configuration (single DR group) For the FC-VI ports, a Quality of Service (QoS) zone is required. A QoS zone has a special name to differentiate it from a regular zone. The FC-VI zone must have the name start with the prefix QOSH1_, followed by a user-defined portion of the name. For example, QOSH1_FCVI_a. QOSH1_FCVI_a: zone for FC-VI port a Color in illustration: Black FC switch Site Switch domain Switch port Connects to... FC_switch_A_1 A 5 0 controller_a_1 port FC-VI a 3 controller_a_2 port FC-VI a FC_switch_B_1 B 7 0 controller_b_1 port FC-VI a 3 controller_b_2 port FC-VI a QOSH1_FCVI_b: zone for FC-VI port b Color in illustration: Black FC switch Site Switch domain Switch port Connects to... FC_switch_A_2 A 6 0 controller_a_1 port FC-VI b 3 controller_a_2 port FC-VI b FC_switch_B_2 B 8 0 controller_b_1 port FC-VI b 3 controller_b_2 port FC-VI b Storage zones for a four-node MetroCluster configuration (single DR group) The following tables show the port membership for each of the storage zones to support a single stack group at each site. The controller ports shown are for onboard initiator ports. The port usage on your controller model might differ. If you have an 8020 system, there might be only two onboard initiator ports, depending on your configuration. In that case, you can use the same zoning as shown in these examples, even though the controller 0c and 0d ports are not present. Note: These examples show the port usage for a Brocade switch. If you are using Cisco switches, the port numbering is one higher than that of Brocade. For example Brocade switch port 1 corresponds to Cisco port 2. MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC1: storage zone for bridge_a_1a-fc1 (connected to port 6 on FC_switch_A_1) Color in illustration: Blue FC switch Site Switch domain Switch port Connects to... FC_switch_A_1 A 5 1 controller_a_1 port 0a 4 controller_a_2 port 0a 6 bridge_a_1a-fc1

50 50 MetroCluster Service Guide MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC1: storage zone for bridge_a_1a-fc1 (connected to port 6 on FC_switch_A_1) Color in illustration: Blue FC switch Site Switch domain Switch port Connects to... FC_switch_B_1 B 7 1 controller_b_1 port 0a 4 controller_b_2 port 0a MC1_INIT_GRP_2_SITE_A_STK_GRP_1_BOT_FC1: storage zone for bridge_a_1b-fc1 (connected to port 7 on FC_switch_A_1) Color in illustration: Red FC switch Site Switch domain Switch port Connects to... FC_switch_A_1 A 5 2 controller_a_1 port 0c 5 controller_a_2 port 0c 7 bridge_a_1b port FC1 FC_switch_B_1 B 7 2 controller_b_1 port 0c 5 controller_b_2 port 0c MC1_INIT_GRP_1_SITE_B_STK_GRP_2_TOP_FC1: storage zone for bridge_b_1a-fc1 (connected to port 6 on FC_switch_B_1) Color in illustration: Blue FC switch Site Switch domain Switch port Connects to... FC_switch_A_1 A 5 1 controller_a_1 port 0a 4 controller_a_2 port 0a FC_switch_B_1 B 7 1 controller_b_1 port 0a 4 controller_b_2 port 0a 6 bridge_b_1a port FC1 MC1_INIT_GRP_2_SITE_B_STK_GRP_2_BOT_FC1: storage zone for bridge_b_1b-fc1 (connected to port 7 on FC_switch_B_1) Color in illustration: Red FC switch Site Switch domain Switch port Connects to... FC_switch_A_1 A 5 2 controller_a_1 port 0c 5 controller_a_2 port 0c FC_switch_B_1 B 7 2 controller_b_1 port 0c 5 controller_b_2 port 0c 7 bridge_b_1b port FC1

51 Performing FC-to-SAS bridge maintenance 51 MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC2: storage zone for bridge_a_1a-fc2 (connected to port 6 on FC_switch_A_2) Color in illustration: Green FC switch Site Switch domain Switch port Connects to... FC_switch_A_2 A 6 1 controller_a_1 port 0b 4 controller_a_2 port 0b 6 bridge_a_1a port FC2 FC_switch_B_2 B 8 1 controller_b_1 port 0b 4 controller_b_2 port 0b MC1_INIT_GRP_2_SITE_A_STK_GRP_1_BOT_FC2: storage zone for bridge_a_1b-fc2 (connected to port 7 on FC_switch_A_2) Color in illustration: Yellow FC switch Site Switch domain Switch port Connects to... FC_switch_A_2 A 6 2 controller_a_1 port 0d 5 controller_a_2 port 0d 7 bridge_a_1b port FC2 FC_switch_B_2 B 8 2 controller_b_1 port 0d 5 controller_b_2 port 0d MC1_INIT_GRP_1_SITE_B_STK_GRP_2_TOP_FC2: storage zone for bridge_b_1a-fc2 (connected to port 6 on FC_switch_B_2) Color in illustration: Green FC switch Site Switch domain Switch port Connects to... FC_switch_A_2 A 6 1 controller_a_1 port 0b 4 controller_a_2 port 0b FC_switch_B_2 B 8 1 controller_b_1 port 0b 4 controller_b_2 port 0b 6 bridge_b_1a port FC2 MC1_INIT_GRP_2_SITE_B_STK_GRP_2_BOT_FC2: storage zone for bridge_b_1b-fc2 (connected to port 7 on FC_switch_B_2) Color in illustration: Yellow FC switch Site Switch domain Switch port Connects to... FC_switch_A_2 A 6 2 controller_a_1 port 0d 5 controller_a_2 port 0d

52 52 MetroCluster Service Guide MC1_INIT_GRP_2_SITE_B_STK_GRP_2_BOT_FC2: storage zone for bridge_b_1b-fc2 (connected to port 7 on FC_switch_B_2) Color in illustration: Yellow FC switch Site Switch domain Switch port Connects to... FC_switch_B_2 B 8 2 controller_b_1 port 0d 5 controller_b_2 port 0d 7 bridge_b_1b port FC2 Summary of zones for an eight-node MetroCluster configuration (two DR groups) The following table shows each of the zones and the ports in each zone. The ports are identified by domain, port pairs. Note: These examples show the port usage for a Brocade switch. If you are using Cisco switches, the port numbering is one higher than that of Brocade. For example Brocade switch port 1 corresponds to Cisco port 2. DR group Zone Domains and ports DR group 1 (MC1) QOSH1_MC1_FAB_1_FCVI 5,0; 5,3; 7,0; 7,3 QOSH1_MC1_FAB_2_FCVI 6,0;6,3;8,0;8,3 MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC1 5,1;5,4;7,1;7,4;5,12 MC1_INIT_GRP_2_SITE_A_STK_GRP_1_BOT_FC1 5,2;5,5;7,2;7,5;5,13 MC1_INIT_GRP_1_SITE_B_STK_GRP_2_TOP_FC1 5,1;5,4;7,1;7,4;7,12 MC1_INIT_GRP_2_SITE_B_STK_GRP_2_BOT_FC1 5,2;5,5;7,2;7,5;7,13 MC1_INIT_GRP_1_SITE_A_STK_GRP_3_TOP_FC1 5,1;5,4;7,1;7,4;5,16 MC1_INIT_GRP_2_SITE_A_STK_GRP_3_BOT_FC1 5,2;5,5;7,2;7,5;5,17 MC1_INIT_GRP_1_SITE_B_STK_GRP_4_TOP_FC1 5,1;5,4;7,1;7,4;7,16 MC1_INIT_GRP_2_SITE_B_STK_GRP_4_BOT_FC1 5,2;5,5;7,2;7,5;7,17 MC1_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC2 6,1;6,4;8,1;8,4;6,12 MC1_INIT_GRP_2_SITE_A_STK_GRP_1_BOT_FC2 6,2;6,5;8,2;8,5;6,13 MC1_INIT_GRP_1_SITE_B_STK_GRP_2_TOP_FC2 6,1;6,4;8,1;8,4;8,12 MC1_INIT_GRP_2_SITE_B_STK_GRP_2_BOT_FC2 6,2;6,5;8,2;8,5;8,13 MC1_INIT_GRP_1_SITE_A_STK_GRP_3_TOP_FC2 6,1;6,4;8,1;8,4;6,16 MC1_INIT_GRP_2_SITE_A_STK_GRP_3_BOT_FC2 6,2;6,5;8,2;8,5;6,17 MC1_INIT_GRP_1_SITE_B_STK_GRP_4_TOP_FC2 6,1;6,4;8,1;8,4;8,16 MC1_INIT_GRP_2_SITE_B_STK_GRP_4_BOT_FC2 6,2;6,5;8,2;8,5;8,17

53 Performing FC-to-SAS bridge maintenance 53 DR group Zone Domains and ports DR group 2 (MC2) QOSH2_MC2_FAB_1_FCVI 5,6;5,9;7,6;7,9 QOSH2_MC2_FAB_2_FCVI 6,6;6,9;8,6;8,9 MC2_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC1 5,7;5,10;7,7;7,10;5, 14 MC2_INIT_GRP_2_SITE_A_STK_GRP_1_BOT_FC1 5,8;5,11;7,8;7,11;5, 15 MC2_INIT_GRP_1_SITE_B_STK_GRP_2_TOP_FC1 5,7;5,10;7,7;7,10;7, 14 MC2_INIT_GRP_2_SITE_B_STK_GRP_2_BOT_FC1 5,8;5,11;7,8;7,11;7, 15 MC2_INIT_GRP_1_SITE_A_STK_GRP_3_TOP_FC1 5,7;5,10;7,7;7,10;5, 18 MC2_INIT_GRP_2_SITE_A_STK_GRP_3_BOT_FC1 5,8;5,11;7,8;7,11;5, 19 MC2_INIT_GRP_1_SITE_B_STK_GRP_4_TOP_FC1 5,7;5,10;7,7;7,10;7, 18 MC2_INIT_GRP_2_SITE_B_STK_GRP_4_BOT_FC1 5,8;5,11;7,8;7,11;7, 19 MC2_INIT_GRP_1_SITE_A_STK_GRP_1_TOP_FC2 6,7;6,10;8,7;8,10;6, 14 MC2_INIT_GRP_2_SITE_A_STK_GRP_1_BOT_FC2 6,8;6,11;8,8;8,11;6, 15 MC2_INIT_GRP_1_SITE_B_STK_GRP_2_TOP_FC2 6,7;6,10;8,7;8,10;8, 14 MC2_INIT_GRP_2_SITE_B_STK_GRP_2_BOT_FC2 6,8;6,11;8,8;8,11;8, 15 MC2_INIT_GRP_1_SITE_A_STK_GRP_3_TOP_FC2 6,7;6,10;8,7;8,10;6, 18 MC2_INIT_GRP_2_SITE_A_STK_GRP_3_BOT_FC2 6,8;6,11;8,8;8,11;6, 19 MC2_INIT_GRP_1_SITE_B_STK_GRP_4_TOP_FC2 6,7;6,10;8,7;8,10;8, 18 MC2_INIT_GRP_2_SITE_B_STK_GRP_4_BOT_FC2 6,8;6,11;8,8;8,11;8, 19

54 54 MetroCluster Service Guide Zoning for FibreBridge 6500N bridges in a four-node MetroCluster configuration You should understand the zones and associated ports for each zone in the Brocade FC switches when using FibreBridge 6500N bridges in a four-node fabric MetroCluster configuration. Required zones You must configure the following zones: FC-VI zones Two zones for the FC-VI connections that allow controller-to-controller traffic Each zone contains four ports, one for each FC-VI cable from each controller. These zones are configured for high priority. Storage zones One zone for each of the FC-to-SAS bridge FC ports that allows traffic between initiators on each controller modules and that FC-to-SAS bridge. Each storage zone contains five ports: Four HBA initiator ports (two connections for each controller) One port connecting to an FC-to-SAS bridge FC port The storage zones use standard zoning. The examples show one pair of bridges connecting a single stack group at each site. Because each bridge uses two FC ports, there is a total of eight storage zones. FC-VI zones For the FC-VI ports, a Quality of Service (QoS) zone is required. A QoS zone has a special name to differentiate it from a regular zone. The FC-VI zone must have the name start with the prefix QOSH1_, followed by a user-defined portion of the name. For example, QOSH1_FCVI_a. Note: The following examples show the port usage for a Brocade switch. If you are using Cisco switches, the port numbering is one higher than that of Brocade switches. For example, Brocade switch port 1 corresponds to Cisco switch port 2. QOSH1_FCVI_a: storage zone for FC-VI port a FC switch Site Switch domain Switch port Connects to... FC_switch_A_1 A 5 0 controller_a_1 port FC-VI a 3 controller_a_2 port FC-VI a FC_switch_B_1 B 7 0 controller_b_1 FC- VI a 3 controller_b_2 FC- VI a

55 Performing FC-to-SAS bridge maintenance 55 QOSH1_FCVI_b: storage zone for FC-VI port b FC switch Site Switch domain Switch port Connects to... FC_switch_A_2 A 6 0 controller_a_1 port FC-VI b 3 controller_a_2 port FC-VI b FC_switch_B_2 B 8 0 controller_b_1 FC- VI b 3 controller_b_2 FC- VI b Bridge naming The bridges use the following example naming: bridge_site_stack grouplocation in pair This portion of the name... Identifies the... Possible values... site stack group location in pair Site on which the bridge pair physically resides. Number of stack group to which the bridge pair connects. FibreBridge 7500N bridges support up to four stacks in the stack group. The stack group can contain no more than forty storage shelves. FibreBridge 6500N bridges support only a single stack in the stack group. The bridge within the bridge pair. A pair of bridges connect to a specific stack group. A or B 1, 2, etc. a or b bridge names for one stack group on each site: bridge_a_1a bridge_a_1b bridge_b_1a bridge_b_1b Storage zones The following tables show the zoning for a single pair of FC-to-SAS bridges at each site. The controller ports shown are for onboard initiator ports. The port usage on your controller model might differ. If you have an 8020 system, there are only two onboard initiator ports. In that case, you can use the same zoning as shown in these examples, even though the controller 0c and 0d ports are not present.

56 56 MetroCluster Service Guide Note: The following examples show the port usage for a Brocade switch. If you are using Cisco switches, the port numbering is one higher than that of Brocade switches. For example, Brocade switch port 1 corresponds to Cisco switch port 2. STOR_A_1a-FC1: storage zone for bridge_a_1a-fc1 (connected to port 6 on FC_switch_A_1) FC switch Site Switch domain Switch port Connects to... FC_switch_A_1 A 5 1 controller_a_1 port 0a 2 controller_a_2 port 0a 4 controller_a_1 port 0c 5 controller_a_2 port 0c 6 bridge_a_1a-fc1 FC_switch_B_1 B 7 1 controller_b_1 port 0a 2 controller_b_2 port 0a 4 controller_b_1 port 0c 5 controller_b_2 port 0c STOR_B-1a-FC1: storage zone for bridge_b_1a-fc1 (connected to port 6 on FC_switch_B_1) FC switch Site Switch domain Switch port Connects to... FC_switch_A_1 A 5 1 controller_a_1 port 0a 2 controller_a_2 port 0a 4 controller_a_1 port 0c 5 controller_a_2 port 0c FC_switch_B_1 B 7 1 controller_b_1 port 0a 2 controller_b_2 port 0a 4 controller_b_1 port 0c 5 controller_b_2 port 0c 6 bridge_b_1b-fc1

57 Performing FC-to-SAS bridge maintenance 57 STOR_A_1b-FC1: storage zone for bridge_a_1b-fc1 (connected to port 6 on FC_switch_A_2) FC switch Site Switch domain Switch port Connects to... FC_switch_A_2 A 6 1 controller_a_1 port 0a 2 controller_a_2 port 0a 4 controller_a_1 port 0c 5 controller_a_2 port 0c 6 bridge_a_1b-fc1 FC_switch_B_2 B 8 1 controller_b_1 port 0a 2 controller_b_2 port 0a 4 controller_b_1 port 0c 5 controller_b_2 port 0c STOR_B-1b-FC1: storage zone for bridge_b_1b-fc1 (connected to port 6 on FC_switch_B_2) FC switch Site Switch domain Switch port Connects to... FC_switch_A_2 A 6 1 controller_a_1 port 0a 2 controller_a_2 port 0a 4 controller_a_1 port 0c 5 controller_a_2 port 0c FC_switch_B_2 B 8 1 controller_b_1 port 0a 2 controller_b_2 port 0a 4 controller_b_1 port 0c 5 controller_b_2 port 0c 6 bridge_b_1b-fc1

58 58 MetroCluster Service Guide Summary of zones The following table shows each of the zones and the ports in each zone. The ports are identified by domain, port pairs. Zone Domains and ports QOSH1_FCVI_a 5,0; 5,3; 7,0; 7,3 QOSH1_FCVI_b 6,0; 6,3; 8,0; 8,3 STOR_A_1a-FC1 5,1; 5,2; 5,4; 5,5; 7,1; 7,2; 7,4; 7,5; 5,6 STOR_B_1a-FC1 5,1; 5,2; 5,4; 5,5; 7,1; 7,2; 7,4; 7,5; 7,6 STOR_A_1b-FC1 6,1; 6,2; 6,4; 6,5; 8,1; 8,2; 8,4; 8,5; 6,6 STOR_B_1b-FC1 6,1; 6,2; 6,4; 6,5; 8,1; 8,2; 8,4; 8,5; 8,6

59 59 FC switch maintenance and replacement If necessary, you can non-disruptively replace the FC switches or upgrade their firmware in the MetroCluster configuration. Choices Upgrading firmware on a Brocade FC switch on page 59 Upgrading firmware on a Cisco FC switch on page 61 Replacing a Brocade FC switch on page 63 Replacing a Cisco FC switch on page 67 Upgrading firmware on a Brocade FC switch To upgrade the firmware on a Brocade FC switch you must use the Brocade-specific commands to disable the switch, perform and verify the upgrade, and reboot and reenable the switch. Before you begin You must have the firmware files. The system must be properly cabled. All paths to the storage shelves must be available. The disk shelf stacks must be stable. The FC switch fabric must be healthy. No failed components can be present in the system. The system must be operating normally. About this task The switch fabric is disabled during the firmware upgrade, and the MetroCluster configuration relies on the second fabric to continue operation. This task must be performed on each of the switch fabrics in succession to ensure that all switches are running the same firmware version. Steps 1. Log in to each of the switches in the fabric. The examples in the following steps use FC_switch_A_1 and FC_switch_B_1. 2. Disable each of the switches in the fabric: switchcfgpersistentdisable If this command is not available, use the switchdisable command. The following example shows the command issued on FC_switch_A_1: FC_switch_A_1:admin> switchcfgpersistentdisable

60 60 MetroCluster Service Guide The following example shows the command issued on FC_switch_B_1: FC_switch_B_1:admin> switchcfgpersistentdisable 3. Start the firmware upgrade: firmwaredownload If prompted, provide the appropriate information. You can run the firmwaredownload command at the same time on both switches, but you must allow the firmware to download and commit properly before moving to the next step. The following example shows the command issued on FC_switch_A_1: FC_switch_A_1:admin> firmwaredownload The following example shows the command issued on FC_switch_B_1: FC_switch_B_1:admin> firmwaredownload 4. Ensure that the firmware has been upgraded and committed to both partitions: firmwareshow The following example shows the command issued on FC_switch_A_1: FC_switch_A_1:admin> firmwareshow The following example shows the command issued on FC_switch_B_1: FC_switch_B_1:admin> firmwareshow 5. Reboot the switches by using either the fastboot or reboot command: reboot Some firmware versions automatically perform an hareboot operation after the firmware upgrade is finished. The reboot in this step is required even if the hareboot was performed. The following example shows the fastboot command issued on FC_switch_A_1: FC_switch_A_1:admin> fastboot The following example shows the fastboot command issued on FC_switch_B_1: FC_switch_B_1:admin> fastboot 6. If the upgrade is to an intermediate firmware level and not the final target release perform this step, and then return to step 4 on page 59 until the target release is installed. 7. Enable the switches: switchcfgpersistentenable

61 FC switch maintenance and replacement 61 If this command is not available, the switch should be in enabled state after the fastboot command was issued. The following example shows the command issued on FC_switch_A_1: FC_switch_A_1:admin> switchcfgpersistentenable The following example shows the command issued on FC_switch_B_1: FC_switch_B_1:admin> switchcfgpersistentenable 8. Verify that the switches are online and all devices are properly logged in: switchshow The following example shows the command issued on FC_switch_A_1: FC_switch_A_1:admin> switchshow The following example shows the command issued on FC_switch_B_1: FC_switch_B_1:admin> switchshow 9. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered. 10. Repeat this procedure for the second switch fabric. Upgrading firmware on a Cisco FC switch To update the firmware on a Cisco FC switch you must use the Cisco-specific commands to disable the switch, perform and verify the upgrade, and reboot and reenable the switch. Before you begin The system must be properly cabled.

62 62 MetroCluster Service Guide All paths to the storage shelves must be available. The disk shelf stacks must be stable. The FC switch fabric must be healthy. All components in the system must be healthy. The system must be operating normally. About this task The switch fabric is disabled during the firmware upgrade and the MetroCluster configuration relies on the second fabric to continue operation. You must repeat this task on each of the switch fabrics in succession to ensure that all switches are running the same firmware version. You must have the firmware files. Steps 1. Log in to each of the switches in the fabric. In the examples, the switches are called FC_switch_A_1 and FC_switch_B_1. 2. Determine whether there is enough space in the bootflash directory on each switch: dir bootflash If not, delete the unwanted firmware files by using the delete bootflash:file_name command. 3. Copy the kickstart and system files to the switches: copy source_file target_file In the following example, the kickstart file (m9200-s2ek9-kickstart-mz bin) and the system file (m9200-s2ek9-mz bin) are located on the FTP server in the /firmware/ path. The following example shows the commands issued on FC_switch_A_1: FC_switch_A_1# copy ftp:// /firmware/m9200-s2ek9-kickstartmz bin bootflash:m9200-s2ek9-kickstart-mz bin FC_switch_A_1# copy ftp:// /firmware/m9200-s2ek9-mz bin bootflash:m9200-s2ek9-mz bin The following example shows the command issued on FC_switch_B_1: FC_switch_B_1# copy ftp:// /firmware/m9200-s2ek9-kickstartmz bin bootflash:m9200-s2ek9-kickstart-mz bin FC_switch_B_1# copy ftp:// /firmware/m9200-s2ek9-mz bin bootflash:m9200-s2ek9-mz bin 4. Disable all the VSANs on both the switches in this fabric. 5. Install the firmware on the switches: install all system bootflash:systemfile_name kickstart bootflash:kickstartfile_name

63 FC switch maintenance and replacement 63 The following example shows the commands issued on FC_switch_A_1: FC_switch_A_1# install all system bootflash:m9200-s2ek9-mz bin kickstart bootflash:m9200-s2ek9-kickstart-mz bin Enter Yes to confirm the installation. The following example shows the command issued on FC_switch_B_1: FC_switch_B_1# install all system bootflash:m9200-s2ek9-mz bin kickstart bootflash:m9200-s2ek9-kickstart-mz bin Enter Yes to confirm the installation. 6. Check the version of the firmware on each switch to make sure the new version has been installed: show version 7. Enable all the VSANs on both the switches in this fabric. 8. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered. 9. Repeat this procedure for the second switch fabric. Replacing a Brocade FC switch You must use this Brocade-specific procedure to replace a failed switch. About this task In the following examples, FC_switch_A_1 is the healthy switch and FC_switch_B_1 is the impaired switch. The switch port usage is shown in the following table: Port connections Ports FC-VI connections 0, 3 HBA connections 1, 2, 4, 5 FC-to-SAS bridge connections 6, 7 ISL connections 10, 11

64 64 MetroCluster Service Guide The examples show two FC-to-SAS bridges. If you have more, you must disable and subsequently enable the additional ports. Steps 1. Fence off the switch undergoing replacement by disabling the ISL ports on the healthy switch in the fabric and the FC-VI and HBA ports on the impaired switch (if the impaired switch is till operating): a. Disable the ISL ports on the healthy switch by issuing the following command for each port: portcfgpersistentdisable port-number FC_switch_A_1:admin> portcfgpersistentdisable 10 FC_switch_A_1:admin> portcfgpersistentdisable 11 b. If the impaired switch is still operational, disable the FC-VI and HBA ports on that switch by issuing the following command for each port: portcfgpersistentdisable port-number FC_switch_B_1:admin> portcfgpersistentdisable 0 FC_switch_B_1:admin> portcfgpersistentdisable 1 FC_switch_B_1:admin> portcfgpersistentdisable 2 FC_switch_B_1:admin> portcfgpersistentdisable 3 FC_switch_B_1:admin> portcfgpersistentdisable 4 FC_switch_B_1:admin> portcfgpersistentdisable 5 2. If the impaired switch is still operational, gather the output from the switchshow command. FC_switch_B_1:admin> switchshow switchname: FC_switch_B_1 switchtype: 71.2 switchstate:online switchmode: Native switchrole: Subordinate switchdomain: 2 switchid: fffc01 switchwwn: 10:00:00:05:33:86:89:cb zoning: OFF switchbeacon: OFF 3. Boot and pre-configure the new switch prior to physically installing it: a. Power up the new switch and let it boot up. b. Check the firmware version on the switch to confirm it matches the version of the other FC switches: firmwareshow c. Configure the new switch as described in the MetroCluster Installation and Configuration Guide, skipping the section "Configuring zoning on Brocade FC switches." You will configure zoning later in this procedure. Note:.At this point, the new switch is not cabled to the MetroCluster configuration.

65 FC switch maintenance and replacement 65 d. Disable the FC-VI, HBA and storage ports on the new switch, and the ports connected to the FC-SAS bridges. FC_switch_B_1:admin> portcfgpersistentdisable 0 FC_switch_B_1:admin> portcfgpersistentdisable 1 FC_switch_B_1:admin> portcfgpersistentdisable 2 FC_switch_B_1:admin> portcfgpersistentdisable 3 FC_switch_B_1:admin> portcfgpersistentdisable 4 FC_switch_B_1:admin> portcfgpersistentdisable 5 FC_switch_B_1:admin> portcfgpersistentdisable 6 FC_switch_B_1:admin> portcfgpersistentdisable 7 4. Physically replace the switch: a. Power off the impaired FC switch. b. Power off the replacement FC switch. c. Uncable and remove the impaired switch, carefully noting which cables connected to which ports. d. Install the replacement switch in the rack. e. Cable the replacement switch exactly as the old switch was cabled. f. Power on the new FC switch. 5. Complete the configuration of the new switch: a. Enable the ISLs: portcfgpersistentenable port-number FC_switch_B_1:admin> portcfgpersistentenable 10 FC_switch_B_1:admin> portcfgpersistentenable 11 b. On the replacement switch (FC_switch_B_1 in our example), verify that the ISLs are online: switchshow FC_switch_B_1:admin> switchshow switchname: FC_switch_B_1 switchtype: 71.2 switchstate:online switchmode: Native switchrole: Principal switchdomain: 4 switchid: fffc03 switchwwn: 10:00:00:05:33:8c:2e:9a zoning: OFF switchbeacon: OFF Index Port Address Media Speed State Proto ============================================== A00 id 16G Online FC E-Port

66 66 MetroCluster Service Guide 10:00:00:05:33:86:89:cb "FC_switch_A_1" B00 id 16G Online FC E-Port 10:00:00:05:33:86:89:cb "FC_switch_A_1" (downstream)... c. Enable the storage ports that connect to the FC bridges. FC_switch_B_1:admin> portcfgpersistentenable 6 FC_switch_B_1:admin> portcfgpersistentenable 7 d. Enable the storage, HBA, and FC-VI ports. The following example shows the commands used to enable the ports connecting HBA adapters: FC_switch_B_1:admin> portcfgpersistentenable 1 FC_switch_B_1:admin> portcfgpersistentenable 2 FC_switch_B_1:admin> portcfgpersistentenable 4 FC_switch_B_1:admin> portcfgpersistentenable 5 The following example shows the commands used to enable the ports connecting the FC-VI adapters: FC_switch_B_1:admin> portcfgpersistentenable 0 FC_switch_B_1:admin> portcfgpersistentenable 3 6. Verify that the ports are online: switchshow 7. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered.

67 FC switch maintenance and replacement 67 Replacing a Cisco FC switch You must use Cisco-specific steps to replace a failed Cisco FC switch. About this task In the examples in this procedure, FC_switch_A_1 is the healthy switch and FC_switch_B_1 is the impaired switch. The switch port usage is shown in the following table: Role Ports FC-VI connections 1, 4 HBA connections 2, 3, 5, 6 FC-to-SAS bridge connections 7, 8 ISL connections 36, 40 The examples show two FC-to-SAS bridges. If you have more, you must disable and subsequently enable the additional ports. Steps 1. Disable the ISL ports on the healthy switch to fence off the impaired switch. These steps are performed on the healthy switch. a. Enter configuration mode: conf t b. Disable the ISL ports on the healthy switch with the interface and shut commands. FC_switch_A_1# conf t FC_switch_A_1(config)# interface fc1/36 FC_switch_A_1(config)# shut FC_switch_A_1(config)# interface fc1/40 FC_switch_A_1(config)# shut c. Exit configuration mode and copy the configuration to the startup configuration. FC_switch_A_1(config)# end FC_switch_A_1# copy running-config startup-config FC_switch_A_1# 2. Fence off the FC-VI and HBA ports on the impaired switch (if it is still running). These steps are performed on the impaired switch. a. Enter configuration mode: conf t b. If the impaired switch is still operational, disable the FC-VI and HBA ports on the impaired switch with the interface and shut commands.

68 68 MetroCluster Service Guide FC_switch_B_1(config)# interface fc1/1 FC_switch_B_1(config)# shut FC_switch_B_1(config)# interface fc1/4 FC_switch_B_1(config)# shut FC_switch_B_1(config)# interface fc1/2-3 FC_switch_B_1(config)# shut FC_switch_B_1(config)# interface fc1/5-6 FC_switch_B_1(config)# shut c. Exit configuration mode and copy the configuration to the startup configuration. FC_switch_B_1(config)# end FC_switch_B_1# copy running-config startup-config FC_switch_B_1# 3. If the impaired switch is still operational, determine the WWN for the switch: show wwn switch FC_switch_B_1# show wwn switch Switch WWN is 20:00:54:7f:ee:e3:86:50 FC_switch_B_1# 4. Boot and preconfigure the replacement switch, prior to physically installing it. At this point the replacement switch is not cabled to the MetroCluster configuration. The ISL ports on the partner switch are disabled (in shut mode) and offline. a. Power on the replacement switch and let it boot up. b. Check the firmware version on the replacement switch to confirm that it matches the version of the other FC switches: show version c. Configure the replacement switch as described in the MetroCluster Installation and Configuration Guide, skipping the Configuring zoning on a Cisco FC switch section. You will configure zoning later in this procedure. d. Disable the FC-VI, HBA, and storage ports on the replacement switch. FC_switch_B_1# conf t FC_switch_B_1(config)# interface fc1/1 FC_switch_B_1(config)# shut FC_switch_B_1(config)# interface fc1/4 FC_switch_B_1(config)# shut FC_switch_B_1(config)# interface fc1/2-3 FC_switch_B_1(config)# shut FC_switch_B_1(config)# interface fc1/5-6 FC_switch_B_1(config)# shut FC_switch_B_1(config)# interface fc1/7-8 FC_switch_B_1(config)# shut FC_switch_B_1# copy running-config startup-config FC_switch_B_1#

69 FC switch maintenance and replacement Physically replace the impaired switch: a. Power off the impaired switch. b. Power off the replacement switch. c. Uncable and remove the impaired switch, carefully noting which cables connected to which ports. d. Install the replacement switch in the rack. e. Cable the replacement switch exactly as the impaired switch was cabled. f. Power on the replacement switch. 6. Enable the ISL ports on the replacement switch. FC_switch_B_1# conf t FC_switch_B_1(config)# interface fc1/36 FC_switch_B_1(config)# no shut FC_switch_B_1(config)# end FC_switch_B_1# copy running-config startup-config FC_switch_B_1(config)# interface fc1/40 FC_switch_B_1(config)# no shut FC_switch_B_1(config)# end FC_switch_B_1# 7. Verify that the ISL ports on the replacement switch are up: show interface brief 8. Adjust the zoning on the replacement switch to match the MetroCluster configuration: a. Distribute the zoning information from the healthy fabric. In this example, FC_switch_B_1 has been replaced and the zoning information is retrieved from FC_switch_A_1: FC_switch_A_1(config-zone)# zoneset distribute full vsan 10 FC_switch_A_1(config-zone)# zoneset distribute full vsan 20 FC_switch_A_1(config-zone)# end b. On the replacement switch, verify that the zoning information was properly retrieved from the healthy switch: show zone FC_switch_B_1# show zone zone name FC-VI_Zone_1_10 vsan 10 interface fc1/1 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/4 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/1 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/4 swwn 20:00:54:7f:ee:b8:24:c0 zone name STOR_Zone_1_20_25A vsan 20 interface fc1/2 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/3 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/5 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/6 swwn 20:00:54:7f:ee:e3:86:50

70 70 MetroCluster Service Guide interface fc1/2 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/3 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/5 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/6 swwn 20:00:54:7f:ee:b8:24:c0 zone name STOR_Zone_1_20_25B vsan 20 interface fc1/2 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/3 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/5 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/6 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/2 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/3 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/5 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/6 swwn 20:00:54:7f:ee:b8:24:c0 FC_switch_B_1# c. Find the WWNs of the switches. In this example, the two switch WWNs are as follows: FC_switch_A_1: 20:00:54:7f:ee:b8:24:c0 FC_switch_B_1: 20:00:54:7f:ee:c6:80:78 FC_switch_B_1# show wwn switch Switch WWN is 20:00:54:7f:ee:c6:80:78 FC_switch_B_1# FC_switch_A_1# show wwn switch Switch WWN is 20:00:54:7f:ee:b8:24:c0 FC_switch_A_1# d. Remove zone members that do not belong to the switch WWNs of the two switches. In this example, no member interface in the output shows that the following members are not associated with the switch WWN of either of the switches in the fabric and must be removed: zone name FC-VI_Zone_1_10 vsan 10 interface fc1/1 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/2 swwn 20:00:54:7f:ee:e3:86:50 zone name STOR_Zone_1_20_25A vsan 20 interface fc1/5 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/8 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/9 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/10 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/11 swwn 20:00:54:7f:ee:e3:86:50 zone name STOR_Zone_1_20_25B vsan 20 interface fc1/8 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/9 swwn 20:00:54:7f:ee:e3:86:50

71 FC switch maintenance and replacement 71 interface fc1/10 swwn 20:00:54:7f:ee:e3:86:50 interface fc1/11 swwn 20:00:54:7f:ee:e3:86:50 The following example shows the removal of these interfaces: FC_switch_B_1# conf t FC_switch_B_1(config)# zone name FC-VI_Zone_1_10 vsan 10 FC_switch_B_1(config-zone)# no member interface fc1/1 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# no member interface fc1/2 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# zone name STOR_Zone_1_20_25A vsan 20 FC_switch_B_1(config-zone)# no member interface fc1/5 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# no member interface fc1/8 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# no member interface fc1/9 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# no member interface fc1/10 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# no member interface fc1/11 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# zone name STOR_Zone_1_20_25B vsan 20 FC_switch_B_1(config-zone)# no member interface fc1/8 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# no member interface fc1/9 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# no member interface fc1/10 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# no member interface fc1/11 swwn 20:00:54:7f:ee:e3:86:50 FC_switch_B_1(config-zone)# save running-config startup-config FC_switch_B_1(config-zone)# zoneset distribute full 10 FC_switch_B_1(config-zone)# zoneset distribute full 20 FC_switch_B_1(config-zone)# end FC_switch_B_1# copy running-config startup-config e. Add the ports of the replacement switch to the zones. All the cabling on the replacement switch must be the same as on the impaired switch: FC_switch_B_1# conf t FC_switch_B_1(config)# zone name FC-VI_Zone_1_10 vsan 10 FC_switch_B_1(config-zone)# member interface fc1/1 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# member interface fc1/2 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# zone name STOR_Zone_1_20_25A vsan 20 FC_switch_B_1(config-zone)# member interface fc1/5 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# member interface fc1/8 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# member interface fc1/9 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# member interface fc1/10 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# member interface fc1/11 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# zone name STOR_Zone_1_20_25B vsan 20 FC_switch_B_1(config-zone)# member interface fc1/8 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# member interface fc1/9 swwn 20:00:54:7f:ee:c6:80:78

72 72 MetroCluster Service Guide FC_switch_B_1(config-zone)# member interface fc1/10 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# member interface fc1/11 swwn 20:00:54:7f:ee:c6:80:78 FC_switch_B_1(config-zone)# save running-config startup-config FC_switch_B_1(config-zone)# zoneset distribute full 10 FC_switch_B_1(config-zone)# zoneset distribute full 20 FC_switch_B_1(config-zone)# end FC_switch_B_1# copy running-config startup-config f. Verify that the zoning is properly configured: show zone The following example output shows the three zones: FC_switch_B_1# show zone zone name FC-VI_Zone_1_10 vsan 10 interface fc1/1 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/2 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/1 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/2 swwn 20:00:54:7f:ee:b8:24:c0 zone name STOR_Zone_1_20_25A vsan 20 interface fc1/5 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/8 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/9 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/10 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/11 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/8 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/9 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/10 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/11 swwn 20:00:54:7f:ee:b8:24:c0 zone name STOR_Zone_1_20_25B vsan 20 interface fc1/8 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/9 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/10 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/11 swwn 20:00:54:7f:ee:c6:80:78 interface fc1/5 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/8 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/9 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/10 swwn 20:00:54:7f:ee:b8:24:c0 interface fc1/11 swwn 20:00:54:7f:ee:b8:24:c0 FC_switch_B_1# g. Enable the connectivity to storage and the controllers. The following example shows the port usage: FC_switch_A_1# conf t FC_switch_A_1(config)# interface fc1/1 FC_switch_A_1(config)# no shut FC_switch_A_1(config)# interface fc1/4 FC_switch_A_1(config)# shut FC_switch_A_1(config)# interface fc1/2-3 FC_switch_A_1(config)# shut FC_switch_A_1(config)# interface fc1/5-6 FC_switch_A_1(config)# shut

73 FC switch maintenance and replacement 73 FC_switch_A_1(config)# interface fc1/7-8 FC_switch_A_1(config)# shut FC_switch_A_1# copy running-config startup-config FC_switch_A_1# 9. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered.

74 74 Nondisruptively replacing SAS cables You can nondisruptively replace SAS copper and SAS optical cables in the MetroCluster configuration. About this task Replacing a SAS cable means that you are replacing a cable using the same exact ports for a controller-to-shelf or shelf-to-shelf connection. Situations where you might want to replace a SAS cable can include when a cable has failed, a longer cable is needed, or SAS optical cables are preferred instead of SAS copper cables. You cannot change any disks, disk shelves, or components of a controller module as part of these procedures. Steps 1. Verify that your stretch MetroCluster system is Multi-Path HA by running the following command at the console of both controllers: sysconfig Note: It might take up to a minute for the system to complete discovery. The configuration is listed in the System Storage configuration field. Attention: If your system configuration is shown as something other than Multi-Path HA, you cannot continue with this procedure until you meet the procedure prerequisites. 2. If you are replacing SAS copper cables with SAS optical cables, verify that the disk shelves in the storage system have the latest version of disk shelf firmware by completing the following substeps; otherwise, go to Step 4. a. Enter the following command at the storage system console: sasadmin expander_map b. Locate the disk shelf firmware information for the disk shelves in the output is the disk shelf firmware version for shelf number one (Slot A/IOM A) in the storage system: Expanders on channel 4a: Level 3: WWN 500a ff, ID 1, Serial Number ' SHU G114C', Product 'DS424IOM6 ', Rev '0151', Slot A c. Compare the firmware information in the command output with the disk shelf firmware information at NetApp Downloads: Disk Shelf Firmware to determine the most current disk shelf firmware version. 3. The next step depends on how current the disk shelf firmware is. If the firmware version in the command output is... The same as or later than the most current version on NetApp Support Then... No disk shelf firmware update is needed.

75 Nondisruptively replacing SAS cables 75 If the firmware version in the command output is... An earlier version than the most current version on NetApp Support Then... Download the disk shelf firmware file by using the procedure at NetApp Downloads: Disk Shelf Firmware. You can run the commands from either controller. 4. Replace SAS cables by completing the following substeps: You can ignore cabling messages that might appear on the console. Note: When replacing a SAS cable, wait a minimum of 10 seconds before plugging in the new cable so that the system can detect the cable change. Note: Do not force a connector into a port. Wait at least 10 seconds before connecting the port. The SAS cable connectors are keyed; when oriented correctly into a SAS port, the connector clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector). a. Replace cables on Side A one cable at a time. The Side A cables are the cables connected to IOM A of each disk shelf. b. Verify that you have correctly replaced the SAS cables by entering the following command at the system console: sysconfig You can run the command from either controller. Note: It might take up to a minute for the system to complete discovery. The output should be the same as Step 1: the system should be Multi-Path HA, and the SAS port and attached disk shelf information should be the same. If the output is something other than Multi-Path HA, you must identify the cabling error, correct it, and run the sysconfig command again. c. Repeat substeps a and b for Side B. The Side B cables are the cables connected to IOM B of each disk shelf.

76 76 Hot-adding storage to the MetroCluster configuration You can add storage to MetroCluster configuration without bringing down the system. The procedure you use depends on the type of MetroCluster configuration and whether you are adding a single disk shelf to an existing stack or an entire stack. Choices Hot-adding a SAS disk shelf in a direct-attached MetroCluster configuration using SAS optical cables on page 76 Hot-adding SAS storage to a bridge-attached MetroCluster configuration on page 82 Hot-adding a SAS disk shelf in a direct-attached MetroCluster configuration using SAS optical cables You can use SAS optical cables to hot-add a SAS disk shelf to an existing stack of SAS disk shelves in a direct-attached configuration, or as a new stack to a SAS HBA or an onboard SAS port on the controller. Hot-adding a disk shelf involves installing, cabling, and verifying the disk drive and disk shelf firmware versions. Before you begin You must have verified that your storage system meets the requirements for the disk shelf (and disk drives) that you are hot-adding. You must have ordered and received the appropriate type, number, and length of SAS optical cables required for your configuration. You must have met the following requirements if you are hot-adding a single disk shelf or a stack of disk shelves directly to a system controller: Each controller in your storage system must have enough available PCI SAS HBA or onboard SAS ports. Each controller requires two SAS ports for each stack to which it is connected. For example, two stacks requires four SAS ports on each controller. You must have determined the controller port pairs you will be cabling. The Universal SAS and ACP Cabling Guide provides completed SAS port pair worksheets for common configurations and includes a SAS port pairs worksheet template if a completed worksheet is not provided for your configuration. SAS Disk Shelves Universal SAS and ACP Cabling Guide About this task This task applies to MetroCluster configurations in which the storage is connected directly to the storage controllers with SAS cables. It does not apply to MetroCluster configurations using FC-to-SAS bridges or FC switch fabrics. If you are hot-adding a disk shelf with SAS optical cables to a stack of disk shelves that is connected with SAS copper cables, you can temporarily have both cable types present in the stack. After hot-adding the disk shelf, you must replace the SAS copper cables for the rest of the shelfto-shelf connections in the stack and the shelf-to-controller connections from the first and last

77 Hot-adding storage to the MetroCluster configuration 77 disk shelf in the stack so that the stack meets the cabling rules for using SAS optical and SAS copper cables. This means that you must have ordered the appropriate number of SAS optical cables. You can use longer cables if needed to connect the hot-added shelf. This is a cable replacement and can be done nondisruptively for Multipath HA systems (in Step 1 of the Cabling the hot-added disk shelf procedure). If you are hot-adding a disk shelf to an existing stack, this procedure is written for hot-adding the disk shelf to the logical last disk shelf of the stack. If you are hot-adding more than one disk shelf, you must hot-add one at a time. If you are installing the disk shelf in an equipment rack or NetApp cabinet, you must install the two-post telco tray kit or four-post rail kit that came with your disk shelf. Installing a disk shelf for a hot-add Installing the new SAS disk shelf involves securing the disk shelf in a rack using the applicable twopost telco tray kit or the four-post rail kit and setting the disk shelf ID. About this task Disk shelves do not need to be grounded; grounding is done through the power cords. Steps 1. Properly ground yourself. 2. Install the two-post telco tray kit or the four-post rail kit for your disk shelf model using the installation flyer that came with the kit. Attention: If you are installing multiple disk shelves, you should install them from the bottom to the top of the rack for the best stability. Do not ear-mount the disk shelf into a telco-type rack; the disk shelf will collapse from the rack under its own weight. Attention: For two-post mid-mount installations, you must use the mid-mount brackets in addition to the two-post telco tray kit. 3. Install and secure the disk shelf onto the support brackets and rack. To make the disk shelf lighter and easier to maneuver, remove the power supplies and I/O modules (IOMs). Avoid removing the disk drives or carriers if possible because excessive handling can lead to internal damage. Caution: A fully populated DS4246 disk shelf can weigh approximately 110 lbs (49.9 kg). A fully populated DS2246 disk shelf can weigh approximately 49 lbs (22 kg). 4. Reinstall any power supplies and IOMs you removed to install your disk shelf into the rack. 5. Repeat Steps 3 and 4 for each disk shelf you are installing if you are adding multiple disk shelves. 6. Connect the power supplies for each disk shelf: a. Connect the power cords first to the disk shelves, securing them in place with the power cord retainer, and then to different power sources for resiliency. Note: If you have a disk shelf with four power supplies, connect power supplies in slots 1 and 3 to one power source and power supplies in slots 2 and 4 to a different power source.

78 78 MetroCluster Service Guide b. Turn on the power supplies for each disk shelf and wait for the disk drives to spin up. When the disk shelf has the maximum number of supported power supplies, all disk drives or carriers spin up at the same time. However, if one or two power supplies have faulted in a disk shelf with four power supplies, or if one power supply has faulted in a disk shelf with two power supplies, disk drives spin up in sets of six at 12-second intervals. 7. Change the shelf ID for each disk shelf you hot-added by completing the following substeps: You can verify IDs already in use by entering the sasadmin shelf command at the system console of either node. a. Change the shelf ID to a valid ID that is unique from the other SAS disk shelves in the storage system. b. Power-cycle the disk shelf to make the shelf ID take effect. See Changing the disk shelf ID in the DS4243, DS2246, DS4486, and DS4246 Disk Shelf Installation and Service Guide for more detailed instructions. Cabling the hot-added disk shelf Cabling the hot-added disk shelf involves cabling the SAS connections and, if applicable, assigning disk drive ownership. Before you begin You must have met the requirements given at the beginning of this document. You must have met all the requirements in the Hot-adding a SAS disk shelf using SAS optical cables section and installed your disk shelf. About this task This procedure is written with the assumption that you originally cabled your system so that the controllers connect to the last disk shelf in the stack through the disk shelf's circle ports instead of the square ports. Steps 1. Cable the SAS connections.

79 Hot-adding storage to the MetroCluster configuration 79 If you are cabling a disk shelf... To an existing stack of disk shelves Then... a. Disconnect the SAS cable from the I/O Module (IOM) A circle port on the last shelf in the stack. You can leave the other end of the cable connected to the controller to minimize confusion, or replace the cable with a longer cable if needed. b. Connect (daisy-chain) the IOM A circle port of the last disk shelf in the stack to the IOM A square port of the new disk shelf, using the SAS cables that came with the new disk shelf. c. Connect the cable that you removed in Substep a to the IOM A circle port of the new disk shelf. d. Verify that all cables are securely fastened. e. Repeat substep a through substep d for IOM B. The storage system recognizes the new disk shelf after all the drives spin up. To an existing SAS HBA or onboard SAS port a. Use the completed SAS port pair worksheet. b. Verify that all cables are securely fastened. 2. Verify SAS connectivity by completing the applicable substeps: You can run these commands from the system console of either node. a. Enter the following command to find out what the adapter name is: sasadmin expander_map b. Enter the following command to verify that all disk drives can be seen by the system: sasadmin shelf adapter_name The system displays a representation of your disk shelf populated with all the disk drives it sees. c. Enter the following command to verify that all IOMs (expanders) can be seen by the system (SAS channels and controller ports): sasadmin expander_map adapter_name The following example of output from this command shows that a single expander, IOM B (slot B), in shelf 3 (ID 3) is attached to port 4a (channel 4a) on the controller: Expanders on channel 4a: Level 1: WWN 500a c3f, ID 3, Serial Number 1006SZ00196, Product 'DS224IOM6 ', Rev '0134', Slot B 3. Check that your system has disk autoassignment enabled by entering the following command at the console of either controller: storage disk option show The autoassignment policy is shown in the Auto Assign column.

80 80 MetroCluster Service Guide Node BKg. FW. Upd. Auto Copy Auto Assign Auto Assign Policy node_a_1 on on on default node_a_2 on on on default 2 entries were displayed. 4. If your system does not have disk autoassignment enabled or if disk drives in the same stack are owned by both controllers, assign disk drives to the appropriate pools by using the information in the ONTAP 9.0 Disks and Aggregates Power Guide; otherwise, go to the next step. Note: If you are splitting a single stack of disk shelves between two controllers, you must disable disk autoassignment before assigning disk ownership; otherwise, when you assign any single disk drive, the remaining disk drives might be automatically assigned to the same controller and pool. To disable disk autoassignment, run the storage disk option modify -node nodename -autoassign off command at the console of the controllers. Note: Do not add disk drives to aggregates or volumes until after the disk drive and disk shelf firmware has been updated. 5. If you hot-added a disk shelf with SAS optical cables to a stack of disk shelves that are connected with SAS copper cables, replace the SAS copper cables for the rest of the shelf-to-shelf connections and the controller-to-shelf connections so that the stack meets the cabling rules stated at the beginning of this document; otherwise, go to the next step. You can use the cable replacement procedure in the Replacing SAS Cables in a multipath HA configuration section of this document. 6. Go to Verifying the disk drive and disk shelf firmware versions. Verifying the disk drive and disk shelf firmware versions Because Data ONTAP does not always automatically update the disk drive and disk shelf firmware on hot-added SAS disk shelves, you must verify that the disk drive and disk shelf firmware are the most current versions. If they are not, you must manually update the firmware. Steps 1. Check the console for a message containing dbfu.selected:info and text stating selected for background disk firmware update to determine whether or not you need to update the disk drive firmware. For example, actual output could look similar to Fri Jul 19 13:05:23 PDT [svt : bdfu.selected:info]: Disk svt-16g-sw4:4.126l64 [NETAPP X420_SFIRF300A10 NQ03] S/N [3SE0W R4SV] selected for background disk firmware. After assigning disk drives on the hot-added disk shelf, the disk drive firmware updates should have begun automatically on each disk drive with downrev firmware. A repeated message similar to what is shown above appears on the console every three to five minutes the time it takes to update downrev firmware on a disk drive showing the firmware update progress. If... There is similar output Then... Go to the next step. Disk drives with downrev firmware have been detected and the firmware is being updated automatically.

81 Hot-adding storage to the MetroCluster configuration 81 If... There is no similar output Then... Wait for an hourly message on the console and take the applicable action: If there is no message on the console about disk drive firmware, go to the next step. The disk drive firmware is current and no action is needed. If there is a message containing disk.fw.downrevwarning and text stating disks have downrev firmware that you need to update, update the disk drive firmware to the most current version using the applicable command at the advanced privilege level: For 7-Mode: disk_fw_update For clustered Data ONTAP from the clustershell: storage disk firmware update Attention: These commands manually forces a disk drive firmware update, which is disruptive to client access and can take up to ten minutes for each disk drive. For example, actual output could look similar to: Sun May 5 04:00:01 PDT [svt : disk.fw.downrevwarning:warning]: 1 disks have downrev firmware that you need to update. 2. Verify that the disk shelf firmware is the most current version: a. Enter the following command at the storage system console: sasadmin expander_map b. Locate the disk shelf firmware information for the hot-added disk shelf in the output is the disk shelf firmware version for shelf number one (Slot A/IOM A) in the storage system: Expanders on channel 4a: Level 3: WWN 500a ff, ID 1, Serial Number ' SHU G114C', Product 'DS424IOM6 ', Rev '0151', Slot A c. Compare the firmware information in the command output with the disk shelf firmware information at NetApp Downloads: Disk Shelf Firmware to determine the most current disk shelf firmware version. 3. The next step depends on how current the disk shelf firmware is. If the firmware version in the command output is... The same as or later than the most current version on the NetApp Support Site An earlier version than the most current version on the NetApp Support Site Then... No disk shelf firmware update is needed. Download the disk shelf firmware file by using the procedure at NetApp Downloads: Disk Shelf Firmware. You can run the commands from either node.

82 82 MetroCluster Service Guide Hot-adding SAS storage to a bridge-attached MetroCluster configuration You can hot-add either an individual SAS disk shelf or a stack and set of bridges to an existing MetroCluster system. These procedures apply to MetroCluster configurations using FC-to-SAS bridges that are either directly attached to the storage controllers or attached to an FC fabric. Choices Hot-adding a stack of SAS disk shelves to an existing pair of FibreBridge 7500N bridges on page 82 Hot-adding a stack of SAS disk shelves and bridges to a MetroCluster system on page 88 Hot-adding a SAS disk shelf to a stack of SAS disk shelves on page 96 Hot-adding a stack of SAS disk shelves to an existing pair of FibreBridge 7500N bridges You can hot-add a stack of SAS disk shelves to an existing pair of FibreBridge 7500N bridges that have available ports. Before you begin Download the latest disk and disk shelf firmware. Verify that all disk shelves in the MetroCluster configuration (both the new shelves and existing shelves) are running the same firmware version. 1. Download the disk firmware and make a copy of the disk firmware instructions for reference later. NetApp Downloads: Disk Drive and Firmware 2. Download the disk shelf firmware and make a copy of the disk shelf firmware instructions for reference later. NetApp Downloads: Disk Shelf Firmware About this task This procedure applies only to existing FibreBridge 7500N bridges that are already connected and have available SAS ports. This procedure is written with the assumption that you are using the recommended bridge management interfaces: the ATTO ExpressNAV GUI and the ATTO QuickNAV utility. You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge firmware. You use the ATTO QuickNAV utility to configure the bridge Ethernet management 1 port. You can use other management interfaces, if needed. These options include using a serial port or Telnet to configure and manage a bridge, and to configure the Ethernet management 1 port, and using FTP to update the bridge firmware. If you choose any of these management interfaces, you must meet the applicable requirements in Other bridge management interfaces on page 40. Attention: When you remove a cable from a SAS port, you must wait at least 70 seconds before plugging the cable into a different SAS port. If you fail to do so, the system will not recognize that the cable has been moved to another port.

83 Hot-adding storage to the MetroCluster configuration 83 Steps 1. Properly ground yourself. 2. Verify that your system has disk autoassignment enabled from the console of either controller: storage disk option show The status of disk autoassignment is shown in the Auto Assign column. Node BKg. FW. Upd. Auto Copy Auto Assign Auto Assign Policy node_a_1 on on on default node_a_2 on on on default 2 entries were displayed. 3. Disable the switch ports for the new stack. 4. Update the firmware on each bridge to the latest version by following the instructions starting with Step 4 on the FibreBridge Download page. 5. On each bridge in the pair, enable the SAS port that will connect to the new stack: SASPortEnable port-letter The same SAS port (B, C, or D) must be used on both bridges. 6. Cable the disk shelves to the bridges: a. Daisy-chain the disk shelves in each stack. For information about daisy-chaining disk shelves, see the Installation and Service Guide for your disk shelf model. b. For each stack of disk shelves, cable IOM A square port of the first shelf to SAS port A on FibreBridge A. c. For each stack of disk shelves, cable IOM B circle port of the last shelf to SAS port A on FibreBridge B. Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack through the first shelf, and bridge B connects to the B-side of the stack through the last shelf. Note: The bridge SAS port B is disabled. The following illustration shows a pair of bridges cabled to a stack or stacks of three disk shelves:

84 84 MetroCluster Service Guide FC1 FC2 FC_bridge_x_1_06 FC1 FC2 FC_bridge_x_2_o6 M1 SAS A M1 SAS A Stack of SAS shelves First shelf IOM A IOM B Last shelf ATTO 7500N bridge FC_bridge_A_1a M1 FC1 FC2 SAS A SAS B SAS C SAS D IOM A IOM B IOM A IOM B IOM A IOM B IOM A IOM B SAS A SAS B SAS C SAS D M1 FC1 FC2 ATTO 7500N bridge FC_bridge_A_1b 7. Verify that each bridge can detect all disk drives and disk shelves it is connected to. If you are using the... ATTO ExpressNAV GUI Then... a. In a supported web browser, enter the IP address of a bridge in the browser box. You are brought to the ATTO FibreBridge home page, which has a link. b. Click the link, and then enter your user name and the password that you designated when you configured the bridge. The ATTO FibreBridge status page appears with a menu to the left. c. Click Advanced in the menu. d. Enter the following command, and then click Submit: sastargets

85 Hot-adding storage to the MetroCluster configuration 85 If you are using the... Serial port connection Then... Enter the following command: sastargets The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentially numbered so that you can quickly count the devices. For example, the following output shows that 10 disks are connected: Tgt VendorID ProductID Type SerialNumber 0 NETAPP X410_S15K6288A15 DISK 3QP1CLE UHJV 1 NETAPP X410_S15K6288A15 DISK 3QP1ELF V1BV 2 NETAPP X410_S15K6288A15 DISK 3QP1G3EW U2M0 3 NETAPP X410_S15K6288A15 DISK 3QP1EWMP U1X5 4 NETAPP X410_S15K6288A15 DISK 3QP1FZLE G8YU 5 NETAPP X410_S15K6288A15 DISK 3QP1FZLF TZKZ 6 NETAPP X410_S15K6288A15 DISK 3QP1CEB MGXL 7 NETAPP X410_S15K6288A15 DISK 3QP1G7A FNTT 8 NETAPP X410_S15K6288A15 DISK 3QP1FY0T G8PA 9 NETAPP X410_S15K6288A15 DISK 3QP1FXW VERQ Note: If the text response truncated appears at the beginning of the output, you can use Telnet to connect to the bridge and enter the same command to see all of the output. 8. Verify that the command output shows that the bridge is connected to all of the appropriate disks and disk shelves in the stack. If the output is... Correct Not correct Then... Repeat step Step 9 on page 92 for each remaining bridge. a. Check for loose SAS cables or correct the SAS cabling by repeating step Step 5 on page 90. b. Repeat step Step 9 on page Cable each bridge in the first switch fabric to the FC switches:

86 86 MetroCluster Service Guide This illustration shows Brocade 6505 switches. Brocade 6510 switches use ports 8 and 9 for ISL connections. Bridges connecting to switch_a_1 Bridge 1 Bridge 2 Bridge 3 Bridge 4 FC1 FC1 FC1 FC1 switch_a_1 Brocade switch_a_2 Brocade FC1 FC1 FC1 FC1 Bridge 1 Bridge 2 Bridge 3 Bridge 4 Bridges connecting to switch_a_2 Note: The Brocade and Cisco switches use different port numbering: On Brocade switches, the first port is numbered 0. On Cisco switches, the first port is numbered 1. This is reflected in the tables that follow. Cable bridges to these ports on FC_switch_x_1... Brocade 6510 Brocade 6505 Cisco 9148/9148S Cable each bridge in the second switch fabric to the FC switches: Cable bridges to these ports on FC_switch_x_2... Brocade 6510 Brocade 6505 Cisco 9148/9148S

87 Hot-adding storage to the MetroCluster configuration 87 Cable bridges to these ports on FC_switch_x_2... Brocade 6510 Brocade 6505 Cisco 9148/9148S Update the disk drive firmware to the most current version from the system console: disk_fw_update You must run this command on both controllers. If needed, see the instructions you copied when you downloaded the firmware as part of the preparation for this task. 12. Update the disk shelf firmware to the most current version by using the instructions you copied when you downloaded the firmware as part of the preparation for this task. You can run the commands in the procedure from the system console of either controller. 13. If your system does not have disk autoassignment enabled, assign disk drive ownership. ONTAP 9.0 Disks and Aggregates Power Guide Note: If you are splitting the ownership of a single stack of disk shelves between multiple controllers, you must disable disk autoassignment (storage disk option modify - autoassign off * from both nodes in the cluster) before assigning disk ownership; otherwise, when you assign any single disk drive, the remaining disk drives might be automatically assigned to the same controller and pool. Note: Do not add disk drives to aggregates or volumes until after the disk drive and disk shelf firmware has been updated and verification steps in this task have been completed. 14. Enable the switch ports for the new stack. 15. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered. 16. If applicable, repeat this procedure for the partner site.

88 88 MetroCluster Service Guide Hot-adding a stack of SAS disk shelves and bridges to a MetroCluster system When there are no ports available on an existing FibreBridge 7500N bridge, you can hot-add a pair of 7500N bridges and a stack of SAS disk shelves to a MetroCluster system. There must be available ports on the FC switches and you must update switch zoning to reflect the changes. About this task This procedure is written with the assumption that you are using the recommended bridge management interfaces: the ATTO ExpressNAV GUI and the ATTO QuickNAV utility. You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge firmware. You use the ATTO QuickNAV utility to configure the bridge Ethernet management 1 port. You can use other management interfaces, if needed. These options include using a serial port or Telnet to configure and manage a bridge, and to configure the Ethernet management 1 port, and using FTP to update the bridge firmware. If you choose any of these management interfaces, your system must meet the applicable requirements in Other bridge management interfaces on page 40. Attention: When you remove a cable from a SAS port, you must wait at least 70 seconds before plugging the cable into a different SAS port. If you fail to do so, the system will not recognize that the cable has been moved to another port. Preparing to hot-add a stack of SAS disk shelves and bridges Preparing to hot-add a stack of SAS disk shelves and a pair of bridges involves downloading documents as well as the disk drive and disk shelf firmware. Before you begin Your system must be a supported configuration and must be running a supported version of Data ONTAP. All disk drives and disk shelves in the system must be running the latest firmware version. You might want to update the disk and shelf firmware throughout the MetroCluster configuration prior to adding shelves. ONTAP 9.0 Upgrade and Revert/Downgrade Guide Each FC switch must have one FC port available for one bridge to connect to it. Note: You might need to upgrade the FC switch depending on the FC switch compatibility. The computer you are using to set up the bridges must be running an ATTO supported web browser to use the ATTO ExpressNAV GUI: Internet Explorer 8 or 9, or Mozilla Firefox 3. The ATTO Product Release Notes have an up-to-date list of supported web browsers. You can access this document using the information in the steps. Steps 1. Download or view the following documents from the NetApp Support Site: NetApp Interoperability Matrix Tool The Installation and Service Guide for your disk shelf model. NetApp Documentation: Disk Shelves 2. Download content from the ATTO website and from the NetApp website: a. Go to the ATTO FibreBridge Description page.

89 Hot-adding storage to the MetroCluster configuration 89 b. Using the link on the ATTO FibreBridge Description page, access the ATTO web site and download the following: ATTO FibreBridge Installation and Operation Manual for your bridge model. ATTO QuickNAV utility (to the computer you are using for setup). c. Go to the ATTO FibreBridge Firmware Download page by clicking Continue at the end of the ATTO FibreBridge Description page, and then do the following: Download the bridge firmware file using Steps 1 through 3 of that procedure. In this step, you are only completing the download portion of the instructions provided in the links. You update the firmware on each bridge later, when instructed to do so in the Hot-adding the stack of shelves on page 90 section. Make a copy of the ATTO FibreBridge Firmware Download page and release notes for reference later. 3. Verify that the disk shelf you are hot-adding is supported. NetApp Interoperability Matrix Tool 4. Download the latest disk and disk shelf firmware, and make a copy of the installation portion of the instructions for reference later. All disk shelves in the MetroCluster configuration (both the new shelves and existing shelves) must be running the same firmware version. Note: In this step, you are only completing the download portion of the instructions provided in the links and making a copy of the installation instructions. You update the firmware on each disk and disk shelf later, when instructed to do so in the Hot-adding the stack of shelves on page 90 section. a. Download the disk firmware and make a copy of the disk firmware instructions for reference later. NetApp Downloads: Disk Drive and Firmware b. Download the disk shelf firmware and make a copy of the disk shelf firmware instructions for reference later. NetApp Downloads: Disk Shelf Firmware 5. Gather the hardware and information needed to use the recommended bridge management interfaces the ATTO ExpressNAV GUI and ATTO QuickNAV utility: a. Acquire a standard Ethernet cable to connect from the bridge Ethernet management 1 port to your network. b. Determine a non-default user name and password for accessing the bridges. It is recommended that you change the default user name and password. c. Obtain an IP address, subnet mask, and gateway information for the Ethernet management 1 port on each bridge. d. Disable VPN clients on the computer you are using for setup. Active VPN clients cause the QuickNAV scan for bridges to fail. 6. Acquire four screws for each bridge to flush-mount the bridge L brackets securely to the front of the rack. The openings in the bridge L brackets are compliant with rack standard ETA-310-X for 19-inch (482.6 mm) racks.

90 90 MetroCluster Service Guide 7. If necessary, update the FC switch zoning to accommodate the new bridges that are being added to the configuration. If you are using the Reference Configuration Files provided by NetApp, the zones have been created for all ports, so you do not need to make any zoning updates. There must be a storage zone for each switch port that connects to the FC ports of the bridge. Hot-adding the stack of SAS disk shelves and bridges You can hot-add a stack of SAS disk shelves and bridges to increase the capacity of the bridges. Before you begin The system must meet all the requirements in Preparing to hot-add a stack of SAS disk shelves and bridges on page 88. About this task This is a nondisruptive procedure if all interoperability requirements are met. NetApp Interoperability Matrix Tool Multipath HA is the only supported configuration for MetroCluster systems using bridges. Both controllers must have access through the bridges to the disk shelves in each stack. You should hot-add an equal number of disk shelves at each site. Attention: When you remove a cable from a SAS port, you must wait at least 70 seconds before plugging the cable into a different SAS port. If you fail to do so, the system will not recognize that the cable has been moved to another port. The following illustrations show an example of the addition of a stack of shelves to a pair of FibreBridge 7500N bridges. Steps 1. Properly ground yourself. 2. Check whether your system has disk autoassignment enabled from the console of either controller: storage disk option show The Auto Assign column indicates whether or not autoassignment is enabled. Node BKg. FW. Upd. Auto Copy Auto Assign Auto Assign Policy node_a_1 on on on default node_a_2 on on on default 2 entries were displayed. 3. Disable the switch ports for the new stack. 4. Connect the Ethernet management 1 port on each bridge to your network using an Ethernet cable. The Ethernet management 1 port enables you to quickly download the bridge firmware (using ATTO ExpressNAV or FTP management interfaces) and to retrieve core files and extract logs. 5. Configure the Ethernet management 1 port for each bridge by following the procedure in the ATTO FibreBridge Installation and Operation Manual, section 2.0 of your bridge model. Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that is connected by the Ethernet cable is configured. For example, if you

91 Hot-adding storage to the MetroCluster configuration 91 also wanted to configure the Ethernet management 2 port, you would need to connect the Ethernet cable to port 2 and run QuickNAV. 6. Configure the bridges. Be sure to make note of the user name and password that you designate. The ATTO FibreBridge Installation and Operation Manual for your bridge model has the most current information on available commands and how to use them. a. Configure the IP settings of the bridge. To set the IP address without the Quicknav utility, you need to have a serial connection to the FibreBridge. If using the CLI, issue the following commands: set ipaddress mp1 ip-address set ipsubnetmask mp1 subnet-mask set ipgateway mp1 x.x.x.x set ipdhcp mp1 b. Configure the bridge name. The bridges should each have a unique name within the MetroCluster configuration. bridge names for one stack group on each site: bridge_a_1a bridge_a_1b bridge_b_1a bridge_b_1b If using the CLI, issue the following command: set bridgename bridgename c. Enable SNMP on the bridge: set SNMP enabled 7. Configure the bridge FC ports. a. Configure the data rate/speed of the bridge FC ports. Note: Set the FCDataRate speed to the maximum speed supported by the FC port of the FC switch or the controller module to which the bridge port connects. The most current information on supported distance can be found in the Interoperability Matrix. NetApp Interoperability Matrix Tool After opening the Interoperability Matrix, you can use the Storage Solution field to select your MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray). You use the Component Explorer to select the components and ONTAP version to refine your search. You can click Show Results to display the list of supported configurations that match the criteria.

92 92 MetroCluster Service Guide If using the CLI, issue the following command: set FCDataRate port-number 16Gb b. Configure the connection mode that the port uses to communicate across the FC network. If you have a fabric-attached MetroCluster system, you must set the bridge connection mode to ptp (point-to-point). If you have a stretch MetroCluster system, you must set the bridge connection mode depending on the adapter that the bridge connects to: For 16-Gbps capable adapters, set the connection mode to ptp, even if it is operating at a lower speed. For 8-Gbps and 4-Gbps capable adapters, set the connection mode to loop. If using the CLI, issue the following command: set FCConnMode port-number ptp-or-loop c. If you are configuring an ATTO 7500N bridge and using the second port, repeat the previous substeps for the FC2 port. 8. Save the bridge's configuration. If using the CLI, issue the following command: SaveConfiguration Restart You are prompted to restart the bridge. 9. Update the firmware on each bridge to the latest version by following the instructions starting with Step 4 on the FibreBridge Download page. 10. Cable the disk shelves to the bridges: a. Daisy-chain the disk shelves in each stack. For information about daisy-chaining disk shelves, see the Installation and Service Guide for your disk shelf model. b. For each stack of disk shelves, cable IOM A square port of the first shelf to SAS port A on FibreBridge A. c. For each stack of disk shelves, cable IOM B circle port of the last shelf to SAS port A on FibreBridge B. Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack through the first shelf, and bridge B connects to the B-side of the stack through the last shelf. Note: The bridge SAS port B is disabled. The following illustration shows a set of bridges cabled to a stack of three disk shelves:

93 Hot-adding storage to the MetroCluster configuration 93 FC1 FC2 FC_bridge_x_1_06 FC1 FC2 FC_bridge_x_2_o6 M1 SAS A M1 SAS A Stack of SAS shelves First shelf IOM A IOM B Last shelf 11. Verify that each bridge can detect all of the disk drives and disk shelves it is connected to. If you are using the... ATTO ExpressNAV GUI Then... a. In a supported web browser, enter the IP address of a bridge in the browser box. You are brought to the ATTO FibreBridge home page, which has a link. b. Click the link, and then enter your user name and the password that you designated when you configured the bridge. The ATTO FibreBridge status page appears with a menu to the left. c. Click Advanced in the menu. d. Enter the following command, and then click Submit: sastargets The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentially numbered so that you can quickly count the devices. Serial port connection Enter the following command: sastargets The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentially numbered so that you can quickly count the devices. The following output shows that 10 disks are connected: Tgt VendorID ProductID Type SerialNumber 0 NETAPP X410_S15K6288A15 DISK 3QP1CLE UHJV 1 NETAPP X410_S15K6288A15 DISK 3QP1ELF V1BV 2 NETAPP X410_S15K6288A15 DISK 3QP1G3EW U2M0 3 NETAPP X410_S15K6288A15 DISK 3QP1EWMP U1X5 4 NETAPP X410_S15K6288A15 DISK 3QP1FZLE G8YU 5 NETAPP X410_S15K6288A15 DISK 3QP1FZLF TZKZ

94 94 MetroCluster Service Guide 6 NETAPP X410_S15K6288A15 DISK 3QP1CEB MGXL 7 NETAPP X410_S15K6288A15 DISK 3QP1G7A FNTT 8 NETAPP X410_S15K6288A15 DISK 3QP1FY0T G8PA 9 NETAPP X410_S15K6288A15 DISK 3QP1FXW VERQ Note: If the text response truncated appears at the beginning of the output, you can use Telnet to connect to the bridge and enter the same command to see all the output. 12. Verify that the command output shows that the bridge is connected to all of the appropriate disks and disk shelves in the stack. If the output is... Correct Not correct Then... Repeat step 9 on page 92 for each remaining bridge. a. Check for loose SAS cables or correct the SAS cabling by repeating step 5 on page 90. b. Repeat step 9 on page In case of fabric-attached MetroCluster system, cable each bridge in the first switch fabric to the FC switches: This illustration shows Brocade 6505 switches. Brocade 6505 switches use ports 8 and 9 for ISL connections. Bridges connecting to switch_a_1 Bridge 1 Bridge 2 Bridge 3 Bridge 4 FC1 FC1 FC1 FC1 switch_a_1 Brocade switch_a_2 Brocade FC1 FC1 FC1 FC1 Bridge 1 Bridge 2 Bridge 3 Bridge 4 Bridges connecting to switch_a_2 Note: The Brocade and Cisco switches use different port numbering, as reflected in the tables that follow:

95 Hot-adding storage to the MetroCluster configuration 95 On Brocade switches, the first port is numbered 0. On Cisco switches, the first port is numbered 1. Cable bridges to these ports on FC_switch_x_1... Brocade 6510 Brocade 6505 Cisco 9148/9148S/9250i Cable each bridge in the second switch fabric to the FC switches: Cable bridges to these ports on FC_switch_x_2... Brocade 6510 Brocade 6505 Cisco 9148/9148S/9250i If you are using a bridge-attached MetroCluster system, cable each bridge to the controllers: a. Cable FC port 1 of the bridge to an 16-Gb or 8-Gb FC port on the controller in cluster_a. b. Cable FC port 2 of the bridge to the same speed FC port of the controller in cluster_a. 16. Repeat the previous step on other subsequent bridges too until all bridges have been cabled. 17. Update the disk drive firmware to the most current version from the system console: disk_fw_update You must run this command on both controllers. If needed, see the instructions you copied when you downloaded the firmware as part of the preparation for this task. 18. Update the disk shelf firmware to the most current version by using the instructions you copied when you downloaded the firmware as part of the preparation for this task. You can run the commands in the procedure from the system console of either controller. 19. If your system does not have disk autoassignment enabled, assign disk drive ownership. ONTAP 9.0 Disks and Aggregates Power Guide Note: If you are splitting the ownership of a single stack of disk shelves between multiple controllers, you must disable disk autoassignment on both nodes in the cluster (storage disk option modify -autoassign off *) before assigning disk ownership; otherwise, when you assign any single disk drive, the remaining disk drives might be automatically assigned to the same controller and pool. Note: Do not add disk drives to aggregates or volumes until after the disk drive and disk shelf firmware have been updated and verification steps in this task have been completed. 20. Enable the switch ports for the new stack.

96 96 MetroCluster Service Guide 21. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered. 22. If applicable, repeat this procedure for the partner site. Hot-adding a SAS disk shelf to a stack of SAS disk shelves You can hot-add one or more SAS disk shelves to a stack of SAS disk shelves. Preparing to hot-add SAS disk shelves Preparing to hot-add a SAS disk shelf involves downloading documents as well as the disk drive and disk shelf firmware. Before you begin Your system must be a supported configuration and must be running a supported version of Data ONTAP. All disk drives and disk shelves in the system must be running the latest firmware version. You might want to update the disk and shelf firmware throughout the MetroCluster configuration prior to adding shelves. ONTAP 9.0 Upgrade and Revert/Downgrade Guide Steps 1. Download or view the following documents from the NetApp Support Site: NetApp Interoperability Matrix Tool The Installation and Service Guide for your disk shelf model. NetApp Documentation: Disk Shelves 2. Verify that the disk shelf you are hot-adding is supported. NetApp Interoperability Matrix Tool 3. Download the latest disk and disk shelf firmware: Note: In this step, you are only completing the download portion of the instructions provided in the links. You need to follow the steps found in the Hot-adding a disk shelf on page 97 section for installing the disk shelf.

97 Hot-adding storage to the MetroCluster configuration 97 a. Download the disk firmware and make a copy of the disk firmware instructions for reference later. NetApp Downloads: Disk Drive and Firmware b. Download the disk shelf firmware and make a copy of the disk shelf firmware instructions for reference later. NetApp Downloads: Disk Shelf Firmware Hot-adding a disk shelf You can hot-add a disk shelf when you want to increase storage without any reduction in performance. Before you begin The system must meet all of the requirements in Preparing to hot-add SAS disk shelves on page 96. About this task This procedure is for hot-adding a disk shelf to the last disk shelf in a stack. This procedure is written with the assumption that the last disk shelf in a stack is connected from IOM A to bridge A and from IOM B to bridge B. Your environment must meet one of the following scenarios to hot-add a shelf: You have two FibreBridge 7500N bridges connected to a stack of SAS disk shelves. You have one FibreBridge 7500N bridge and one FibreBridge 6500N bridge connected to a stack of SAS disk shelves. You have two FibreBridge 6500N bridges connected to a stack of SAS disk shelves. This is a nondisruptive procedure. You should hot-add an equal number of disk shelves at each site. If you are hot-adding more than one disk shelf, you must hot-add one disk shelf at a time. Note: Each pair of FibreBridge 7500N bridges can support up to four stacks and a maximum of forty storage shelves. Attention: Hot-adding a disk shelf requires you to update the disk drive firmware on the hot-added disk shelf by running the disk_fw_update command. Running this command can be disruptive if the firmware on existing disk drives in your system is an older version. Attention: When you remove a cable from a SAS port, you must wait at least 70 seconds before plugging the cable into a different SAS port. If you fail to do so, the system will not recognize that the cable has been moved to another port. Steps 1. Properly ground yourself. 2. Verify disk shelf connectivity from the system console of either controller: sysconfig -v The output shows the following: Each bridge on a separate line and under each FC port to which it is visible. For example, hot-adding a disk shelf to a set of FibreBridge 7500N bridges shows the following output:

98 98 MetroCluster Service Guide FC-to-SAS Bridge: cisco_a_1-1:9.126l0: ATTO FibreBridge7500N 2.10 FB7500N cisco_a_1-2:1.126l0: ATTO FibreBridge7500N 2.10 FB7500N Each disk shelf on a separate line under each FC port to which it is visible. For example: Shelf 0: IOM6 Firmware rev. IOM6 A: 0173 IOM6 B: 0173 Shelf 1: IOM6 Firmware rev. IOM6 A: 0173 IOM6 B: 0173 Note: The output might differ based on the version of Data ONTAP your system is running. Each disk drive on a separate line under each FC port to which it is visible. For example: cisco_a_1-1:9.126l1 : NETAPP X421_HCOBD450A10 NA GB ( B/sect) cisco_a_1-1:9.126l2 : NETAPP X421_HCOBD450A10 NA GB ( B/sect) 3. Check whether your system has disk autoassignment enabled from the console of either controller: storage disk option show The autoassignment policy is shown in the Auto Assign column. Node BKg. FW. Upd. Auto Copy Auto Assign Auto Assign Policy node_a_1 on on on default node_a_2 on on on default 2 entries were displayed. 4. If your system does not have disk autoassignment enabled or if disk drives in the same stack are owned by both controllers, assign disk drives to the appropriate pools by using the information in the ONTAP 9.0 Disks and Aggregates Power Guide; otherwise, go to the next step. Note: If you are splitting a single stack of disk shelves between two controllers, you must disable disk autoassignment before assigning disk ownership; otherwise, when you assign any single disk drive, the remaining disk drives might be automatically assigned to the same controller and pool. To disable disk autoassignment, run the storage disk option modify -node nodename -autoassign off command at the console of the controllers. Note: Do not add disk drives to aggregates or volumes until after the disk drive and disk shelf firmware have been updated. 5. Update the disk shelf firmware to the most current version by using the instructions you copied when you downloaded the firmware as part of the preparation for this task. You can run the commands in the procedure from the system console of either controller. 6. Install and cable the disk shelf: If your stack has both FibreBridge 6500N bridges and FibreBridge 7500N bridges: For FibreBridge 6500N bridges:

99 Hot-adding storage to the MetroCluster configuration 99 Note: Do not force a connector into a port. Wait at least 10 seconds before connecting the port. The SAS cable connectors are keyed; when oriented correctly into a SAS port, the connector clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector). For FibreBridge 7500N bridges: Note: Do not force a connector into a port. The mini-sas cables are keyed; when oriented correctly into a SAS port, the SAS cable clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented up (on the topside of the connector). a. Install the disk shelf, power it on, and set the shelf ID. For detailed steps, see the SAS Disk Shelves Installation and Service Guide for DS4243, DS2246, DS4486, and DS4246. Note: You must power-cycle the disk shelf and keep the shelf IDs unique for each SAS disk shelf within the entire storage system. b. Disconnect the SAS cable from the IOM B circle port of the last shelf in the stack, and then reconnect it to the same port in the new shelf. The other end of this cable remains connected to bridge B. c. Daisy-chain the new disk shelf by cabling the new shelf IOM square ports (of IOM A and IOM B) to the last shelf IOM circle ports (of IOM A and IOM B). For an example of a cabled stack of disk shelves, you can reference any of the illustrations in this document. 7. Update the disk drive firmware to the most current version from the system console: disk_fw_update You must run this command on both controllers. If needed, see the instructions you copied when you downloaded the firmware as part of the preparation for this task in the Preparing for the hot-add section. 8. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered. 9. If you are hot-adding more than one disk shelf, repeat the steps, starting with step 2 on page 97, for each disk shelf that you are hot-adding.

100 100 Hot-removing storage from the MetroCluster configuration You can remove storage from the MetroCluster configuration without bringing down the system. The procedure you use depends on how the storage is attached in the MetroCluster configuration. Choices Hot-removing a direct-attached disk shelf on page 100 Hot-removing a disk shelf that is attached to an FC-to-SAS bridge on page 104 Hot-removing a direct-attached disk shelf You can hot-remove disk shelves physically remove disk shelves that have had the aggregates removed from the disk drives from a MetroCluster configuration that is up and serving data. You can hot-remove one or more disk shelves from anywhere within a stack of disk shelves, or you can remove a stack of disk shelves. You must use a specific procedure when the disk shelves are directly attached to the storage controllers with SAS cables. Before you begin Your storage system must be a multipath HA system. You must have already removed all aggregates from the disk drives in the disk shelves you are removing. Attention: If you attempt this procedure on non-metrocluster configurations with aggregates on the disk shelf you are removing, you could cause the system to fail with a multidisk panic. Removing aggregates involves splitting the mirrored aggregates on the disk shelves you are removing and then re-creating the mirrored aggregates with another set of disk drives. ONTAP 9.0 Disks and Aggregates Power Guide You must remove disk drive ownership after you remove the aggregates from the disk drives in the disk shelves you are removing. Note: The procedure for removing ownership from disk drives requires you to disable disk autoassignment. You reenable disk autoassignment when prompted at the end of this shelf hotremove procedure. ONTAP 9.0 Disks and Aggregates Power Guide If you are removing one or more disk shelves from within a stack, you must have factored the distance to bypass the disk shelves you are removing; therefore, if the current cables are not long enough, you need to have longer cables available. About this task This task applies only to direct-attached MetroCluster configurations, in which the storage shelves are directly connected to the storage controllers with SAS cables. This procedure follows the cabling rules in the Universal SAS and ACP Cabling Guide; therefore, references to IOMs and IOM SAS and ACP square and circle ports align with the cabling rules. If your multipath HA or single-controller-dual-path (multipath) configuration is cabled differently from what is prescribed in the cabling guide, the IOM and/or IOM ports might be different.

101 Hot-removing storage from the MetroCluster configuration 101 Path A refers to the A-side disk shelf I/O module (IOM A) located in the top or left of the disk shelf depending on your disk shelf model. Path B refers to the B-side disk shelf I/O module (IOM B) located in the bottom or right of the disk shelf depending on your disk shelf model. The first disk shelf in the stack is the disk shelf with the SAS square ports directly connected to the controllers. The interim disk shelf in the stack is the disk shelf directly connected to other disk shelves in the stack. The last disk shelf in the stack is the disk shelf with SAS circle ports directly connected to the controllers. The next disk shelf is the disk shelf downstream of the disk shelf being removed, in depth order. The previous disk shelf is the disk shelf upstream of the disk shelf being removed, in depth order. After completing this procedure for the local site, you are instructed to repeat this procedure for the remote site if you are also removing disk shelves for the remote site. Steps 1. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check: metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered. 2. Verify that your system configuration is Multi-Path HA: run local sysconfig It might take up to a minute for the system to complete discovery. The configuration is listed in the System Storage Configuration field. Attention: If your system shows something other than Multi-Path HA, you cannot continue with this procedure. Your system must meet the prerequisites stated in the Before you begin section of this procedure. 3. Verify that ownership is removed from the disk drives on the disk shelves you are removing: a. Enter the following command at the console of either controller: storage disk show -shelf shelf_number b. Check the output and take the appropriate action:

102 102 MetroCluster Service Guide If the output shows... unassigned or broken Any disk drives in the disk shelves you are removing have ownership Then do this... Go to the next step. The disk drives in the disk shelves you are removing are in the correct state. See the ONTAP 9.0 Disks and Aggregates Power Guide for information about removing disk ownership. The following output for the storage disk show -shelf3 command shows that disk drives on the disk shelf being removed (disk shelf 3) have the ownership removed; therefore, unassigned appears in the Container Type column. Two disk drives are failed; therefore, broken appears in the Container Type column: fas6200::> storage disk show -shelf 3 Usable Container Disk Size Shelf Bay Type Position Aggregate Owner ie6200-1:0a unassigned present - - ie6200-1:0a unassigned present - - ie6200-1:0a broken present - - ie6200-1:0a broken data Turn on the LEDs for each disk shelf you are removing so that the disk shelves are physically identifiable, by entering the following command in the node shell for each disk shelf you are removing: storage shelf identify adapter_name.shelfshelf_id on You must be in advanced privilege level to enter this command. You can enter the command from the console of either controller. All LEDs on the front and back of the specified disk shelves flash. LEDs continue to flash for up to five minutes. Note: You need to be certain of which disk shelves you are removing so that you can correctly recable path A and path B later in this procedure. Even if you are removing an entire stack of disk shelves, it is recommended that you use this command. To physically identify disk shelf 3, you enter storage shelf identify 0b.shelf3 on. 5. If you are removing an entire stack of disk shelves, complete the following substeps; otherwise, go to the next step: a. Remove all SAS cables on path A and path B. This includes controller-to-shelf cables and shelf-to-shelf cables for all disk shelves in the stack you are removing. b. Go to 9 on page If you are removing one or more disk shelves from a stack (but keeping the stack), recable the applicable path A stack connections to bypass the disk shelves you are removing by completing the applicable set of substeps:

103 Hot-removing storage from the MetroCluster configuration 103 If you are removing more than one disk shelf, complete the applicable set of substeps one disk shelf at a time. If you need a graphical system cabling reference, use the platform specific Installation and Setup Instructions document that ships with each platform. Installation and Setup Instructions FAS8040/FAS8060 Systems Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules Installation and Setup Instructions FAS8020 systems Installation and Setup Instructions 62xx Systems Installation and Setup Instructions 32xx Systems If you are removing... The first disk shelf in a stack Then do this... a. Remove the cable connecting the IOM A SAS circle port of the first disk shelf and the IOM A SAS square port of the second disk shelf in the stack and set it aside. b. Move the cable connecting the controller to the IOM A SAS square port of the first disk shelf to the IOM A SAS square port of the second disk shelf in the stack An interim disk shelf in a stack a. Remove the cable connecting the IOM A SAS circle port of the disk shelf being removed and the IOM A SAS square port of the next disk shelf in the stack and set it aside. b. Move the cable connecting the IOM A SAS square port of the disk shelf being removed to the IOM A SAS square port of the next disk shelf in the stack The last disk shelf in a stack a. Remove the cable connecting the IOM A SAS square port of the last disk shelf and the IOM A SAS circle port of the previous disk shelf in the stack and set it aside. b. Move the cable connecting the controller to the IOM A SAS circle port of the last disk shelf to the IOM A SAS circle port of the previous disk shelf in the stack 7. Verify that the cabling on path A has successfully bypassed the disk shelves you are removing and all disk drives on the disk shelves you are removing are still connected through path B, by entering the following command from the nodeshell of either controller: storage show disk -p It might take up to a minute for the system to complete discovery. In this example of how the output should appear, the disk shelf being removed is disk shelf 3. One line item appears for each disk drive connected through path B (now the primary path); therefore, the disk drives are listed in the PRIMARY column and B appears in the first PORT column. There is no connectivity through path A for any of the disk drives in the disk shelf being removed; therefore, no information is shown in the SECONDARY or second PORT columns: fas6200> storage show disk -p PRIMARY PORT SECONDARY PORT SHELF BAY d.64 B 3 0

104 104 MetroCluster Service Guide 0d.65 B 3 1 0d.66 B 3 2 0d.67 B 3 3 0d.68 B 3 4 0d.69 B 3 5 0d.70 B 3 6 0d.71 B Attention: If the output shows anything other than all the disk drives connected only through path B, you must correct the cabling by repeating Step Complete the following substeps: a. Repeat step 6 on page 102 and step 7 on page 103 for path B. b. Repeat step 2 on page 101 to confirm that your system configuration is the same as before you began this procedure. c. Go to the next step. 9. If, when you removed ownership from the disk drives as part of the preparation for this procedure, you disabled disk autoassignment, then reenable disk autoassignment by entering the following command; otherwise, go to the next step: storage disk option modify -autoassign on 10. Power off the disk shelves you disconnected and unplug the power cords from the disk shelves. 11. Remove the disk shelves from the rack or cabinet. To make disk shelves lighter and easier to maneuver, remove the power supplies and IOMs. Avoid removing the disk drives or carriers if possible, because excessive handling can lead to internal damage. You should use a mechanical hoist or lift if you are moving a fully populated 4U disk shelf. For example, a fully populated DS4243 or DS4246 disk shelf can weigh approximately 110 lbs (49.9 kg). 12. If you are removing disk shelves at the remote site, repeat this procedure. Hot-removing a disk shelf that is attached to an FC-to-SAS bridge You can hot-remove disk shelves physically remove disk shelves that have had the aggregates removed from the disk drives from a MetroCluster configuration that is up and serving data. You can hot-remove one or more disk shelves from anywhere within a stack of disk shelves or remove a stack of disk shelves. Before you begin All aggregates must be removed from the disk drives in the disk shelves you are removing. Attention: Do not attempt this procedure unless you have already removed all aggregates from the disk shelves you are removing. You might want to use the vol move or disk copy commands to move the data to other disk shelves. ONTAP 9.0 Data Protection Guide Using SnapMirror and SnapVault Technology All disk drive ownership must be removed from the disk drives in the disk shelves you are removing.

105 Hot-removing storage from the MetroCluster configuration 105 Note: The procedure for removing ownership from disk drives requires you to disable disk autoassignment. You reenable disk autoassignment when prompted at the end of this shelf hotremove procedure. ONTAP 9.0 Disks and Aggregates Power Guide In a four-node MetroCluster configuration, the local HA pair cannot be in a takeover state. If you are removing one or more disk shelves from within a stack, you must have factored the distance to bypass the disk shelves you are removing; therefore, if the current cables are not long enough, you need to have longer cables available. NetApp Interoperability Matrix Tool About this task This task applies only to fabric-attached or bridge-attached MetroCluster configurations in which the storage shelves are connected using FC-to-SAS bridges. This procedure follows cabling best practices; therefore, references to IOMs and IOM SAS square and circle ports align with the best practices. If your storage system is cabled differently from what is prescribed as best practice, the IOM and IOM ports might be different. This procedure uses the following terms to describe the relative locations of the disk shelves and cabling: Path A refers to the A-side disk shelf I/O module (IOM A) located in the top or left of the disk shelf depending on your disk shelf model. Path B refers to the B-side disk shelf I/O module (IOM B) located in the bottom or right of the disk shelf depending on your disk shelf model. The first disk shelf in the stack is the disk shelf with a SAS square port directly connected to a bridge. The interim disk shelf in the stack is the disk shelf directly connected to other disk shelves in the stack. The last disk shelf in the stack is the disk shelf with a SAS circle port directly connected to a bridge. The next disk shelf is the disk shelf downstream of the disk shelf being removed, in depth order. The previous disk shelf is the disk shelf upstream of the disk shelf being removed, in depth order. After completing this procedure for the local site, you are instructed to repeat this procedure for the partner site if you are also removing disk shelves for the partner site. Steps 1. Verify the operation of the MetroCluster configuration in Data ONTAP: a. Check for any health alerts on both clusters: system health alert show b. Confirm that the MetroCluster is configured and in normal mode: metrocluster show c. Perform a MetroCluster check:

106 106 MetroCluster Service Guide metrocluster check run d. Display the results of the MetroCluster check: metrocluster check show e. Run Config Advisor. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/download/tools/config_advisor/. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered. 2. Verify that your system configuration is Multi-Path HA by entering the following command from the console of either controller: run local sysconfig It might take up to a minute for the system to complete discovery. The configuration is listed in the System Storage Configuration field. Attention: If your system configuration is shown as something other than Multi-Path HA, you cannot continue with this procedure. Your system must meet the prerequisites stated in the Before you begin section of this procedure. 3. Verify that no aggregates are on the disk drives in the disk shelves you are removing: a. Enter the following command at the console of either controller: storage aggregate plex offline -aggregate aggregate_name -plex plex_name b. Check the output of the SHELF column to verify that the disk shelves from which you have removed aggregates do not appear. Attention: If any disk drives in the disk shelves you are removing are anything other than spares or failed, you cannot continue with this procedure. Your system must meet the prerequisites stated in the Before you begin section of this procedure. The following aggr status -r output is for a MetroCluster system with disk shelf 3 being removed. Because aggregates have been removed from disk shelf 3, only disk shelves in the stack with aggregates (disk shelves 2 and 4) appear in the SHELF column: cluster_a> aggr status -r RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) dparity fc_a_1: l58 2c 2 2 FC:A 1 BSAS / / parity fc_a_1: l58 2c 2 11 FC:A 1 BSAS / / data fc_a_1: l58 2c 4 14 FC:A 1 BSAS / / Verify that ownership is removed from the disk drives on the disk shelves you are removing: a. Enter the following command at the console of either controller: run local disk show -v b. Check the output and take the appropriate action:

107 Hot-removing storage from the MetroCluster configuration 107 If the output shows... unassigned or broken for all disk drives Any disk drives in the disk shelves you are removing have ownership Then do this... Go to the next step. The disk drives in the disk shelves you are removing are in the correct state. See the ONTAP 9.0 Disks and Aggregates Power Guide for information about removing disk ownership. The following output for the run local disk show -v command on a MetroCluster system shows that disk drives on the disk shelf being removed (disk shelf 3) are in OWNER state, a correct state for removal. Two disk drives have the ownership removed; therefore, Not Owned appears in the column. Two disk drives are failed; therefore, FAILED appears in the POOL column: site_a> run local disk show -v DISK OWNER POOL SERIAL NUMBER DR HOME CHKSUM fcswa_1:3-125l24 Not Owned NONE N11DNLML node1b( ) Block fcswa_1:3-125l27 Not Owned NONE 9QMC2777 node1a( ) Zoned fcswa_1:3-125l43 node1a( ) FAILED 9QMCPPWP node1a( ) Block fcswa_1:3-125l9 node1b( ) FAILED 9QMCDZQZ node1b( ) Block 5. Physically identify and note the disk shelf IDs on the front of the disk shelves you are removing. You need to be certain of which disk shelves you are removing so that you can correctly recable path A and path B later in this procedure. 6. If you are removing an entire stack of disk shelves, complete the following substeps; otherwise, go to the next step: a. Remove all SAS cables on path A and path B. This includes bridge-to-shelf cables and shelf-to-shelf cables for all disk shelves in the stack you are removing. b. Go to Hot-removing a shelf in a MetroCluster system on page If you are removing one or more disk shelves from a stack (but keeping the stack), recable the applicable path A stack connections to bypass the disk shelves you are removing by completing the applicable set of substeps: If you are removing more than one disk shelf, complete the applicable set of substeps one disk shelf at a time. If you need a graphical system cabling reference, use the platform specific Installation and Setup Instructions document that ships with each platform. Installation and Setup Instructions FAS8040/FAS8060 Systems Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules

108 108 MetroCluster Service Guide Installation and Setup Instructions FAS8020 systems Installation and Setup Instructions 62xx Systems Installation and Setup Instructions 32xx Systems If you are removing... The first disk shelf in a stack Then... For path A, do the following: a. Remove the cable connecting the IOM A SAS circle port of the first disk shelf and the IOM A SAS square port of the second disk shelf in the stack and set it aside. b. Move the cable connecting the bridge to the IOM A SAS square port of the first disk shelf to the IOM A SAS square port of the second disk shelf in the stack. Later, when you are instructed to repeat this procedure for path B, do the following: a. Remove the cable connecting the IOM B SAS circle port of the first disk shelf and the IOM B SAS square port of the second disk shelf in the stack and set it aside. b. Go to the next step in this procedure. You have completed the recabling for path B. An interim disk shelf in a stack For path A, do the following: a. Remove the cable connecting the IOM A SAS circle port of the disk shelf being removed and the IOM A SAS square port of the next disk shelf in the stack and set it aside. b. Move the cable connecting the IOM A SAS square port of the disk shelf being removed to the IOM A SAS square port of the next disk shelf in the stack. Later, when you are instructed, repeat these substeps for path B. The last disk shelf in a stack For path A, do the following: a. Remove the cable connecting the IOM A SAS square port of the last disk shelf and the IOM A SAS circle port of the previous disk shelf in the stack and set it aside. b. Go to the next step in this procedure. You have completed the recabling for path A. Later, when you are instructed to repeat this procedure for path B, do the following: a. Remove the cable connecting the IOM B SAS square port of the last disk shelf and the IOM B SAS circle port of the previous disk shelf in the stack and set it aside. b. Move the cable connecting the bridge to the IOM B SAS circle port of the last disk shelf to the IOM B SAS circle port of the previous disk shelf in the stack. 8. Verify that the cabling on path A has successfully bypassed the disk shelves you are removing and all disk drives on the disk shelves you are removing are still connected through path B, by entering the following command from either controller: run local storage show disk -p It might take up to a minute for the system to complete discovery.

109 Hot-removing storage from the MetroCluster configuration 109 In this example of how the output should appear, the disk shelf being removed is disk shelf 3. One line item appears for each disk drive connected through path B (now the primary path); therefore, the disk drives are listed in the PRIMARY column and B appears in the first PORT column. There is no connectivity through path A for any of the disk drives in the disk shelf being removed; therefore, no information is shown in the SECONDARY or second PORT columns: site_a> storage show disk -p PRIMARY PORT SECONDARY PORT SHELF BAY fc_sw_a_1:3-0d.64 B 3 0 fc_sw_a_1:3-0d.65 B 3 1 fc_sw_a_1:3-0d.66 B 3 2 fc_sw_a_1:3-0d.67 B 3 3 fc_sw_a_1:3-0d.68 B 3 4 fc_sw_a_1:3-0d.69 B 3 5 fc_sw_a_1:3-0d.70 B 3 6 fc_sw_a_1:3-0d.71 B Attention: If the output shows anything other than all the disk drives connected only through path B, you must correct the cabling by repeating Step Complete the following substeps: a. Repeat step 7 on page 107 and step 8 on page 108 for path B. b. Repeat step 2 on page 106 to confirm that your system configuration is the same as before you began this procedure. c. Go to the next step. 10. If, when you removed ownership from the disk drives as part of the preparation for this procedure, you disabled disk autoassignment, then reenable disk autoassignment; otherwise, go to the next step: storage disk option modify -autoassign on Enter the applicable command from the clustershell of each controller. 11. Power off the disk shelves you disconnected and unplug the power cords from the disk shelves. 12. Remove the disk shelves from the rack or cabinet. To make disk shelves lighter and easier to maneuver, remove the power supplies and IOMs. Avoid removing the disk drives or carriers if possible, because excessive handling can lead to internal damage. It is recommended that you use a mechanical hoist or lift if you are moving a fully populated 4U disk shelf. For example, a fully populated DS4243 or DS4246 disk shelf can weigh approximately 110 lbs (49.9 kg). 13. If you are removing disk shelves at the remote site, repeat this procedure.

110 110 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration involves adding a controller to each cluster to form an HA pair at each MetroCluster site, and then refreshing the MetroCluster configuration. Before you begin The existing MetroCluster configuration must be healthy. About this task Before performing this procedure, the MetroCluster configuration consists of two single-node clusters: cluster_a cluster_b DR Group One node_a_1 DR pair node_b_1 At the conclusion of this procedure, the MetroCluster configuration consists of two HA pairs, one at each site: cluster_a cluster_b DR Group One node_a_1 DR pair node_b_1 HA pair HA pair node_a_2 DR pair node_b_2

111 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 111 Both sites must be expanded equally. A MetroCluster configuration cannot consist of an uneven number of nodes. This procedure can take over an hour per site, with additional time needed to initialize the disks. The time to initialize the disks depends on the size of the disks. Steps 1. Verifying the state of the MetroCluster configuration on page Zoning for the new controller ports when adding a controller module in a fabric-attached MetroCluster configuration on page Adding a new controller module to each cluster on page Refreshing the MetroCluster configuration with new controllers on page Checking the MetroCluster configuration on page Verifying the configuration with the Config Advisor on page Verifying switchover, healing, and switchback on page 148 Verifying the state of the MetroCluster configuration Before you add nodes to the MetroCluster configuration, you should identify the existing controllers and confirm the disaster recovery (DR) relationships between them, that the controllers are in normal mode, and the aggregates are mirrored. Steps 1. Display the details of the nodes in the MetroCluster configuration by issuing the following command from any node in the configuration: metrocluster node show -fields node,dr-partner,dr-partner-systemid The following output shows the existing nodes in the MetroCluster configuration. There is a single DR group and one node in each cluster. cluster_a::> metrocluster node show -fields node,dr-partner,drpartner-systemid dr-group-id cluster node dr-partner drpartner-systemid cluster_a controller_a_1 controller_b_ cluster_b controller_b_1 controller_a_ entries were displayed. 2. Display the state of the MetroCluster configuration: metrocluster show The following output shows that the existing nodes in the MetroCluster configuration are in normal mode:

112 112 MetroCluster Service Guide cluster_a::> metrocluster show Configuration: two-node-fabric Cluster Entry Name State Local: cluster_a Configuration State configured Mode normal auso-on-clusterdisaster Remote: controller_b_1_siteb configured normal auso-on-clusterdisaster AUSO Failure Domain Configuration State Mode AUSO Failure Domain 3. Issue the following command on each node in the MetroCluster configuration to check the state of the aggregates: storage aggregate show The following output shows that the aggregates on cluster_a are online and mirrored: cluster_a::> storage aggregate show Aggregate Size Available Used% State #Vols Nodes RAID Status aggr0_controller_a_1_0 1.38TB 68.63GB 95% online 1 controller_a_1 raid_dp,mirrored controller_a_1_aggr1 4.15TB 4.14TB 0% online 2 controller_a_1 raid_dp,mirrored controller_a_1_aggr2 4.15TB 4.14TB 0% online 1 controller_a_1 raid_dp,mirrored 3 entries were displayed. cluster_a::> Zoning for the new controller ports when adding a controller module in a fabric-attached MetroCluster configuration The FC switch zoning must accommodate the new controller connections. If you used the NetAppsupplied reference configuration files (RCFs) to configure your switches, the zoning is preconfigured and you do not need to make any changes. About this task If you manually configured your FC switches, you must ensure that the zoning is correct for the initiator connections from the new controller modules. See the sections on zoning in the Fabricattached MetroCluster Installation and Configuration Guide. ONTAP 9.0 Fabric-attached MetroCluster Installation and Configuration Guide

113 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 113 Adding a new controller module to each cluster You must add a new controller module to each site, creating an HA pair in each site. This is a multistep process involving both hardware and software changes that must be performed in the proper order. You must perform this process twice, once at each site. Before you begin The new controller module must be received from NetApp as part of the upgrade kit. You should check that PCIe cards in the new controller module are compatible and supported by the new controller module. NetApp Hardware Universe Your system must have an empty slot available for the new controller module when upgrading to a single-chassis HA pair (an HA pair in which both controller modules reside in the same chassis). Note: This configuration is not supported on all systems. You must have rack space and cables for the new controller module when upgrading to a dualchassis HA pair (an HA pair in which the controller modules reside in separate chassis. Note: This configuration is not supported on all systems. Each controller module must be connected to the management network through its e0a port or, if your system has one, the e0m port (wrench port). About this task These tasks must be repeated at each site. The preexisting controller modules are referred to as the existing controller modules; the examples in this procedure have the console prompt existing_ctlr>. The controller modules that are being added are referred to as the new controller modules; the examples in this procedure have the console prompt new_ctlr>. Steps 1. Preparing for the upgrade on page Preparing cluster ports on the existing controller module on page Preparing the netboot server on page Setting the HA mode on the existing controller module on page Shutting down the existing controller on page Installing and cabling the new controller on page Verifying and setting the HA state of the controller module and chassis on page Assigning disks to the new controller on page Netbooting and setting up ONTAP on the new controller module on page Mirroring the root aggregate on the new controller on page Configuring intercluster LIFs on page Creating a mirrored data aggregate on each node on page Ensuring redundant protection for the CRS metadata volume on page Installing licenses for the new controller module on page Creating unmirrored data aggregates on page Verifying storage failover on both controller modules and setting cluster HA on page 143

114 114 MetroCluster Service Guide 17. Installing the firmware after adding a controller module on page 144 Preparing for the upgrade Before upgrading to an HA pair, you must make sure that your system meets all requirements and that you have all the necessary information. Steps 1. Ensure that your system has enough available disks for the new controller module. You need to identify unassigned disks or spare disks with available partitions that you can assign to the new controller module. 2. How you proceed depends on the results of the previous step. If the output showed... Enough spare disks available for the new controller module Not enough spares for the new controller module on a system with no root-data partitioning Then... Go to the next step. Complete the following substeps: a. Determine where the aggregates for the existing node are located: storage aggregate show Note: If you do not have enough free disks for your system, you need to add more storage. Contact technical support for more information. b. If disk ownership automatic assignment is on, turn it off: storage disk option modify -node node_name - autoassign off c. Remove ownership on disks that do not have aggregates on them: storage disk removeowner disk_name d. Repeat the previous step for as many disks as you need for the new node. 3. Ensure that you have cables ready for the following connections: Cluster connections If you are creating a two-node switchless cluster, you need two cables to connect the controller modules. Otherwise, you need a minimum of four cables, two for each controller module connection to the cluster network switch. Other systems (like the 80xx series) have defaults of either four or six cluster connections. HA interconnect connections, if the system is in a dual-chassis HA pair Storage connections to the FC switches Cluster peering connections 4. Ensure that you have a serial port console available for the controller modules. 5. Ensure that your environment meets site and system requirements. NetApp Hardware Universe 6. Gather all IP addresses and other network parameters for the new controller module.

115 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 115 Preparing cluster ports on the existing controller module Before installing the new controller module, you must configure cluster ports on the existing controller module so that they can provide cluster communication with the new controller module. About this task If you are creating a two-node switchless cluster (with no cluster-network switches), you must enable the switchless-cluster networking mode. For detailed information about port, LIF, and network configuration in clustered Data ONTAP, see the Clustered Data ONTAP Network Management Guide on the the NetApp Support Site. Steps 1. Determine which ports should be used as the node's cluster ports. For a list of the default port roles for your platform, see the Hardware Universe at hwu.netapp.com. The Installation and Setup Instructions for your platform on the NetApp Support Site at mysupport.netapp.com also have information about the ports for cluster network connections. 2. For each cluster port, identify the port roles: network port show In the following example, ports e0a, e0b, e0c, and e0d must be changed to cluster ports: cluster_a::> network port show Node: controller_a_1 Speed(Mbps) Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status e0m Default mgmt_bd_1500 up 1500 auto/1000 healthy e0a Default Default up 1500 auto/10000 healthy e0b Default Default up 1500 auto/10000 healthy e0c Default Default up 1500 auto/10000 healthy e0d Default Default up 1500 auto/10000 healthy e0i Default Default down 1500 auto/10 - e0j Default Default down 1500 auto/10 - e0k Default Default down 1500 auto/10 - e0l Default Default down 1500 auto/10 - e2a Default Default up 1500 auto/10000 healthy e2b Default Default up 1500 auto/10000 healthy e4a Default Default up 1500 auto/10000 healthy e4b Default Default up 1500 auto/10000 healthy 13 entries were displayed. 3. If the port roles are not set to cluster: a. Change the port role to the correct role by entering the following command for each incorrect port role: network port broadcast-domain remove-ports -ipspace Default - broadcast-domain Default -ports node_name:port_name Note: Your domain name might be different from the name shown in the example. b. Add the port to the Cluster domain: network port broadcast-domain add-ports -ipspace Cluster -broadcastdomain Cluster -ports node_name:port_name 4. Verify that the port roles have changed: network port show The following example shows that ports e0a, e0b, e0c, and e0d are now cluster ports:

116 116 MetroCluster Service Guide Node: controller_a_1 Speed(Mbps) Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status e0m Default mgmt_bd_1500 up 1500 auto/1000 healthy e0a Cluster Cluster up 9000 auto/10000 healthy e0b Cluster Cluster up 9000 auto/10000 healthy e0c Cluster Cluster up 9000 auto/10000 healthy e0d Cluster Cluster up 9000 auto/10000 healthy e0i Default Default down 1500 auto/10 - e0j Default Default down 1500 auto/10 - e0k Default Default down 1500 auto/10 - e0l Default Default down 1500 auto/10 - e2a Default Default up 1500 auto/10000 healthy e2b Default Default up 1500 auto/10000 healthy e4a Default Default up 1500 auto/10000 healthy e4b Default Default up 1500 auto/10000 healthy 13 entries were displayed. 5. For each cluster port, use the network interface modify command to change the home port of any data LIFs on that port to a data port. The following command changes a home port of a data LIF to a data port: cluster1::> network interface modify -lif datalif1 -vserver vs1 -homeport e1b 6. For each LIF that you modified, use the network interface revert command to revert it to its new home port. The following command reverts the LIF datalif1 to its new home port of e1b: cluster1::> network interface revert -lif datalif1 -vserver vs1 7. If your system is part of a switched cluster, use the network interface create command to create cluster LIFs on the cluster ports. The following command creates a cluster LIF on one of the node's cluster ports. The -auto parameter configures the LIF to use a link-local IP address. cluster1::> network interface create -vserver Cluster -lif clus1 - role cluster -home-node node0 -home-port e1a -auto true 8. If you are creating a two-node switchless cluster, enable the switchless-cluster networking mode: a. Change to the advanced privilege level from either node: set -privilege advanced You can respond y when prompted to continue into advanced mode. The advanced mode prompt appears (*>). b. Enable switchless-cluster mode: network options switchless-cluster modify -enabled true c. Return to the admin privilege level: set -privilege admin

117 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 117 Important: Cluster interface creation for the existing node in a two-node switchless cluster system is completed after cluster setup is completed via netboot, on the new controller module. Preparing the netboot server When you are ready to prepare the netboot server, you must download the correct Data ONTAP netboot image from the NetApp Support Site to the netboot server and know its IP address. Before you begin You must have an HTTP server that you can access with the system before and after adding the new controller module. You must have access to the NetApp Support Site at mysupport.netapp.com. This enables you to download the necessary system files. About this task You must download the necessary system files for your platform and version of Data ONTAP from the the NetApp Support Site at mysupport.netapp.com. Both controller modules in the HA pair must run the same version of Data ONTAP. Steps 1. Download and extract the file used for performing the netboot of your system: a. Download the appropriate netboot.tgz file for your platform from the NetApp Support Site to a web-accessible directory. b. Change to the web-accessible directory. c. Extract the contents of the netboot.tgz file to the target directory by entering the following command: tar -zxvf netboot.tgz Your directory listing should contain the following directory: netboot/ 2. Download the image.tgz file from the NetApp Support Site to the web-accessible directory. Your directory listing should contain the following file and directory: image.tgz netboot/ 3. Determine the IP address of the existing controller module. This address is referred to later in this procedure as ip-address-of-existing controller. 4. Ping ip-address-of-existing controller to ensure that the address is reachable. Setting the HA mode on the existing controller module You must use the storage failover modify command to set the mode on the existing controller module. The mode value is enabled later after you reboot the controller module. Step 1. Set the mode to HA:: storage failover modify -mode ha -node existing_node_name

118 118 MetroCluster Service Guide Shutting down the existing controller You must perform a clean shutdown of the existing controller to ensure that all data has been written to disk. You must also disconnect the power supplies. Steps 1. Enter the following command from the existing controller's prompt: halt local If you are prompted to continue the halt procedure, enter y when prompted and wait until the system stops at the LOADER prompt. Attention: You must perform a clean system shutdown before replacing system components to avoid losing unwritten data in the NVRAM or NVMEM: In a FAS22xx system, the NVMEM LED is marked with a battery symbol and is located on the controller module to the left of the label showing the MAC address. This LED blinks if there is unwritten data in the NVRAM. If this LED is flashing red after you enter the halt command, reboot your system and try halting it again. In a 32xx system, the NVMEM LED is located on the controller module to the right of the network ports, marked with a battery symbol. This LED blinks if there is unwritten data in the NVRAM. If this LED is flashing red after you enter the halt command, reboot your system and try halting it again. In a 62xx, the NVRAM LED is located on the controller module to the right of the network ports, marked with a battery symbol. This LED shows blinks if there is unwritten data in the NVRAM. If this LED is flashing red after you enter the halt command, reboot your system and try halting it again. In a 80xx, the NVRAM LED is located on the controller module to the right of the network ports, marked with a battery symbol. This LED shows blinks if there is unwritten data in the NVRAM. If this LED is flashing green after you enter the halt command, reboot your system and try halting it again. 2. If you are not already grounded, properly ground yourself. 3. Turn off the power supplies and disconnect the power, using the correct method for your system and power-supply type: If your system uses.. AC power supplies DC power supplies Then... Unplug the power cords from the power source, and then remove the power cords. Remove the power at the DC source, and remove the DC wires, if necessary. Installing and cabling the new controller You must physically install the new controller module in the chassis and cable it. Steps 1. If you have an I/O expansion module (IOXM) in your system and are creating a single-chassis HA pair, you must uncable and remove the IOXM. You can then use the empty bay for the new controller module. However, the new configuration will not have the extra I/O provided by the IOXM.

119 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration Physically install the new controller module and, if necessary, install additional fans: If you are adding a controller module... To an empty bay to create a single-chassis HA pair and the system belongs to one of the following platforms: Then perform these steps... a. Install three additional fans in the chassis to cool the new controller module: i. Remove the bezel by using both hands to hold it by the openings on each side, and then pull the bezel away from the chassis until it releases from the four ball studs on the chassis frame. ii. Remove the blank plate that covers the bay that will contain the new fans. iii. Install the fans as described in the Replacing a fan module document for your system on the NetApp Support Site at mysupport.netapp.com. b. Remove the blank plate in the rear of the chassis that covers the empty bay that will contain the new controller module. c. Gently push the controller module halfway into the chassis. To an empty bay to create a single-chassis HA pair and the system belongs to one of the following platforms: 32xx a. Remove the blank plate in the rear of the chassis that covers the empty bay that will contain the new controller module. b. Gently push the controller module halfway into the chassis. To prevent the controller module from automatically booting, do not fully seat it in the chassis until later in this procedure In a separate chassis from its HA partner to create a dualchassis HA pair when the existing configuration is in a controller-iox module configuration. Install the new system in the rack or system cabinet. FAS32xx FAS62xx FAS Cable the HA interconnect if you have a dual-chassis HA pair. 4. Except for CNA ports and cluster network ports, cable the ports on the new controller module, including the HA interconnect if you have a dual-chassis HA pair. Also, connect the controller module to any storage assigned to the HA pair. If the new controller has CNA ports, do not cable them until you verify their configuration as described later in this procedure. See the system s Installation and Setup Instructions, the Clustered Data ONTAP High- Availability Configuration Guide for your version of Data ONTAP, and, if applicable, the Universal SAS and ACP Cabling Guide. 5. Cable the cluster network connections, as necessary:

120 120 MetroCluster Service Guide a. Identify the ports on the controller module for the cluster connections. These ports were identified in Step 1 of Preparing cluster ports on the existing controller on page 115. b. If you are configuring a switched cluster, identify the ports that you will use on the cluster network switches. See the Clustered Data ONTAP Switch Setup Guide for Cisco Switches, NetApp 10G Cluster- Mode Switch Installation Guide or NetApp 1G Cluster-Mode Switch Installation Guide, depending on what switches you are using. c. Connect cables to the cluster ports: If the cluster is... A two-node switchless cluster A switched cluster Then... Directly connect the cluster ports on the existing controller module to the corresponding cluster ports on the new controller module. Connect the cluster ports on each controller to the ports on the cluster network switches identified in substep b. 6. Power up the existing controller module. 7. Depending on your configuration, power up the new controller module and interrupt the boot process: If the new controller module is... In the same chassis as the existing controller module Then... a. Push the controller module firmly into the bay. When fully seated, the controller module receives power and automatically boots. b. Interrupt the boot process by pressing Ctrl-C. c. Tighten the thumb screw on the cam handle. d. Install the cable management device. e. Bind the cables to the cable management device with the hook and loop strap. In a separate chassis from the existing controller module a. Turn on the power supplies on the new controller module. b. Interrupt the boot process by pressing Ctrl-C. The system displays the LOADER prompt (LOADER>, LOADER-A>, or LOADER-B>). Note: If there is no LOADER prompt, record the error message and contact technical support. If the system displays the boot menu, reboot and attempt to interrupt the boot process again. Cabling the FC-VI ports to the FC switches The FC-VI ports on each controller module must be cabled to the site FC switches in the MetroCluster configuration. About this task This task must be performed on each site in the MetroCluster configuration.

121 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 121 Step 1. Cable the FC-VI ports. Note: The Brocade and Cisco switches use different port numbering: On Brocade switches, the first port is numbered 0. On Cisco switches, the first port is numbered 1. The following tables show both Brocade and Cisco port numbering: Cabling to FC_switch_x_1 Connect this site component and port... To this port on FC_switch_x_1... Brocade Cisco controller_x_1 FC-VI port a 0 1 controller_x_2 FC-VI port a 3 4 controller_x_3 FC-VI port a 6 7 controller_x_4 FC-VI port a 9 10 Connect this site component and port... Cabling to FC_switch_x_2 To this port on FC_switch_x_2... Brocade controller_x_1 FC-VI port b 0 1 controller_x_2 FC-VI port b 3 4 controller_x_3 FC-VI port b 6 7 Cisco controller_x_4 FC-VI port b 9 10 Cabling the FC-VI and HBA connections to the FC switches The FC-VI ports and HBAs (host bus adapters) must be cabled to the site FC switches on each of the new controller modules. About this task This task must be performed on both MetroCluster sites. Step 1. Cable the FC-VI ports and HBA ports, using the table for your configuration and switch model. Note: The Brocade and Cisco switches use different port numbering: On Brocade switches, the first port is numbered 0. On Cisco switches, the first port is numbered 1. The following tables show both Brocade and Cisco port numbering.

122 122 MetroCluster Service Guide MetroCluster four-node configuration Component Port Brocade 6505 or 6510 FC_switch_x_1 FC_switch_x_2 controller_x_1 FC-VI port a 0 - FC-VI port b - 0 HBA port a 1 - HBA port b - 1 HBA port c 2 - HBA port d - 2 controller_x_2 FC-VI port a 3 - FC-VI port b - 3 HBA port a 4 - HBA port b - 4 HBA port c 5 - HBA port d - 5 MetroCluster four-node configuration Component Port Cisco 9148, 9148S, or 9250i FC_switch_x_1 FC_switch_x_2 controller_x_1 FC-VI port a 1 - FC-VI port b - 1 HBA port a 2 - HBA port b - 2 HBA port c 3 - HBA port d - 3 controller_x_2 FC-VI port a 4 - FC-VI port b - 4 HBA port a 5 - HBA port b - 5 HBA port c 6 - HBA port d - 6 Port assignments for FC switches in a four-node configuration You need to verify that you are using the specified port assignments when you cable the FC switches. The port assignments shown here match the preconfigured reference configuration files (RCFs). Ports that are not used for attaching initiator ports, FC-VI ports or ISLs can be reconfigured to act as storage ports. However, if the supported RCFs are being used, the zoning must be changed accordingly.

123 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 123 Port numbering differences between Brocade and Cisco switches The Brocade and Cisco switches use different port numbering: On Brocade switches, the first port is numbered 0. On Cisco switches, the first port is numbered 1. This difference is reflected in the following tables. Note: The cabling is the same for each FC switch in the switch fabric. Brocade port usage for controllers in a four-node MetroCluster configuration The cabling is the same for each FC switch in the switch fabric. MetroCluster four-node configuration Component Port Brocade 6505 or 6510 FC_switch_x_1 FC_switch_x_2 controller_x_1 FC-VI port a 0 - FC-VI port b - 0 HBA port a 1 - HBA port b - 1 HBA port c 2 - HBA port d - 2 controller_x_2 FC-VI port a 3 - FC-VI port b - 3 HBA port a 4 - HBA port b - 4 HBA port c 5 - HBA port d - 5 Brocade port usage for bridges in a four-node MetroCluster configuration The cabling is the same for each FC switch in the switch fabric. The following table shows bridge port usage up to port 17 when using FibreBridge 7500 bridges. Additional bridges can be cabled to ports 18 through 23. FibreBridge 7500 bridge MetroCluster four-node configuration Port Brocade 6510 Brocade 6505 FC_switch_x_ 1 FC_switch_x_2 FC_switch_ x_1 FC_switch_x _2 bridge_x_1a FC FC bridge_x_1b FC FC

124 124 MetroCluster Service Guide FibreBridge 7500 bridge MetroCluster four-node configuration Port Brocade 6510 Brocade 6505 FC_switch_x_ 1 FC_switch_x_2 FC_switch_ x_1 FC_switch_x _2 bridge_x_2a FC FC bridge_x_2b FC FC bridge_x_3a FC FC bridge_x_3b FC FC bridge_x_4a FC FC bridge_x_4b FC FC additional bridges can be cabled through port 19, then ports 24 through 47 The following table shows bridge port usage when using FibreBridge 6500 bridges: FibreBridge 6500 bridge additional bridges can be cabled through port 23 Port Brocade 6510 Brocade 6505 FC_switch_x _1 FC_switch_x_2 FC_switch_x _1 FC_switch_x _2 bridge_x_1a FC bridge_x_1b FC bridge_x_2a FC bridge_x_2b FC bridge_x_3a FC bridge_x_3b FC bridge_x_4a FC bridge_x_4b FC bridge_x_5a FC bridge_x_5b FC bridge_x_6a FC bridge_x_6b FC bridge_x_7a FC bridge_x_7b FC

125 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 125 FibreBridge 6500 bridge Port Brocade 6510 Brocade 6505 FC_switch_x _1 FC_switch_x_2 FC_switch_x _1 FC_switch_x _2 bridge_x_8a FC bridge_x_8b FC additional bridges can be cabled through port 19, then ports 24 through 47 additional bridges can be cabled through port 23 Brocade port usage for ISLs in a four-node MetroCluster configuration The following table shows ISL port usage: MetroCluster four-node configuration ISL port Brocade 6510 Brocade 6505 FC_switch_x _1 FC_switch_ x_2 FC_switch_x_1 FC_switch_x_2 ISL port ISL port ISL port ISL port Cisco port usage for controllers in a four-node MetroCluster configuraiton The cabling is the same for each FC switch in the switch fabric. The following table shows controller port usage on Cisco switches: MetroCluster four-node configuration Component Port Cisco 9148, 9148S, or 9250i FC_switch_x_1 FC_switch_x_2 controller_x_1 FC-VI port a 1 - FC-VI port b - 1 HBA port a 2 - HBA port b - 2 HBA port c 3 - HBA port d - 3 controller_x_2 FC-VI port a 4 - FC-VI port b - 4 HBA port a 5 - HBA port b - 5 HBA port c 6 - HBA port d - 6

126 126 MetroCluster Service Guide Cisco port usage for FC-to-SAS bridges in a four-node MetroCluster configuration The following table shows bridge port usage up to port 14 when using FibreBridge 7500 bridges. Additional bridges can be attached to ports 15 through 32 following the same pattern. FibreBridge 7500 bridge Port MetroCluster four-node configuration Cisco 9148, 9148S, or 9250i FC_switch_x_1 FC_switch_x_2 bridge_x_1a FC1 7 - FC2-7 bridge_x_1b FC1 8 - FC2-8 bridge_x_2a FC1 9 - FC2-9 bridge_x_2b FC FC2-10 bridge_x_3a FC FC2-11 bridge_x_3b FC FC2-12 bridge_x_4a FC FC2-13 bridge_x_4b FC FC2-14 The following table shows bridge port usage when using FibreBridge 6500 bridges up to port 14. Additional bridges can be attached to ports 15 through 32 following the same pattern. FibreBridge 6500 bridge Port Cisco 9148, 9148S, or 9250i FC_switch_x_1 FC_switch_x_2 bridge_x_1a FC1 7 - bridge_x_1b FC1-7 bridge_x_2a FC1 8 - bridge_x_2b FC1-8 bridge_x_3a FC1 9 - bridge_x_3b FC1-9 bridge_x_4a FC bridge_x_4b FC1-10 bridge_x_5a FC bridge_x_5b FC1-11 bridge_x_6a FC1 12 -

127 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 127 FibreBridge 6500 bridge Port Cisco 9148, 9148S, or 9250i FC_switch_x_1 FC_switch_x_2 bridge_x_6b FC1-12 bridge_x_7a FC bridge_x_7b FC1-13 bridge_x_8a FC bridge_x_8b FC1-14 Additional bridges can be attached to ports 15 through 32 following the same pattern. Port usage on the Cisco 9250i switch The Cisco 9250i switch uses the FCIP ports for the ISL. See NetApp KB Article : How to configure the Cisco 9250i FC storage backend switch in MetroCluster for clustered Data ONTAP for limitations and procedures for using the FCIP ports. Ports 40 through 48 are 10 GbE ports and are not used in the MetroCluster configuration. Cabling the cluster peering connections You must cable the new controller modules to the cluster peering network so that they have connectivity with the cluster on the partner site. About this task This task must be performed on each of the new controllers. At least two ports on each controller should be used for cluster peering. The recommended minimum bandwidth for the ports and network connectivity is 1 GbE. Step 1. Identify and cable at least two ports for cluster peering and verify they have network connectivity with the partner cluster. Cluster peering can be done on dedicated ports or on data ports. Using dedicated ports provides higher throughput for the cluster peering traffic. Configuring and cabling CNA ports (80xx systems only) If you are adding a controller module to a 80xx system, you must check the configuration of the CNA ports on the new controller module and, if necessary, change the defaults to match the CNA port configuration of the existing controller module. Before you begin You must have the SFP+ modules for the CNA ports, or, if the ports are set to a 10-GbE personality, you can use Twinax cables. Steps 1. Boot to Maintenance mode on the new node, if it is not in Maintenance mode, by entering halt to go to the LOADER prompt: If you are running Data ONTAP 8.2 and earlier, enter boot_ontap, press Ctrl-C when prompted to got to the boot menu, and then select Maintenance mode from the menu.

128 128 MetroCluster Service Guide If you are running Data ONTAP and later, enter boot_ontap maint at the loader prompt. If you are prompted to continue with the boot, enter y to continue. 2. On the existing controller module console, check how the ports are currently configured: system node hardware unified-connect show The system displays output similar to the following example: node_name::> system node hardware unified-connect show Current Current Pending Pending Node Adapter Mode Type Mode Type Status f-a 0e fc initiator - - online f-a 0f fc initiator - - online f-a 0g cna target - - online On the console of the new node, display the port settings: ucadmin show The system displays output similar to the following example: *> ucadmin show Current Current Pending Pending Node Adapter Mode Type Mode Type Status f-a 0e fc initiator - - online f-a 0f fc initiator - - online f-a 0g cna target - - online If the current SFP+ module does not match the desired use, replace it with the correct SFP+ module. 5. If the current configuration does not match the existing node's configuration, enter one of the following commands to change the configuration as needed: If the desired use is for... FC initiator FC target Ethernet Then enter the following command... ucadmin modify -t initiator adapter_name ucadmin modify -t target adapter_name ucadmin modify -m cna adapter_name Note: If you changed the port personality, it will take effect when the new node is booted. Make sure that you verify the settings after the boot. 6. Cable the port. Verifying and setting the HA state of the controller module and chassis When you expand a MetroCluster configuration you must update the HA state of the storage system hardware components to indicate that they are in a MetroCluster configuration. Steps 1. Halt the existing controller module and boot it to Maintenance mode: haltboot_ontap maint 2. Boot the existing controller module to Maintenance mode: boot_ontap maint 3. In Maintenance mode from the existing controller module, display the HA state of the new controller module and chassis:

129 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 129 ha-config show The HA state should be mcc for all components. 4. If the displayed system state of the controller is not mcc, set the HA state for the controller module: ha-config modify controller mcc 5. If the displayed system state of the chassis is not mcc, set the HA state for the chassis: ha-config modify chassis mcc 6. Retrieve the system ID for the current node: sysconfig Note the system ID. You need it when you assign disks to the new node. 7. Repeat these steps on the new controller module. 8. Exit Maintenance mode: exit The existing nodes should remain at the LOADER prompt until you complete the steps in Netbooting and setting up ONTAP on the new controller module on page 130. At that point ONTAP is running on the new controller module and its root aggregate is configured. Assigning disks to the new controller Before you complete the configuration of the new controller module through netboot, you must assign disks to it. Before you begin You must have made sure that there are enough spares, unassigned disks, or assigned disks that are not part of an existing aggregate. Preparing for the upgrade on page 114 About this task These steps are performed on the existing controller module. Steps 1. Assign the root disk to the new controller module: storage disk assign -disk disk_name -root true -sysid new_controller_sysid -force true 2. Assign the remaining required disks to the new controller module by entering the following command for each disk: storage disk assign -disk disk_name -sysid new_controller_sysid -force true 3. Verify that the disk assignments are correct by examining the output from the storage disk show -partitionownership command and correct as needed. 4. After you finish assigning disks, halt the node to the LOADER prompt: halt

130 130 MetroCluster Service Guide Netbooting and setting up ONTAP on the new controller module You must perform a specific sequence of steps to netboot and install the ONTAP operating system on the new controller module. About this task This procedure includes initializing disks. The amount of time you need to initialize the disks depends on the size of the disks. The system automatically assigns two disks to the new controller module. Steps 1. Enter the following commands at the LOADER prompt to configure the IP address of the new controller module: If DHCP is... Available Not available Then enter the following command... ifconfig e0m -auto Enter the following command: ifconfig e0m -addr=filer_addr -mask=netmask - gw=gateway -dns=dns_addr -domain=dns_domain filer_addr is the IP address of the storage system. netmask is the network mask of the storage system. gateway is the gateway for the storage system. dns_addr is the IP address of a name server on your network. dns_domain is the Domain Name System (DNS) domain name. If you use this optional parameter, you do not need a fully qualified domain name in the netboot server URL; you need only the server s host name. Note: Other parameters might be necessary for your interface. For details, use the help ifconfig command at the LOADER prompt. 2. At the LOADER prompt, enter the following command: netboot 3. Select the Install new software first option from the displayed menu. This menu option downloads and installs the new Data ONTAP image to the boot device. Select y when prompted with the message that this procedure is not supported for nondisruptive upgrade on an HA pair active/active configuration. Select y when warned that this process will replace existing Data ONTAP software with new software. Enter the path as follows when prompted for the URL of the image.tgz file: 4. Enter y when prompted regarding non-disruptive upgrade or replacement of the software. 5. Enter the path to the image.tgz file when you see the following prompt: What is the URL for the package?

131 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration Enter n to skip the backup recovery when you see the following prompt: **************************************************************** * Restore Backup Configuration * * This procedure only applies to storage controllers that * * are configured as an HA pair. * * * * Choose Yes to restore the "varfs" backup configuration * * from the SSH server. Refer to the Boot Device Replacement * * guide for more details. * * Choose No to skip the backup recovery and return to the * * boot menu. * **************************************************************** Do you want to restore the backup configuration now? {y n} n 7. Reboot by entering y when you see the following prompt: The node must be rebooted to start using the newly installed software. Do you want to reboot now? {y n}y 8. If necessary, select the option to Clean configuration and initialize all disks after the node has booted. Because you are configuring a new controller and the new controller's disks are empty, you can respond y when the system warns you that this will erase all disks. Note: The amount of time needed to initialize disks will depend on the size of your disks and configuration. 9. The Node Setup wizard starts after the disks are initialized. Complete the following substeps: a. Enter the node management LIF information on the console as shown in the following example: Welcome to node setup. You can enter the following commands at any time: "help" or "?" - if you want to have a question clarified, "back" - if you want to change previously answered questions, and "exit" or "quit" - if you want to quit the cluster setup wizard. Any changes you made before quitting will be saved. To accept a default or omit a question, do not enter a value. This system will send event messages and weekly reports to NetApp Technical Support. To disable this feature, enter "autosupport modify -support disable" within 24 hours. Enabling AutoSupport can significantly speed problem determination and resolution should a problem occur on your system. For further information on AutoSupport, please see: Type yes to confirm and continue{yes}: yes Enter the node management interface port [e0m]: Enter the node management interface IP address: Enter the node management interface netmask:

132 132 MetroCluster Service Guide Enter the node management interface default gateway: A node management interface on port e0c with IP address has been created. This node has its management address assigned and is ready for cluster setup. To complete cluster setup after all nodes are ready, download and run the System Setup utility from the NetApp Support Site and use it to discover the configured nodes. For System Setup, this node's management address is: b. Enter the node management LIF information on the console as shown in the following example: c. Manually enter the admin login ID when prompted to do so. d. Manually start the Cluster Setup wizard by entering the following command at the prompt: cluster setup 10. With the Cluster Setup wizard running, join the node to the cluster: join Welcome to the cluster setup wizard. You can enter the following commands at any time: "help" or "?" - if you want to have a question clarified, "back" - if you want to change previously answered questions, and "exit" or "quit" - if you want to quit the cluster setup wizard. Any changes you made before quitting will be saved. You can return to cluster setup at any time by typing "cluster setup". To accept a default or omit a question, do not enter a value. Do you want to create a new cluster or join an existing cluster? {create, join}: join 11. Respond yes when prompted to set storage failover to HA mode. Non-HA mode, Reboot node to activate HA Warning: Ensure that the HA partner has started disk initialization before rebooting this node to enable HA. Do you want to reboot now to set storage failover (SFO) to HA mode? {yes, no} [yes]: yes Rebooting now After the node reboots, the Cluster Setup wizard displays Welcome to node setup and prompts you to complete the node setup. 12. Log into the node, and enter the cluster setup and then enter join when prompted to join the cluster.

133 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 133 Do you want to create a new cluster or join an existing cluster? {create, join}:join 13. Respond to the remaining prompts as appropriate for your site. See the Clustered Data ONTAP Software Setup Guide for your version of ONTAP for additional details. 14. If the system is a two-node switchless cluster configuration, create the cluster interfaces on the existing node using the network interface create command to create cluster LIFs on the cluster ports. The following is an example command for creating a cluster LIF on one of the node's cluster ports. The -auto parameter configures the LIF to use a link-local IP address: cluster1::> network interface create -vserver Cluster -lif clus1 - role cluster -home-node node0 -home-port e1a -auto true 15. After setup is complete, verify that the node is healthy and eligible to participate in the cluster: cluster show The following example shows a cluster after the second node (cluster1-02) has been joined to it: cluster1::> cluster show Node Health Eligibility cluster_a true true cluster_b true true You can access the Cluster Setup wizard to change any of the values you entered for the admin Storage Virtual Machine (SVM) or node SVM by using the cluster setup command. 16. Confirm that you have four ports configured as cluster interconnects: network port show The following example shows output for two controllers in cluster_a. cluster_a::> network port show Speed (Mbps) Node Port IPspace Broadcast Domain Link MTU Admin/Oper node_a_1 e0a Cluster Cluster up 1500 auto/1000 e0b Cluster Cluster up 1500 auto/1000 e0c Default Default up 1500 auto/1000 e0d Default Default up 1500 auto/1000 e0e Default Default up 1500 auto/1000 e0f Default Default up 1500 auto/1000 e0g Default Default up 1500 auto/1000 node_a_2 e0a Cluster Cluster up 1500 auto/1000 e0b Cluster Cluster up 1500 auto/1000 e0c Default Default up 1500 auto/1000 e0d Default Default up 1500 auto/1000

134 134 MetroCluster Service Guide e0e Default Default up 1500 auto/1000 e0f Default Default up 1500 auto/1000 e0g Default Default up 1500 auto/ entries were displayed. Mirroring the root aggregate on the new controller You must mirror the root aggregate to provide data protection when you are adding a controller to a MetroCluster configuration. About this task This task must be performed on the new controller module. Step 1. Mirror the root aggregate: storage aggregate mirror aggr_id The following command mirrors the root aggregate for controller_a_1: controller_a_1::> storage aggregate mirror aggr0_controller_a_1 This creates an aggregate with a local plex located at the local MetroCluster site and a remote plex located at the remote MetroCluster site. Configuring intercluster LIFs You must create intercluster LIFs on ports used for communication between the MetroCluster partner clusters. You can use dedicated ports or ports that also have data traffic. Choices Configuring intercluster LIFs to use dedicated intercluster ports on page 134 Configuring intercluster LIFs to share data ports on page 138 Configuring intercluster LIFs to use dedicated intercluster ports Configuring intercluster LIFs to use dedicated data ports allows greater bandwidth than using shared data ports on your intercluster networks for cluster peer relationships. About this task Creating intercluster LIFs that use dedicated ports involves creating a failover group for the dedicated ports and assigning LIFs to those ports. In this procedure, a two-node cluster exists in which each node has two data ports that you have added, e0e and e0f. These ports are ones you will dedicate for intercluster replication and currently are in the default IPspace. These ports will be grouped together as targets for the intercluster LIFs you are configuring. You must configure intercluster LIFs on the peer cluster before you can create cluster peer relationships. In your own environment, you might replace the ports, networks, IP addresses, subnet masks, and subnets with those specific to your environment. Steps 1. List the ports in the cluster by using network port show command.

135 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 135 cluster01::> network port show Speed (Mbps) Node Port IPspace Broadcast Domain Link MTU Admin/Oper cluster01-01 e0a Cluster Cluster up 1500 auto/1000 e0b Cluster Cluster up 1500 auto/1000 e0c Default Default up 1500 auto/1000 e0d Default Default up 1500 auto/1000 e0e Default Default up 1500 auto/1000 e0f Default Default up 1500 auto/1000 cluster01-02 e0a Cluster Cluster up 1500 auto/1000 e0b Cluster Cluster up 1500 auto/1000 e0c Default Default up 1500 auto/1000 e0d Default Default up 1500 auto/1000 e0e Default Default up 1500 auto/1000 e0f Default Default up 1500 auto/ Determine whether any of the LIFs are using ports that are dedicated for replication by using the network interface show command. Ports e0e and e0f do not appear in the following output; therefore, they do not have any LIFs located on them: cluster01::> network interface show -fields home-port,curr-port vserver lif home-port curr-port Cluster cluster01-01_clus1 e0a e0a Cluster cluster01-01_clus2 e0b e0b Cluster cluster01-02_clus1 e0a e0a Cluster cluster01-02_clus2 e0b e0b cluster01 cluster_mgmt e0c e0c cluster01 cluster01-01_mgmt1 e0c e0c cluster01 cluster01-02_mgmt1 e0c e0c 3. If a LIF is using a port that you want dedicated to intercluster connectivity, migrate the LIF to a different port. a. Migrate the LIF to another port by using the network interface migrate command. The following example assumes that the data LIF named cluster01_data01 uses port e0e and you want only an intercluster LIF to use that port: cluster01::> network interface migrate -vserver cluster01 -lif cluster01_data01 -dest-node cluster dest-port e0d b. You might need to modify the migrated LIF home port to reflect the new port where the LIF should reside by using the network interface modify command. cluster01::> network interface modify -vserver cluster01 -lif cluster01_data01 -home-node cluster home-port e0d 4. Group the ports that you will use for the intercluster LIFs by using the network interface failover-groups create command.

136 136 MetroCluster Service Guide cluster01::> network interface failover-groups create -vserver cluster01 -failover-group intercluster01 -targets cluster01-01:e0e,cluster01-01:e0f, cluster01-02:e0e,cluster01-02:e0f 5. Display the failover group that you created by using the network interface failovergroups show command. cluster01::> network interface failover-groups show Failover Vserver Group Targets Cluster Cluster cluster01-01:e0a, cluster01-01:e0b, cluster01-02:e0a, cluster01-02:e0b cluster01 Default cluster01-01:e0c, cluster01-01:e0d, cluster01-02:e0c, cluster01-02:e0d, cluster01-01:e0e, cluster01-01:e0f cluster01-02:e0e, cluster01-02:e0f intercluster01 cluster01-01:e0e, cluster01-01:e0f cluster01-02:e0e, cluster01-02:e0f 6. Create an intercluster LIF on the admin SVM cluster01 by using the network interface create command. This example uses the LIF naming convention adminsvmname_icl# for the intercluster LIF: cluster01::> network interface create -vserver cluster01 -lif cluster01_icl01 -role intercluster -home-node cluster home-port e0e -address netmask failover-group intercluster01 cluster01::> network interface create -vserver cluster01 -lif cluster01_icl02 -role intercluster -home-node cluster home-port e0e -address netmask failover-group intercluster01 7. Verify that the intercluster LIFs were created properly by using the network interface show command. cluster01::> network interface show Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home Cluster cluster01-01_clus_1 up/up xxx/24 cluster01-01 e0a true cluster01-01_clus_2 up/up xxx/24 cluster01-01 e0b true cluster01-02_clus_1 up/up xxx/24 cluster01-01 e0a true cluster01-02_clus_2 up/up xxx/24 cluster01-01 e0b true cluster01 cluster_mgmt up/up xxx/24 cluster01-01 e0c true cluster01_icl01 up/up /24 cluster01-01 e0e true cluster01_icl02 up/up /24 cluster01-02 e0e true cluster01-01_mgmt1 up/up xxx/24 cluster01-01 e0c true cluster01-02_mgmt1 up/up xxx/24 cluster01-02 e0c true

137 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration Verify that the intercluster LIFs are configured for redundancy by using the network interface show command with the -role intercluster and -failover parameters. The LIFs in this example are assigned the e0e home port on each node. If the e0e port fails, the LIF can fail over to the e0f port. cluster01::> network interface show -role intercluster failover Logical Home Failover Failover Vserver Interface Node:Port Policy Group cluster01-01 cluster01-01_icl01 cluster01-01:e0e local-only intercluster01 Failover Targets: cluster01-01:e0e, cluster01-01:e0f cluster01-01_icl02 cluster01-02:e0e local-only intercluster01 Failover Targets: cluster01-02:e0e, cluster01-02:e0f 9. Display the routes in the cluster by using the network route show command to determine whether intercluster routes are available or you must create them. Creating a route is required only if the intercluster addresses in both clusters are not on the same subnet and a specific route is needed for communication between the clusters. In this example, no intercluster routes are available: cluster01::> network route show Vserver Destination Gateway Metric Cluster / cluster / If communication between intercluster LIFs in different clusters requires routing, create an intercluster route by using the network route create command. The gateway of the new route should be on the same subnet as the intercluster LIF. In this example, is the gateway address for the /24 network. If the destination is specified as /0, then it becomes a default route for the intercluster network. cluster01::> network route create -vserver cluster01 -destination /0 -gateway metric Verify that you created the routes correctly by using the network route show command. cluster01::> network route show Vserver Destination Gateway Metric Cluster / cluster / / Repeat these steps to configure intercluster networking in the peer cluster. 13. Verify that the ports have access to the proper subnets, VLANs, and so on.

138 138 MetroCluster Service Guide Dedicating ports for replication in one cluster does not require dedicating ports in all clusters; one cluster might use dedicated ports, while the other cluster shares data ports for intercluster replication. Configuring intercluster LIFs to share data ports Configuring intercluster LIFs to share data ports enables you to use existing data ports to create intercluster networks for cluster peer relationships. Sharing data ports reduces the number of ports you might need for intercluster networking. About this task Creating intercluster LIFs that share data ports involves assigning LIFs to existing data ports. In this procedure, a two-node cluster exists in which each node has two data ports, e0c and e0d, and these data ports are in the default IPspace. These are the two data ports that are shared for intercluster replication. You must configure intercluster LIFs on the peer cluster before you can create cluster peer relationships. In your own environment, you replace the ports, networks, IP addresses, subnet masks, and subnets with those specific to your environment. Steps 1. List the ports in the cluster by using the network port show command: cluster01::> network port show Speed (Mbps) Node Port IPspace Broadcast Domain Link MTU Admin/Oper cluster01-01 e0a Cluster Cluster up 1500 auto/1000 e0b Cluster Cluster up 1500 auto/1000 e0c Default Default up 1500 auto/1000 e0d Default Default up 1500 auto/1000 cluster01-02 e0a Cluster Cluster up 1500 auto/1000 e0b Cluster Cluster up 1500 auto/1000 e0c Default Default up 1500 auto/1000 e0d Default Default up 1500 auto/ Create an intercluster LIF on the admin SVM cluster01 by using the network interface create command. This example uses the LIF naming convention adminsvmname_icl# for the intercluster LIF: cluster01::> network interface create -vserver cluster01 -lif cluster01_icl01 -role intercluster -home-node cluster home-port e0c -address netmask cluster01::> network interface create -vserver cluster01 -lif cluster01_icl02 -role intercluster -home-node cluster home-port e0c -address netmask Verify that the intercluster LIFs were created properly by using the network interface show command with the -role intercluster parameter:

139 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 139 cluster01::> network interface show role intercluster Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home cluster01 cluster01_icl01 up/up /24 cluster01-01 e0c true cluster01_icl02 up/up /24 cluster01-02 e0c true 4. Verify that the intercluster LIFs are configured to be redundant by using the network interface show command with the -role intercluster and -failover parameters. The LIFs in this example are assigned the e0c port on each node. If the e0c port fails, the LIF can fail over to the e0d port. cluster01::> network interface show -role intercluster failover Logical Home Failover Failover Vserver Interface Node:Port Policy Group cluster01 cluster01_icl01 cluster01-01:e0c local-only /24 Failover Targets: cluster01-01:e0c, cluster01-01:e0d cluster01_icl02 cluster01-02:e0c local-only /24 Failover Targets: cluster01-02:e0c, cluster01-02:e0d 5. Display the routes in the cluster by using the network route show command to determine whether intercluster routes are available or you must create them. Creating a route is required only if the intercluster addresses in both clusters are not on the same subnet and a specific route is needed for communication between the clusters. In this example, no intercluster routes are available: cluster01::> network route show Vserver Destination Gateway Metric Cluster / cluster / If communication between intercluster LIFs in different clusters requires routing, create an intercluster route by using the network route create command. The gateway of the new route should be on the same subnet as the intercluster LIF. In this example, is the gateway address for the /24 network. If the destination is specified as /0, then it becomes a default route for the intercluster network. cluster01::> network route create -vserver cluster01 -destination /0 -gateway metric Verify that you created the routes correctly by using the network route show command.

140 140 MetroCluster Service Guide cluster01::> network route show Vserver Destination Gateway Metric Cluster / cluster / / Repeat these steps on the cluster to which you want to connect. Creating a mirrored data aggregate on each node You must create a mirrored data aggregate on each node in the DR group. Before you begin You should know what drives or array LUNs will be used in the new aggregate. If you have multiple drive types in your system (heterogeneous storage), you should understand how you can ensure that the correct drive type is selected. About this task Drives and array LUNs are owned by a specific node; when you create an aggregate, all drives in that aggregate must be owned by the same node, which becomes the home node for that aggregate. Aggregate names should conform to the naming scheme you determined when you planned your MetroCluster configuration. The Clustered Data ONTAP Data Protection Guide contains more information about mirroring aggregates. Steps 1. Display a list of available spares: storage disk show -spare -owner node_name 2. Create the aggregate by using the storage aggregate create -mirror true command. If you are logged in to the cluster on the cluster management interface, you can create an aggregate on any node in the cluster. To ensure that the aggregate is created on a specific node, use the -node parameter or specify drives that are owned by that node. You can specify the following options: Aggregate's home node (that is, the node that owns the aggregate in normal operation) List of specific drives or array LUNs that are to be added to the aggregate Number of drives to include Checksum style to use for the aggregate Type of drives to use Size of drives to use Drive speed to use RAID type for RAID groups on the aggregate

141 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 141 Maximum number of drives or array LUNs that can be included in a RAID group Whether drives with different RPM are allowed For more information about these options, see the storage aggregate create man page. The following command creates a mirrored aggregate with 10 disks: controller_a_1::> storage aggregate create aggr1_controller_a_1 - diskcount 10 -node controller_a_1 -mirror true [Job 15] Job is queued: Create aggr1_controller_a_1. [Job 15] The job is starting. [Job 15] Job succeeded: DONE 3. Verify the RAID group and drives of your new aggregate: storage aggregate show-status -aggregate aggregate-name Ensuring redundant protection for the CRS metadata volume Before you begin About this task Step 1. Installing licenses for the new controller module You must add licenses for the new controller module for any Data ONTAP services that require standard (node-locked) licenses. For features with standard licenses, each node in the cluster must have its own key for the feature. About this task For detailed information about licensing, see the knowledgebase article : Data ONTAP 8.2 Licensing Overview and References on the NetApp Support Site and the System Administration Reference Manual. Steps 1. If necessary, obtain license keys for the new node on the NetApp Support Site in the My Support section under Software licenses. If the site does not have the license keys you need, contact your sales or support representative. 2. Issue the following command to install each license key: system license add -license-code license_key The license_key is 28 digits in length. Repeat this step for each required standard (node-locked) license.

142 142 MetroCluster Service Guide Creating unmirrored data aggregates You can optionally create unmirrored data aggregates for data that does not require the redundant mirroring provided by MetroCluster configurations. Unmirrored aggregates are not protected in the event of a site disaster or if a site wide maintenance in one site is performed. Before you begin You should know what drives or array LUNs will be used in the new aggregate. If you have multiple drive types in your system (heterogeneous storage), you should understand how you can be sure that the correct drive type is selected. About this task Attention: Data in unmirrored aggregates in the MetroCluster configuration is not protected in the event of a MetroCluster switchover operation. Drives and array LUNs are owned by a specific node; when you create an aggregate, all drives in that aggregate must be owned by the same node, which becomes the home node for that aggregate. Aggregate names should conform to the naming scheme you determined when you planned your MetroCluster configuration. The Clustered Data ONTAP Data Protection Guide contains more information about mirroring aggregates. Steps 1. Display a list of available spares: storage disk show -spare -owner node_name 2. Create the aggregate by using the storage aggregate create command. If you are logged in to the cluster on the cluster management interface, you can create an aggregate on any node in the cluster. To verify that the aggregate is created on a specific node, use the -node parameter or specify drives that are owned by that node. You can specify the following options: Aggregate's home node (that is, the node that owns the aggregate in normal operation) List of specific drives or array LUNs that are to be added to the aggregate Number of drives to include Checksum style to use for the aggregate Type of drives to use Size of drives to use Drive speed to use RAID type for RAID groups on the aggregate Maximum number of drives or array LUNs that can be included in a RAID group Whether drives with different RPM are allowed For more information about these options, see the storage aggregate create man page.

143 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 143 The following command creates a mirrored aggregate with 10 disks: controller_a_1::> storage aggregate create aggr1_controller_a_1 - diskcount 10 -node controller_a_1 [Job 15] Job is queued: Create aggr1_controller_a_1. [Job 15] The job is starting. [Job 15] Job succeeded: DONE 3. Verify the RAID group and drives of your new aggregate: storage aggregate show-status -aggregate aggregate-name Verifying storage failover on both controller modules and setting cluster HA You must ensure that storage failover is enabled on both controller modules. If you have a two-node cluster, you must separately enable cluster HA. Steps 1. Verify that HA mode is enabled by entering the following command: storage failover show The output should be similar to the following: Node Partner Possible State Description old-ctlr new-ctlr true Connected to new-ctlr new-ctlr old-ctlr true Connected to old-ctlr 2 entries were displayed. 2. If storage failover is not enabled, enable it and verify that it is enabled: a. Enter the following command: storage failover modify -enabled true -node existing-node-name The single command enables storage failover on both controller modules. b. Verify that storage failover is enabled by entering the following command: storage failover show The output should be similar to the following: Node Partner Possible State Description old-ctlr new-ctlr true Connected to new-ctlr new-ctlr old-ctlr true Connected to old-ctlr 2 entries were displayed. 3. If the cluster has only two nodes, enter the following command to enable cluster HA: cluster ha modify -configured true

144 144 MetroCluster Service Guide Cluster high availability (HA) must be configured in a cluster if it contains only two nodes and it differs from the HA provided by storage failover. You can skip this step if the cluster has more than two nodes. Installing the firmware after adding a controller module After adding the controller module, you must install the latest firmware on the new controller module to ensure that the controller module functions properly with ONTAP. Step 1. Log in to the NetApp Support Site, select the most current version of firmware for your system from those listed at support.netapp.com/now/cgi-bin/fw, and then follow the instructions for downloading and installing the new firmware. After you finish If you have not already done so, repeat the tasks for adding a second controller on the MetroCluster partner site. Refreshing the MetroCluster configuration with new controllers You must run the metrocluster configure command to start data protection in the MetroCluster configuration. Before you begin There must be at least two nonroot mirrored data aggregates on each cluster, and all aggregates must be mirrored. You can verify this with the storage aggregate show command. The ha-config state of the controllers and chassis must be mcc. This state is preconfigured on systems shipped from the factory. The intracluster connection must be working. About this task You issue the metrocluster configure command once, on any of the nodes, to enable the MetroCluster configuration. You need not issue the command on each of the sites or nodes, and it does not matter which node or site you choose to issue the command on. The metrocluster configure command automatically pairs the two nodes with lowest system IDs in each of the two clusters as disaster recovery (DR) partners. In a four-node MetroCluster configuration, there are two DR partner pairs. The second DR pair is created from the two nodes with higher system IDs. Steps 1. Refresh the MetroCluster configuration: metrocluster configure -refresh true The following command enables MetroCluster configuration on all nodes in the DR group that contains controller_a_1:

145 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 145 controller_a_1::*> metrocluster configure -node-name controller_a_1 [Job 726] Job succeeded: Configure is successful. 2. Verify the networking status on site A: network port show The following example shows the network port usage on a four-node MetroCluster configuration: cluster_a::> network port show Speed (Mbps) Node Port IPspace Broadcast Domain Link MTU Admin/Oper controller_a_1 e0a Cluster Cluster up 9000 auto/1000 e0b Cluster Cluster up 9000 auto/1000 e0c Default Default up 1500 auto/1000 e0d Default Default up 1500 auto/1000 e0e Default Default up 1500 auto/1000 e0f Default Default up 1500 auto/1000 e0g Default Default up 1500 auto/1000 controller_a_2 e0a Cluster Cluster up 9000 auto/1000 e0b Cluster Cluster up 9000 auto/1000 e0c Default Default up 1500 auto/1000 e0d Default Default up 1500 auto/1000 e0e Default Default up 1500 auto/1000 e0f Default Default up 1500 auto/1000 e0g Default Default up 1500 auto/ entries were displayed. 3. Confirm the MetroCluster configuration from both sites in the MetroCluster configuration. a. Confirm the configuration from site A: metrocluster show cluster_a::> metrocluster show Cluster Configuration State Mode Local: cluster_a configured normal Remote: cluster_b configured normal b. Confirm the configuration from site B: metrocluster show cluster_b::> metrocluster show Cluster Configuration State Mode Local: cluster_b configured normal Remote: cluster_a configured normal c. Confirm that the DR relationships have been created correctly: metrocluster node show -fields dr-cluster,dr-auxiliary,node-objectlimit,automatic-uso,ha-partner,dr-partner

146 146 MetroCluster Service Guide metrocluster node show -fields dr-cluster,dr-auxiliary,node-object-limit,automaticuso,ha-partner,dr-partner dr-group-id cluster node ha-partner dr-cluster dr-partner dr-auxiliary nodeobject-limit automatic-uso cluster_a rha1-a1 node_a_2 cluster_b node_b_1 node_b_2 on true 2 cluster_a rha1-a2 node_a_1 cluster_b node_b_2 node_b_1 on true 2 cluster_b rha1-b1 node_b_2 cluster_a node_a_1 node_a_2 on true 2 cluster_b rha1-b2 node_b_1 cluster_a node_a_2 node_a_1 on true 4 entries were displayed. Checking the MetroCluster configuration You can check that the components and relationships in the MetroCluster configuration are working correctly. You should do a check after initial configuration, and after making any changes to the MetroCluster configuration, and before a negotiated (planned) switchover or a switchback operation. After you run the metrocluster check run command, you then display the results of the check with various metrocluster check show commands. About this task If the metrocluster check run command is issued twice within a short time, on either or both clusters, a conflict can occur and the command might not collect all data. Subsequent metrocluster check show commands will not show the expected output. Steps 1. Check the configuration: metrocluster check run The command will run as a background job: controller_a_1::> metrocluster check run [Job 60] Job succeeded: Check is successful. Run "metrocluster check show" command to view the results of this operation. controller_a_1::> metrocluster check show Last Checked On: 5/22/ :23:28 Component Result nodes ok lifs ok config-replication ok aggregates ok clusters ok 5 entries were displayed. 2. Display more detailed results from the most recent metrocluster check run command: metrocluster check aggregate show metrocluster check cluster show metrocluster check config-replication show metrocluster check lif show

147 Expanding a two-node MetroCluster configuration to a four-node MetroCluster configuration 147 metrocluster check node show The metrocluster check show commands show the results of the most recent metrocluster check run command. You should always run the metrocluster check run command prior to using the metrocluster check show commands to ensure that the information displayed is current. The following example shows the metrocluster check aggregate show output for a healthy four-node MetroCluster configuration: controller_a_1::> metrocluster check aggregate show Last Checked On: 8/5/ :42:58 Node Aggregate Check Result controller_a_1 aggr0_controller_a_1_0 mirroring-status ok disk-pool-allocation ok controller_a_1_aggr1 mirroring-status ok disk-pool-allocation ok controller_a_2 aggr0_controller_a_2 mirroring-status ok disk-pool-allocation ok controller_a_2_aggr1 mirroring-status ok disk-pool-allocation ok controller_b_1 aggr0_controller_b_1_0 mirroring-status ok disk-pool-allocation ok controller_b_1_aggr1 mirroring-status ok disk-pool-allocation ok controller_b_2 aggr0_controller_b_2 mirroring-status ok disk-pool-allocation ok controller_b_2_aggr1 mirroring-status ok disk-pool-allocation ok 16 entries were displayed. The following example shows the metrocluster check cluster show output for a healthy four-node MetroCluster configuration. It indicates that the clusters are ready to perform a negotiated switchover if necessary. Last Checked On: 5/22/ :23:28 Cluster Check Result cluster_a negotiated-switchover-ready ok switchback-ready notapplicable job-schedules ok licenses ok cluster_b negotiated-switchover-ready ok switchback-ready notapplicable job-schedules ok licenses ok 8 entries were displayed.

148 148 MetroCluster Service Guide Verifying the configuration with the Config Advisor The Config Advisor utility verifies that the controller modules are properly configured for failover. This utility checks licenses, network configurations, options, and so on, and provides output that shows when error conditions occur. Steps 1. Go to the Config Advisor page on the NetApp Support Site at support.netapp.com/now/ download/tools/config_advisor/. 2. Use the links and information on the page to download and run the tool. Verifying switchover, healing, and switchback You should verify the switchover, healing, and switchback operations of the MetroCluster configuration. Step 1. Use the procedures for negotiated switchover, healing, and switchback that are mentioned in the MetroCluster Management and Disaster Recovery Guide. ONTAP 9.0 MetroCluster Management and Disaster Recovery Guide

149 149 Glossary of MetroCluster terms aggregate A grouping of physical storage resources (disks or array LUNs) that provides storage to volumes associated with the aggregate. Aggregates provide the ability to control the RAID configuration for all associated volumes. data SVM Formerly known as data Vserver. In clustered Data ONTAP, a Storage Virtual Machine (SVM) that facilitates data access from the cluster; the hardware and storage resources of the cluster are dynamically shared by data SVMs within a cluster. admin SVM Formerly known as admin Vserver. In clustered Data ONTAP, a Storage Virtual Machine (SVM) that has overall administrative access to all objects in the cluster, including all objects owned by other SVMs, but does not provide data access to clients or hosts. inter-switch link (ISL) A connection between two switches using the E-port. destination The storage to which source data is backed up, mirrored, or migrated. disaster recovery (DR) group The four nodes in a MetroCluster configuration that synchronously replicate each others' configuration and data. disaster recovery (DR) partner A node's partner at the remote MetroCluster site. The node mirrors its DR partner's NVRAM or NVMEM partition. disaster recovery auxiliary (DR auxiliary) partner HA pair The HA partner of a node's DR partner. The DR auxiliary partner mirrors a node's NVRAM or NVMEM partition in the event of an HA takeover after a MetroCluster switchover operation. In Data ONTAP 8.x, a pair of nodes whose controllers are configured to serve data for each other if one of the two nodes stops functioning. Depending on the system model, both controllers can be in a single chassis, or one controller can be in one chassis and the other controller can be in a separate chassis. In the Data ONTAP 7.3 and 7.2 release families, this functionality is referred to as an active/active configuration. HA partner A node's partner within the local HA pair. The node mirrors its HA partner's NVRAM or NVMEM cache. high availability (HA) healing In Data ONTAP 8.x, the recovery capability provided by a pair of nodes (storage systems), called an HA pair, that are configured to serve data for each other if one of the two nodes stops functioning. In the Data ONTAP 7.3 and 7.2 release families, this functionality is referred to as an active/active configuration.

150 150 MetroCluster Service Guide The two required MetroCluster operations that prepare the storage located at the DR site for switchback. The first heal operation resynchronizes the mirrored plexes. The second heal operation returns ownership of root aggregates to the DR nodes. LIF (logical interface) A logical network interface, representing a network access point to a node. LIFs currently correspond to IP addresses, but could be implemented by any interconnect. A LIF is generally bound to a physical network port; that is, an Ethernet port. LIFs can fail over to other physical ports (potentially on other nodes) based on policies interpreted by the LIF manager. NVRAM nonvolatile random-access memory. NVRAM cache Nonvolatile RAM in a storage system, used for logging incoming write data and NFS requests. Improves system performance and prevents loss of data in case of a storage system or power failure. NVRAM mirror node A synchronously updated copy of the contents of the storage system NVRAM (nonvolatile random access memory) contents kept on the partner storage system. In Data ONTAP, one of the systems in a cluster or an HA pair. To distinguish between the two nodes in an HA pair, one node is sometimes called the local node and the other node is sometimes called the partner node or remote node. In Protection Manager and Provisioning Manager, the set of storage containers (storage systems, aggregates, volumes, or qtrees) that are assigned to a dataset and designated either primary data (primary node), secondary data (secondary node), or tertiary data (tertiary node). A dataset node refers to any of the nodes configured for a dataset. A backup node refers to either a secondary or tertiary node that is the destination of a backup or mirror operation. A disaster recovery node refers to the dataset node that is the destination of a failover operation. remote storage The storage that is accessible to the local node, but is at the location of the remote node. root volume A special volume on each Data ONTAP system. The root volume contains system files and configuration information, and can also contain data. It is required for the system to be able to boot and to function properly. Core dump files, which are important for troubleshooting, are written to the root volume if there is enough space. switchback The MetroCluster operation that restores service back to one of the MetroCluster sites. switchover The MetroCluster operation that transfers service from one of the MetroCluster sites. A negotiated switchover is planned in advance and cleanly shuts down components of the target MetroCluster site. A forced switchover immediately transfers service; the shut down of the target site might not be clean.

151 151 Copyright information Copyright NetApp, Inc. All rights reserved. Printed in the U.S. No part of this document covered by copyright may be reproduced in any form or by any means graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system without prior written permission of the copyright owner. Software derived from copyrighted NetApp material is subject to the following license and disclaimer: THIS SOFTWARE IS PROVIDED BY NETAPP "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp. The product described in this manual may be protected by one or more U.S. patents, foreign patents, or pending applications. RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS (October 1988) and FAR (June 1987).

152 152 Trademark information NetApp, the NetApp logo, Go Further, Faster, AltaVault, ASUP, AutoSupport, Campaign Express, Cloud ONTAP, Clustered Data ONTAP, Customer Fitness, Data ONTAP, DataMotion, Fitness, Flash Accel, Flash Cache, Flash Pool, FlashRay, FlexArray, FlexCache, FlexClone, FlexPod, FlexScale, FlexShare, FlexVol, FPolicy, GetSuccessful, LockVault, Manage ONTAP, Mars, MetroCluster, MultiStore, NetApp Insight, OnCommand, ONTAP, ONTAPI, RAID DP, RAID-TEC, SANtricity, SecureShare, Simplicity, Simulate ONTAP, Snap Creator, SnapCenter, SnapCopy, SnapDrive, SnapIntegrator, SnapLock, SnapManager, SnapMirror, SnapMover, SnapProtect, SnapRestore, Snapshot, SnapValidator, SnapVault, StorageGRID, Tech OnTap, Unbound Cloud, and WAFL and other names are trademarks or registered trademarks of NetApp, Inc., in the United States, and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. A current list of NetApp trademarks is available on the web.

153 153 How to send comments about documentation and receive update notifications You can help us to improve the quality of our documentation by sending us your feedback. You can receive automatic notification when production-level (GA/FCS) documentation is initially released or important changes are made to existing production-level documents. If you have suggestions for improving this document, send us your comments by . To help us direct your comments to the correct division, include in the subject line the product name, version, and operating system. If you want to be notified automatically when production-level documentation is released or important changes are made to existing production-level documents, follow Twitter You can also contact us in the following ways: NetApp, Inc., 495 East Java Drive, Sunnyvale, CA U.S. Telephone: +1 (408) Fax: +1 (408) Support telephone: +1 (888)

154 154 MetroCluster Service Guide Index 6500N bridges, FibreBridge support for replacing with FibreBridge 7500N bridges in MetroCluster configurations N FibreBridge bridges hot-swapping with a pair of FibreBride 7500N bridges N bridges, FibreBridge examples of zoning in eight-node or four-node MetroCluster configurations 41 support for, in MetroCluster configurations 9 A aggregates confirming that they are mirrored when expanding a MetroCluster configuration 111 mirrored data, creating on each node of a MetroCluster configuration 140, 142 aggregates, root mirroring when expanding a MetroCluster configuration 134 assignments, port for FC switches in four-node configuration 122 ATTO bridges hot-swapping 12 installing with disk shelves to increase storage capacity 90 available spare disks adding a second controller 114 checking before upgrade to HA pair 114 B bridge maintenance performing FC-to-SAS, introduction to 9 bridge-attached MetroCluster configurations introduction to hot-adding SAS storage to 82 bridges hot-replacing failed power supply modules 41 hot-swapping 12 bridges, FC-to-SAS consolidating multiple storage stacks 21 disabling unused ports 38 introduction to hot-adding SAS storage to MetroCluster systems with 82 verifying connectivity 11, 22 bridges, FibreBridge requirements for using serial port, Telnet, and FTP to manage 40 updating firmware 10 bridges, FibreBridge 6500N hot-swapping with a pair of FibreBride 7500N bridges 23 hot-swapping with FibreBridge 7500N bridges 17 support for replacing with FibreBridge 7500N bridges in MetroCluster configurations 9 bridges, FibreBridge 7500 updating zoning when consolidating storage behind 32 bridges, FibreBridge 7500N cabling the SAS ports when consolidating storage 28 cabling the second FC port when consolidating storage on 36 support for, in MetroCluster configurations 9 bridges, SAS preparing to hot-add 88 Brocade FC switch replacing 63 Brocade FC switches port assignments in four-node configuration 122 upgrading firmware 59 zoning in a four-node MetroCluster configuration 54 Brocade switches zoning examples for a FibreBridge 7500N bridge in eight-node or four-node MetroCluster configurations 41 C cables moving SAS 74 replacing SAS 74 cables, SAS copper replacing 39 cabling CNA ports 127 FC-VI ports in a MetroCluster configuraiton 120 hot-added disk shelves 78 cabling procedures hot-swapping FibreBridge 6500N bridges with a pair of 7500N bridges 23 chassis verifying the HA state of when expanding a MetroCluster configuration 128 checking MetroCluster configuration operations 146 Cisco FC switches port assignments in four-node configuration 122 replacing 67 upgrading firmware on 61 zoning in a four-node MetroCluster configuration 54 Cisco switches zoning examples for a FibreBridge 7500N bridge in eight-node or four-node MetroCluster configurations 41 cluster peering connections cabling in MetroCluster configurations 127 cluster ports configuring on an existing controller module when expanding a MetroCluster configuration 115 CNA ports cabling 127 changing the defaults 127 configuring 127 verifying the configuration on the new controller 127 commands

155 Index 155 for starting data protection 144 for verifying storage connectivity 11, 22 comments how to send feedback about documentation 153 components, MetroCluster hardware where to find correct procedures for replacing 7 Config Advisor verifying configuration with 148 configurations expanding MetroCluster 144 verifying with Config Advisor 148 configurations, bridge-attached MetroCluster introduction to hot-adding SAS storage to 82 configurations, four-node port assignments for FC switches 122 configurations, MetroCluster cabling HBAs to FC switches 121 support for using FibreBridge 7500N bridges 9 configuring CNA ports 127 connections, cluster peering cabling in MetroCluster configurations 127 connectivity, storage verifying before replacing FC-to-SAS bridges 11, 22 controller installing and cabling when adding 118 controller module installing firmware on after adding 144 verifying the HA state of when expanding a MetroCluster configuration 128 controller modules adding to create an HA pair in a MetroCluster configuration 113 assigning disks when expanding a MetroCluster configuration 129 cabling FC-VI ports in a MetroCluster configuraiton 120 cabling HBAs to FC switches in MetroCluster configurations 121 configuring cluster ports when expanding a MetroCluster configuration 115 prerequisites for expanding MetroCluster configuration from two to four nodes 110 setting up ONTAP on new controller modules when expanding a MetroCluster configuration 130 controller ports checking connectivity with the partner site in MetroCluster configurations 127 controllers identifying when expanding a MetroCluster configuration 111 copper cables, SAS replacing 39 D data aggregates mirrored, creating on each node of a MetroCluster configuration 140, 142 data ports configuring intercluster LIFs to share 138 data ports, dedicated intercluster configuring intercluster LIFs to use 134 data protection mirroring root aggregates when expanding a MetroCluster configuration 134 dedicated data ports, intercluster configuring intercluster LIFs to use 134 disaster recovery relationships confirming when expanding a MetroCluster configuration 111 disk drives or carriers verifying firmware versions 80 disk shelves cabling hot-added 78 hot-adding 97 hot-removing from MetroCluster configurations 104 hot-removing in a direct-attached MetroCluster 100 verifying firmware versions after hot-adding 80 disk shelves, SAS installing to increase storage capacity 90 preparing to hot-add 96 preparing to hot-add bridges and stacks of 88 requirements for hot-adding 76 disks assigning to new controller modules when expanding a MetroCluster configuration 129 documentation how to receive automatic notification of changes to 153 how to send feedback about 153 where to find MetroCluster documentation 5 E eight-node or four-node MetroCluster configurations example zoning for a FibreBridge 7500N bridge and Brocade switch 41 example zoning with a FibreBridge 7500N bridge 41 example zoning FibreBridge 6500N bridges 54 examples zoning for FibreBridge 6500N bridges in a four-node MetroCluster configuraiton 54 examples, zoning FibreBridge 7500N bridge in eight-node or fournode MetroCluster configurations 41 existing controller shutting down the 118 existing controller module setting the HA mode on when expanding a MetroCluster configuration 117 F failed power supply modules hot-replacing 41 failover verifying configuration of, with Config Advisor 148 FC ports cabling when consolidating storage with FibreBridge 7500N bridges 36 disabling on FC-to-SAS bridges 38 FC switch replacing Brocade switches 63

156 156 MetroCluster Service Guide FC switch zoning updating when expanding a MetroCluster configuration to four nodes 112 FC switches cabling controller FC-VI ports in a MetroCluster configuraiton 120 cabling HBAs to, in MetroCluster configurations 121 port assignments in four-node configuration 122 replacing Cisco 67 upgrading firmware on Brocade 59 upgrading firmware on Cisco 61 FC-to-SAS bridges consolidating multiple storage stacks 21 disabling unused ports 38 hot-swapping 12 installing with disk shelves to increase storage capacity 90 introduction to hot-adding SAS storage to MetroCluster systems with 82 introduction to performing maintenance on 9 updating zoning when consolidating storage behind for FibreBridge 7500N bridges 32 verifying connectivity 11, 22 FC-VI ports cabling in a MetroCluster configuraiton 120 zoning examples when using FibreBridge 7500 bridges 41 FC/SAS bridge upgrades hot-swapping a pair of FibreBridge 6500N bridges with a pair of FibreBridge 7500N bridges 23 feedback how to send comments about documentation 153 FibreBridge 6500N bridges hot-swapping with a pair of FibreBride 7500N bridges 23 hot-swapping with FibreBridge 7500N bridges 17 replacing with FibreBridge 7500N bridges 21 support for replacing with FibreBridge 7500N bridges in MetroCluster configurations 9 zoning configuration 54 FibreBridge 7500 bridges updating zoning when consolidating storage behind 32 FibreBridge 7500N bridges cabling SAS ports when consolidating storage 28 cabling the second FC port when consolidating storage on 36 examples of zoning in eight-node or four-node MetroCluster configurations 41 hot-swapping FibreBridge 6500N bridges with 17 support for, in MetroCluster configurations 9 FibreBridge bridges hot-adding 82 hot-adding to a MetroCluster system 88 requirements for using serial port, Telnet, and FTP to manage 40 updating firmware 10 firmware installing after adding a controller module 144 updating FibreBridge bridge 10 upgrading on Brocade FC switches 59 firmware updates verifying disk, shelf 80 four-node configurations port assignments for FC switches 122 four-node MetroCluster configurations prerequisites for expanding two-node MetroCluster configurations to 110 H HA mode setting on existing controller module on when expanding a MetroCluster configuration 117 HA pair creating by adding a second controller to a site in a MetroCluster configuration 113 preparing for upgrade to 114 HA state setting 128 verifying when expanding a MetroCluster configuration 128 ha-config modify command using verifying when expanding a MetroCluster configuration 128 ha-config show command using when expanding a MetroCluster configuration 128 hardware components, MetroCluster where to find correct procedures for replacing 7 HBAs cabling to FC switches in MetroCluster configurations 121 healing verifying in a MetroCluster configuration 148 hot-add adding SAS shelves 82 adding SAS shelves and FibreBridges 88 Hot-add a SAS shelf 96 hot-adding disk shelves 97 installing disk shelf 77 SAS disk shelves using SAS optical cables, requirements for 76 SAS disk shelves, preparing to 96 SAS shelf stacks and bridges, preparing to 88 verifying disk, shelf and firmware after 80 hot-adding disk shelves cabling 78 enabling ACP 78 hot-removing disk shelves from MetroCluster configurations 104 disk shelves in a direct-attached MetroCluster 100 hot-replacing failed power supply modules 41 hot-swapping bridges 12 cabling the SAS ports of FibreBridge 7500N bridges when consolidating storage 28 cabling the second FC port on FibreBridge 7500N bridges when consolidating storage 36 FibreBridge 6500N bridges with FibreBridge 7500N bridges 17 hot-swapping same model bridge 12

157 Index 157 hot-swaps a FibreBridge 11 FC/SAS bridge 11 FibreBridge 6500N bridges to a pair of FibreBridge 7500N bridges 23 I information how to send feedback about improving documentation 153 initiator ports zoning examples when using FibreBridge 7500 bridges 41 installing power supply modules 41 installing firmware after adding a controller module 144 intercluster LIFs configuring to share data ports 138 configuring to use dedicated intercluster ports 134 intercluster networks configuring intercluster LIFs for 134, 138 intercluster ports configuring intercluster LIFs to use dedicated 134 interfaces, management requirements for using serial port, Telnet, and FTP to manage FibreBridge bridges 40 L license keys adding to the new controller module 141 LIFs, intercluster configuring to share data ports 138 configuring to use dedicated intercluster ports 134 M maintenance performing FC-to-SAS bridge, introduction to 9 maintenance tasks where to find correct procedures for performing on MetroCluster hardware components 7 management interfaces requirements for using serial port, Telnet, and FTP to manage FibreBridge bridges 40 MetroCluster configuration cabling cluster peering connections 127 expanding by adding a controller module to each cluster 113 installing and cabling a new controller in 118 setting up ONTAP on new controllers in 130 MetroCluster configurations cabling FC-VI ports 120 cabling HBAs to FC switches 121 creating mirrored data aggregates on each node of 140, 142 expanding 144 introduction to hot-adding storage 76 introduction to hot-removing storage 100 prerequisites for expanding from two to four nodes 110 support for using FibreBridge 7500N bridges 9 verifying correct operation of 146 MetroCluster configurations, bridge-attached introduction to hot-adding SAS storage to 82 MetroCluster hardware components where to find correct procedures for replacing 7 metrocluster refresh command adding MetroCluster relationships 144 modules, controller cabling HBAs to FC switches in MetroCluster configurations 121 moving SAS cables 74 SAS copper cables 39 multiple storage stacks consolidating FC-to-SAS bridges 21 N netboot server preparing the 117 netbooting using to set up the new controller when expanding a MetroCluster configuration 130 networks, intercluster configuring intercluster LIFs for 134 new controller module adding license keys 141 new controller modules setting up ONTAP when expanding a MetroCluster configuration 130 setting up using netboot when expanding a MetroCluster configuration 130 nodes creating mirrored data aggregates on each MetroCluster 140, 142 prerequisites for expanding MetroCluster configuration from two to four nodes 110 numbers, port for Brocade FC switches in four-node configurations 122 for Cisco FC switches in four-node configurations 122 O onboard SAS ports requirements for hot-adding a disk shelf as a new stack 76 ONTAP operating system installing latest controller module firmware to ensure compatibility with 144 installing on new controller modules when expanding a MetroCluster configuration 130 P partner clusters cabling cluster peering connections in MetroCluster configurations 127

158 158 MetroCluster Service Guide peering connections, cluster cabling in MetroCluster configurations 127 port assignments for FC switches in four-node configuration 122 port numbers for Brocade FC switches in four-node configurations 122 for Cisco FC switches in four-node configurations 122 ports, cluster configuring on an existing controller module when expanding a MetroCluster configuration 115 ports, controller checking connectivity with the partner site in MetroCluster configurations 127 ports, data configuring intercluster LIFs to share 138 ports, FC cabling when consolidating storage with FibreBridge 7500N bridges 36 disabling on FC-to-SAS bridges 38 zoning configuration 54 ports, HBA cabling to FC switches in MetroCluster configurations 121 ports, intercluster configuring intercluster LIFs to use dedicated 134 ports, SAS cabling on FibreBridge 7500N bridges when consolidating storage 28 disabling on FC-to-SAS bridges 38 power supply modules, failed hot-replacing 41 preparing to hot-add SAS shelf stacks and bridges 88 R RCF files when expanding a MetroCluster configuration to four nodes 112 reboot and verify storage failover om the controller modules 143 relationships adding MetroCluster 144 removing disk shelves from MetroCluster configurations 104 disk shelves in a direct-attached MetroCluster 100 replace a FibreBridge 11 FC/SAS bridge 11 replacements hot-swapping a pair of FibreBridge 6500N bridges with a pair of 7500N bridges 23 replacing failed power supply modules 41 FC-to-SAS bridges with bridges of the same model 12 MetroCluster hardware components, where to find correct procedures for 7 SAS cables 74 SAS copper cables 39 requirements for upgrading to HA pair 114 for using serial port, Telnet, and FTP to manage FibreBridge bridges 40 hot-adding SAS disk shelves using SAS optical cables 76 root aggregates mirroring when expanding a MetroCluster configuration 134 S SAS bridges installing to increase storage capacity 90 preparing to hot-add 88 SAS cables moving 74 replacing 74 SAS copper cables replacing 39 SAS disk shelves installing to increase storage capacity 90 preparing to hot-add 96 SAS HBAs requirements for hot-adding a disk shelf 76 SAS ports cabling on FibreBridge 7500N bridges when consolidating storage 28 disabling on FC-to-SAS bridges 38 SAS shelves hot-adding to a FibreBridge 7500N 82 hot-adding to a MetroCluster system 88 preparing to hot-add stack of 88 SAS storage introduction to hot-adding to bridge-attached MetroCluster configurations 82 server preparing the netboot server 117 setting the HA mode on an existing controller module when expanding a MetroCluster configuration 117 shelves cabling hot-added 78 hot-adding 97 hot-removing from MetroCluster configurations 104 hot-removing in a direct-attached MetroCluster 100 preparing to hot-add SAS bridges and stacks of 88 shelves, SAS preparing to hot-add 96 requirements for hot-adding 76 shut down 118 site FC switches cabling HBAs to, in MetroCluster configurations 121 spares checking when adding a second controller 114 stacks preparing to hot-add SAS bridge and shelf 88 stacks, multiple storage consolidating FC-to-SAS bridges 21 storage cabling SAS ports on FibreBridge 7500N bridges when consolidating 28 updating zoning when consolidating behind FibreBridge 7500 bridges 32

159 Index 159 storage configuration assigning disks to the controller module when expanding a MetroCluster configuration 129 storage connectivity verifying before replacing FC-to-SAS bridges 11, 22 storage controller installing and cabling when adding 118 installing firmware on after adding 144 storage controllers adding to create an HA pair in a MetroCluster configuration 113 assigning disks when expanding a MetroCluster configuration 129 cabling FC-VI ports in a MetroCluster configuraiton 120 setting up ONTAP on new controller modules when expanding a MetroCluster configuration 130 storage failover verifying on the controller modules after reboot 143 storage stacks, multiple consolidating FC-to-SAS bridges 21 storage, SAS introduction to hot-adding to bridge-attached MetroCluster configurations 82 suggestions how to send feedback about documentation 153 switchback verifying in a MetroCluster configuration 148 switches replacing Cisco FC 67 upgrading firmware on Brocade FC 59 upgrading firmware on Cisco FC 61 switches, FC cabling HBAs to, in MetroCluster configurations 121 port assignments in four-node configuration 122 switchover verifying in a MetroCluster configuration 148 T tasks, maintenance where to find correct procedures for performing on MetroCluster hardware components 7 the existing controller 118 Twitter how to receive automatic notification of documentation changes 153 two-node MetroCluster configurations prerequisites for expanding to four nodes 110 U unused ports disabling on FC-to-SAS bridges 38 upgrades meeting requirements for upgrade 114 upgrades, FC/SAS bridge hot-swapping a pair of FibreBridge 6500N bridges with a pair of FibreBridge 7500N bridges 23 utilities verifying configuration with Config Advisor 148 V verifying configuration with Config Advisor 148 MetroCluster configuration operations 146 Z zones for new controller ports when expanding a MetroCluster configuration to four nodes 112 zoning differences between eight-node and four-node MetroCluster configuraitons 41 examples for FibreBridge 6500N bridges in a fournode MetroCluster configuration 54 updating when consolidating storage behind FibreBridge 7500N bridges 32 zoning examples FibreBridge 7500N bridge in eight-node or fournode MetroCluster configurations 41