VLAN Tagging and Routing on EMC CLARiiON

VLAN Tagging and Routing on EMC CLARiiON Best Practices Planning Abstract This white paper describes how to configure and manage virtual local area networks (VLANs) for EMC CLARiiON iscsi ports and management ports. It also includes routing best practices for management and iscsi ports configured with or without VLAN tagging. October 2009

Copyright 2009 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part Number h6634.1 Best Practices Planning 2

Table of Contents Executive summary...4 Introduction...4 Audience... 4 Terminology... 4 VLAN overview...5 VLAN implementation on CLARiiON...6 VLAN tagging... 6 How to configure VLANs for CLARiiONs without VLAN tagging... 6 How to configure VLANs for CLARiiONs with VLAN tagging (FLARE release 29)... 7 VLAN operations... 7 Sending and receiving untagged frames... 7 Sending and receiving tagged frames... 8 Exceptions when using a native VLAN... 8 Configuring VLAN tagging with Navisphere...8 Configuring a management port for VLAN tagging... 8 Configuring an iscsi port for VLAN tagging... 10 iscsi connections for SAN Copy and MirrorView... 13 Routing on CLARiiON...13 Management traffic is routed through 10 Gb/s iscsi, or 10 Gb/s iscsi traffic is routed through the management port... 14 Multiple 10 Gb/s iscsi data ports on the same subnet... 17 Routing best practices... 19 Conclusion...19 References...19 Best Practices Planning 3

Executive summary EMC strives to allow its customers to do more with less, and to improve the total cost of ownership. To achieve this goal, EMC has introduced a new advanced IEEE Standard feature called VLAN tagging on CLARiiON midtier storage systems. By deploying virtual LANs (VLANs), you can create multiple virtual LANs (as opposed to multiple physical connections) in your Ethernet infrastructure. VLANs help reduce traffic, decrease costs, improve performance, increase security, and provide more connectivity options. In FLARE release 29, CLARiiON storage systems added support for VLAN tagging on 1 Gb/s and 10 Gb/s iscsi interfaces and storage-processor management ports. With VLAN tagging, CLARiiON provides even more connectivity options for iscsi and management interfaces. Introduction With 10 Gb/s and 1 Gb/s iscsi I/O modules on CLARiiON systems, it is important to consider best practices when integrating CLARiiON storage systems in your environment. The best practices in this white paper will help you maximum performance and minimize disruption when configuring your environment. This paper provides an overview of CLARiiON VLAN tagging. It also provides best practices for configuring routing (with or without VLAN tagging) through iscsi and management ports. Audience The intended audiences for this white paper are members of the EMC field community who interact with customers and need to understand how to configure VLANs on CLARiiON storage systems, and are interested in implementing 1 Gb/s or 10 Gb/s iscsi technologies in their environment. Terminology Default gateway In configurations with multiple interfaces and subnets, each having its own gateway, the default gateway is the node used by a host when an IP packet's destination address is outside the local subnet(s). Each interface on a host can have a gateway, but there can only be one default gateway. Default VLAN ID When a trunk port on the switch receives an untagged frame, the frame is assigned the default VLAN ID. For example, assume a trunk port is configured with VLAN ID 172, 201, 202, and the default VLAN ID is 172. Any untagged frame received on the trunk port will be tagged with VLAN ID 172. Gateway In a TCP/IP network, the gateway is the logical address that identifies the router that is used to access hosts not on the local subnets. Multihomed Typically used to describe a host that has two or more network addresses or is connected to two or more networks. Native VLAN ID A virtual switch uses the native VLAN ID (usually VLAN ID 0001) to associate or flow untagged frames through the switching fabric. Ports must be configured for the default VLAN ID to send or receive untagged frames. Router A physical device that links multiple separate networks. Subnet Typically, a subset of an organization's network. For example, a subnet may consist of all the machines at one geographic location, in one building, or on the same LAN. Switch port A physical connection on a switch that can support VLANs. Best Practices Planning 4

Tagged switch port A switch port that tags Ethernet frames with a specific VLAN tag. A tagged switch port tags a frame by inserting the VLAN tag into the frame. Trunk switch port A switch port that sends Ethernet frames to other switches or hosts. Trunk ports use the VLAN ID to decide where to send frames. Virtual network adapter A logical adapter associated with a physical network interface. To an operating system, a virtual network adapter appears to be a real physical adapter. VLAN ID A 12-bit number that varies from 1-4095. The VLAN ID specifies which logical network a frame can go to if routing is not used. VLAN tag A 4-byte field added to an Ethernet frame that is used to route a frame to the correct virtual network adapter interface. VLAN trunk Allows a single physical network adapter to behave as a number of virtual network adapters. VLAN overview A VLAN is a virtual local area network (LAN). Unlike a LAN, hosts and storage systems on a VLAN do not have to be located near each other. VLANs provide advantages including increased security and scalability, superior network management, and more connectivity options. Figure 1 shows an example in which a company wants each department to communicate over a separate LAN, but only wants to purchase one physical network. The company does this by assigning a different VLAN to each department. The VLANs share the same physical network. Communications sent over the network include VLAN tags that specify which VLAN the message belongs to. Figure 1. VLAN architecture Best Practices Planning 5

VLAN implementation on CLARiiON VLANs establish multiple broadcast domains through switch-port association, MAC addresses, or networklayer parameters (such as IP subnets). Switch-port association allows CLARiiON storage systems to understand and process the VLAN tags on messages coming from a network switch. VLAN tagging A VLAN-tagged frame carries an explicit identification of the VLAN to which it belongs. It carries this non-null VLAN ID within the frame header. CLARiiON storage systems support the IEEE 802.1Q VLAN tagging on the network. It therefore uses only the VLAN ID field and not the priority field as shown in Figure 2. The format for VLAN tags is shown in Figure 2. Please note that in IEEE 802.1Q, a priority is not included so this field is null. Figure 2. Format of an Ethernet frame with a VLAN tag To configure VLAN tagging on a CLARiiON, it must be running FLARE release 29 or later. CLARiiON storage systems running FLARE 29 or earlier can also be part of a VLAN without implementing VLAN tagging. How to configure VLANs for CLARiiONs without VLAN tagging Configure a network-switch port as a tagged-switch port with a VLAN identifier and connect the CLARiiON iscsi port or management network port to the tagged-switch port. This is how clients are typically configured for VLANs. The CLARiiON is unaware that it is part of the VLAN, and no special configuration of the CLARiiON is needed. The VLAN ID is set to zero since no VLAN tagging is configured on the CLARiiON storage system. As shown in Figure 3, a router is used to route frames to the correct hosts since hosts and storage systems are in different networks. Figure 3. VLANs for CLARiiON without VLAN tagging Best Practices Planning 6

How to configure VLANs for CLARiiONs with VLAN tagging (FLARE release 29) Configure a network-switch port as a trunk port with a VLAN identifier and connect the CLARiiON iscsi port or management-network port to the trunk port. The trunk port allows the CLARiiON to be a member of multiple VLANs, and their corresponding subnets, through a single physical connection. The CLARiiON interprets the VLAN tags and processes the packets. The CLARiiON also sends tagged frames that include the VLAN ID for the source address. It drops frames that do not map to the virtual network interface. As shown in Figure 4, CLARiiON can implement multiple VLAN interfaces, each having its own IP address and VLAN ID. VLAN tagging on the CLARiiON reduces the network latency on routers; a Layer 2 switch configured with a trunk port can connect to the CLARiiON storage system port instead of a router. Figure 4. Configuring CLARiiON with a Layer 2 switch and a trunk port VLAN operations Sending and receiving untagged frames When VLAN tagging is not enabled, untagged frames can be sent from the CLARiiON to a host in the same VLAN. When the untagged frame reaches the switch, the switch port adds a VLAN tag to the frame based on the destination MAC address. The switch then sends the frame to the destination port (that is a member of the same VLAN) unless it is a broadcast frame, in which case the frame is sent to all of the members on the VLAN. As the frame exits the switch, the VLAN information is stripped from the frame and the frame is sent to the respective host; however if the frame is sent from a trunk port the VLAN tag information is not stripped out. If a host sends an untagged frame to the switch, the switch port adds a VLAN tag to the frame based on the destination MAC address. The switch then sends the frame to all ports that are members of the VLAN. As the frame exits the switch, the VLAN info is stripped from the frame and the frame is sent to the CLARiiON; however if the frame is sent from a trunk port the VLAN tag information is not stripped out. Best Practices Planning 7

Sending and receiving tagged frames A tagged frame sent from the CLARiiON must go through a trunk port on the switch, otherwise the frame is dropped. In addition, the trunk port must be in the VLAN specified in the tagged frame, otherwise the frame is dropped. If the trunk port is in the same VLAN as the tagged frame, the switch sends the tagged frame to the destination port that is a member of the VLAN, unless it is a broadcast frame, in which case the frame is sent to all members of the same VLAN. As the frame exits the switch, the VLAN info is stripped from the frame and the frame is sent to the respective host; however if the frame is sent from a trunk port the VLAN tag information is not stripped out. A tagged frame sent from the host must go through a trunk port on the switch, otherwise the frame is dropped. In addition, the trunk port must be in the VLAN specified in the tagged frame, otherwise the frame is dropped. If the trunk port is in the same VLAN, the switch sends the frame to all ports that are members of the VLAN. As the frame exits the switch, the VLAN info is stripped from the frame and the frame is sent to the CLARiiON. If the CLARiiON storage system is connected to a trunk port, the VLAN tag information is not stripped out. Exceptions when using a native VLAN You can also configure a native VLAN on a network switch. Ports belonging to a native VLAN do not have frames tagged. Untagged frames are sent to all members of the native VLAN. When an untagged frame designated to a native VLAN is received on a trunk port, the trunk port does not drop the frame; instead it receives the untagged frame. Configuring VLAN tagging with Navisphere VLAN tagging is supported for management ports and iscsi ports on 1 Gb/s and 10 Gb/s I/O modules. With virtual ports, the number of iscsi ports for connectivity has greatly increased. Table 1 lists the maximum number of VLANs for iscsi and management ports that are supported for each storage processor. Table 1. VLAN limits for the CLARiiON storage system Interface VLANs per port 1 Gb/s iscsi 2 10 Gb/s iscsi 8 SP management port 1 Configuring a management port for VLAN tagging Management ports are configured in the Network Properties Management dialog box shown in Figure 5. As illustrated, you must select the Enable VLAN Tagging checkbox and provide a VLAN ID. Best Practices Planning 8

Figure 5. Configuring a VLAN tag for a CLARiiON management port Navisphere Secure CLI get and set commands now display virtual port information for SPs. The output of the network get command for SPA follows: Best Practices Planning 9

Configuring an iscsi port for VLAN tagging 1. Open the Port Management dialog box, select the iscsi port, and click the Properties tab on the bottom to open the iscsi Port Properties dialog box shown in Figure 6. In the iscsi Port Properties dialog box, click Add to open the iscsi Virtual Port Properties dialog box. Figure 6. Select a specific iscsi for VLAN configuration Best Practices Planning 10

2. In the SCSI Virtual Port Properties dialog box shown in Figure 7, select Enable VLAN Tagging and specify a VLAN ID for the port. When configuring the switch port to connect to this iscsi port, ensure that the switch port and all the hosts are configured with the same VLAN ID so that they are all on the same VLAN. Additional virtual ports for that iscsi port can be configured if needed. If an iscsi or SP management port is already configured prior to upgrading to release 29 of FLARE, the previous port parameter will be assigned to virtual port 0. The VLAN ID information will be listed as disabled. If necessary, you can then configure a VLAN tag to that of the iscsi or SP management port. Figure 7. Enable VLAN tagging for a specific iscsi port Best Practices Planning 11

3. After configuring the iscsi port with the VLAN tag, click OK and ensure that your new virtual port is listed in the Virtual Port Properties section in the iscsi Port Properties dialog box. Figure 8. Configuring Enable VLAN tagging for a specific iscsi port The Navisphere Secure CLI command iscsi getport and setport commands also display virtual port information for SPs. The output of the network get command for SPA follows: Best Practices Planning 12

iscsi connections for SAN Copy and MirrorView Virtual ports can be configured for SAN Copy and MirrorView connections. When MirrorView or SAN Copy automatically creates an iscsi connection between storage systems, they try to use a nonrouted (on the same subnet) path between storage systems. If there are several non-routed options available, they use the option with the lowest virtual port number. If you do not want to use the default selection criteria for iscsi connections, you can create connections manually as shown in Figure 9. This allows you to choose specific virtual ports to use for MirrorView or iscsi connections. Figure 9. Selecting virtual ports for MirrorView or SAN Copy iscsi connections between storage systems For more details on configuring iscsi ports with MirrorView and SAN Copy, please see the MirrorView Knowledgebook: FLARE 29 A Detailed Review and the EMC CLARiiON SAN Copy A Detailed Review white papers available on Powerlink. Routing on CLARiiON In a weak host model, an IP host (either IPv4 or IPv6) can send packets on an interface that is not assigned the source IP address of the packet being sent. An IP host can also receive packets on an interface that is not assigned the host s IP. This is known as weak host receive behavior. With weak host behavior enabled on both its data and management interfaces, the storage system shown in Figure 10 can send or receive packets from 172.20.1.140 and 10.5.3.140 on its iscsi interface, and packets from 172.20.1.140 and 10.5.3.140 on its management interface Best Practices Planning 13

Figure 10. Multihomed network model CLARiiON storage systems implement a weak host model. FLARE, CLARiiON s operating system, uses the OS TCP/IP stack to maintain all network connections. FLARE does not handle multiple default routes, hence only one default gateway can be configured in the FLARE OS TCP/IP stack. TCP offload engine (TOE) iscsi functionality is still maintained, but the connection setup for the network interfaces is done through the OS TCP/IP protocol stack. The 1 Gb/s iscsi interfaces offload the entire iscsi stack onto the 1 Gb/s I/O module. Consequently, all processing regarding the care and handling of connections is done by the TCP/IP software that runs on the 1 Gb/s adapter on the I/O module. The 10 Gb/s iscsi interfaces take a different approach. This architecture uses the storage system s TCP/IP stack to perform initial connection setup and then offloads the processing to the 10 Gb/s on-board card after the session is established. Because FLARE implements the weak host model, the response to a packet received on, say, a 10 Gb/s iscsi interface can just as easily be returned via, say, a CLARiiON SP management port if the routing tables direct it to do so since both the 10 Gb/s iscsi and CLARiiON management port uses the FLARE operating system TCP/IP stack. This poses these potential problems: Management data traffic is routed to a 10 Gb/s iscsi data interface. 10 Gb/s iscsi data traffic is routed through the management port interface. With multiple 10 Gb/s iscsi interfaces, the wrong interface may be chosen. To help alleviate these problems, CLARiiON has made a few enhancements and recommendations with FLARE release 29. For array-to-array connections, host-specific routes are created under the covers to ensure that the correct interface is used. If all else fails, CLI route commands have been implemented that allow users to manually intervene to ensure that the correct interface is used. The management port is given the lower route metric of all other interfaces. A lower route metric (number) refers to the most desirable route while a higher route metric refers to a least desirable route. A more in-depth discussion of potential problems is provided in the following sections. Management traffic is routed through 10 Gb/s iscsi, or 10 Gb/s iscsi traffic is routed through the management port Management traffic can be routed over an iscsi interface, or iscsi traffic can be routed over a management interface when: The host s iscsi interface is not on the storage system s iscsi subnet(s). The host has a single interface and that interface is on the storage system s iscsi subnet. Best Practices Planning 14

Figure 11. Host on a different subnet from the storage system interfaces In Figure 11, the only path the storage system has to the host is through its router(s). Since the management port is given the highest priority, in some cases, when the storage system receives a request from the host, the storage systems will mistakenly route the response through the management interface instead of the iscsi interface. (As mentioned earlier, this only happens with 10 Gb/s iscsi interfaces; it does not occur with 1 Gb/s iscsi interfaces.) The only way to guarantee that the response is correctly routed from the correct interface of the storage system is to force the traffic to that host onto a specific iscsi interface. To accomplish that, EMC added the route option to the naviseccli connection command. With this option, a host-specific route for the 10 Gb/s interface is shown below: Naviseccli connection route sp a add portid 3 vportid 0 -destination 128.222.132.65 subnetmask 255.255.255.255 metric 10 persistent However, what happens if there is traffic from the host that you want to send to the storage system s management interface? For example, this can occur with management utilities such as naviseccli, host agent, or the Navisphere Server Utility. If the host route was added, the responses will be returned to the host via the 10 Gb/s iscsi interface instead of the management interface. The best solution to this problem is to ensure that all the iscsi clients reside on the same subnets as the array s iscsi ports as shown in Figure 12. This circumvents any routing issues and prevents latency and performance issues. However, there is a problem with this solution, too. Best Practices Planning 15

Figure 12. Hosts on the same subnet Figure 13 demonstrates that the single host interface resides on the storage system s iscsi subnet. In this case, the iscsi data traffic exits through the correct interface without an additional route being added because the host resides on one of the storage system s subnets. Furthermore, if this is a 10 Gb/s interface, the iscsi session will be offloaded to the 10 Gb/s TOE after the session is established. But what path does the management traffic take? When the host issues commands to the storage system s management IP address, it will enter through the storage system s management port. However, the responses will exit through the storage system s 10 Gb/s iscsi data port, because the host s IP address matches one of the storage system s subnets. By providing separate dedicated network interfaces on the host for management and iscsi data traffic as shown in Figure 12, no additional route is necessary and iscsi data traffic will flow normally between the systems. Network traffic actually addressed to the management port will be returned via the management port because that is the default gateway. Best Practices Planning 16

Figure 13. Dedicated interfaces on the host for management and iscsi traffic Multiple 10 Gb/s iscsi data ports on the same subnet If multiple data ports are configured on the same subnet, we no longer recommend that you configure multiple ports (either single physical ports or virtual ports) on the same subnet, even though it is possible to do this. As stated above, the iscsi data ports share the same OS routing priority, and this can lead to unexpected results, as shown in Figure 14. In this example, both 10 Gb/s links are assigned to the 172.20.1.0/24 subnet. Since the host is on a different subnet, a host-specific route must be added. Otherwise, as we have seen in previous examples, packets would be returned to the host via the management port interface. Note, however, that with both interfaces on the same subnet, adding a route on one interface will direct all traffic to the host to that interface, and adding a second host-specific route to a different interface is not allowed. The result is that this host cannot connect to multiple target portals on the storage system. Best Practices Planning 17

Figure 14. Dual 10 Gb/s interfaces on the same subnet with the host on a disjoint subnet To solve this problem, we recommend that you ensure that all iscsi clients reside on the same subnets as the storage systems iscsi ports reside on, as shown in Figure 15. Empirical evidence has shown that packets addressed to 172.20.1.140 (SPA, Port 2) are returned via SPA, Port 2. Furthermore, packets addressed to 172.20.1.141 (SPA, Port 3) are returned on SPA, Port 3. Ethernet-link captures on the host side confirm that the source MAC address of the returning packets corresponds with the interface matching the respective IP address. The sessions also get offloaded to the correct 10 Gb/s adapter port. Figure 15. 10 Gb/s dual interfaces and host on the same subnet Best Practices Planning 18

Routing best practices All network ports on the CLARiiON (including management ports and iscsi ports) should be on different subnets. Configure the host IP address and SP iscsi ports to be in the same VLAN/subnet. If the host iscsi interface is on an IP subnet that is not on the storage system s IP subnet, a host route must be added to force iscsi traffic through a specific iscsi interface on the storage system. The management interface can be on any subnet. Conclusion The addition of virtual ports (and VLAN trunking and tagging) to the CLARiiON storage system greatly enhances its ability to participate on many different IP subnets, thus reducing network latency on routers in the network as well as improving performance, tightening security, and reducing costs. The addition of the 10 Gb/s I/O interface is a great step forward in the evolution of the CLARiiON product; however the changes necessary to incorporate the technology require users to follow certain routing guidelines when configuring their IP network infrastructure. References The following documents and resources can be found on Powerlink, EMC s password-protected extranet for partners and customers: Navisphere Manager Help 6.29 Navisphere Command Line Interface (CLI) Reference 6.29 MirrorView Knowledgebook: FLARE 29 A Detailed Review white paper EMC CLARiiON SAN Copy A Detailed Review white paper Managing EMC CLARiiON iscsi Data and Control Traffic in a Routed IP Storage Network Environment Technical Note EMC Networked Storage Topology Guide E-Lab Navigator Best Practices Planning 19