EqualLogic PS Series Reference Architecture for Juniper Networks EX4500

Transcription

1 EqualLogic PS Series Reference Architecture for Juniper Networks EX4500 Two-Switch SAN Reference Storage Infrastructure and Solutions Engineering Dell Product Group April 2011

2 THIS REPORT IS FOR INFORMATIONAL PURPOSES ONLY AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Dell, the DELL logo, and the DELL badge, PowerConnect, Dell EqualLogic, PowerEdge and PowerVault are trademarks of Dell Inc. Juniper Networks, the Juniper Networks logo, Junos, NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and other countries. Intel is a registered trademark of Intel Corporation in the U.S. and other countries. Microsoft, Windows, Windows Server, and Active Directory are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. i

3 Table of Contents Table of Tables... iii Table of Figures... iii 1 Introduction Objective Conclusions Reference Architecture Overview About the Juniper EX4500 Switch Reference Architecture Summary Reference Architecture Details Server Configuration Array Configuration Switch Configuration Overview of the Switch Configuration Global Switch Setting: Enabling Spanning Tree Protocol Array and Server Port Configuration Interfaces Spanning Tree Edge Mode Configuration Link Aggregation Group (LAG) Configuration...13 Appendix A Switch Port Mapping Appendix B TCP/IP Configuration B.1 Arrays B.2 Hosts ii

4 Table of Tables Table 1 Server Specifications and Configuration Settings... 9 Table 2 Workload to Storage Pool Mappings Table 3 Switch Configuration Overview Table of Figures Figure 1- Two-Switch / 12-Array Reference Configuration... 8 iii

5 Acknowledgements This publication was produced by the PG Storage Infrastructure and Solutions team in April 2011 at the Dell Labs facility in Round Rock, Texas. Development team: Gregory (Brent) Seaton, Tony Ansley Feedback We encourage readers of this publication to provide feedback on the quality and usefulness of this information. You can submit feedback as follows: Use the Post a new thread link here: 4

6 1 Introduction The Juniper Networks EX4500 line of Ethernet Switches delivers a scalable, compact, highperformance platform for supporting high-density 10 gigabit per second (Gbps) data center, campus, and service provider deployments. The EX4500 offers an economical, power efficient, and compact solution for iscsi storage, providing simplicity, scalability, availability, adaptability, and modularity to a network. This document illustrates how to build a storage area network (SAN) consisting of two (2) Juniper EX4500 switches using industry standard link aggregation groups (LAGs) for inter-switch connections for use with EqualLogic PS Series storage. It also provides data to support our position that the EX4500 can support a SAN using a two-switch solution, and that it can support twelve (12) or more 10 Gb/s arrays with twelve (12) or more 10 Gb/s hosts in a fully redundant configuration while offering excellent overall SAN performance for a variety of workloads. 2 Objective This document provides details on the configuration of the Juniper EX4500 switch, leveraging industry-standard link aggregation groups (LAGs) for inter-switch connections for use with EqualLogic PS Series storage. The goal of this exercise is not to provide a comprehensive set of possible configurations, but to illustrate one possible solution that provides excellent performance and scalability as validated by testing in our labs. 3 Conclusions Dell Labs conducted testing that proves the value of the Juniper EX4500 when used to build a SAN using the Dell EqualLogic series of virtual storage arrays. Based on this testing, we determined that the EX4500 is an excellent switch for use when building SANs that consist of one or more 10-Gbps EqualLogic PS 6010 arrays. Our lab tests of this SAN solution yielded the following results: With two Juniper Networks EX4500 switches, the SAN scaled easily to support 12+ arrays and/or 12+ hosts. Sequential Write performance scaled linearly to 88% of the theoretical baseline in terms of throughput as measured at the host. Sequential Read Performance scaled linearly to 92% in terms of throughput as measured at the host. The Random Read/Write performance in terms of IOs per second scaled linearly to 99% of the theoretical baseline on a per server performance basis. TCP retransmissions from arrays, as polled periodically from array counters and SAN Headquarters (SANHQ), were low (< 0.1%) across all test configurations. This is another 5

7 indicator that there are no bottlenecks or design issues within the switch that limited the ability of the switch to support the reference architecture. In conclusion, our test results indicate that the only limitation this reference architecture has with respect to scaling an EqualLogic SAN is in the number of available ports that two switches provide, not with any limitations in the switch s performance. We conducted additional testing that demonstrates that the Juniper Networks EX4500 switch can support larger SAN solutions if additional switches are added to the configuration. A reference architecture paper for a four-switch SAN solution is also available if your SAN requires a larger number of arrays or hosts. 4 Reference Architecture Overview 4.1 About the Juniper EX4500 Switch The Juniper EX4500 switch is a non-blocking, low latency, high-density 10-gigabit switch with 40 fixed ports. EX4500 switches support a LAG interconnect by using up to eight SFP+ 10 GbE line-rate links, allowing switches to reside in different locations. The EX4500 offers many different options to fit the customer s storage area network (SAN) configuration requirements, including providing modular expansion for an additional eight 10-GbE ports and an optional 128Gbps Virtual Chassis stacking solution for up to two switches. The reference architecture used in our testing uses only the switch s forty (40) fixed ports. It does not use the Virtual Chassis feature or make use of the modular expansion ports. 4.2 Reference Architecture Summary The reference architecture described in this paper consists of two (2) Juniper Networks EX4500 switches. Connected to these switches are a total of twelve (12) PS6010 arrays and twelve (12) PowerEdge servers, which make use of all fixed ports available on the two (2) EX4500 switches. In this reference architecture, the switches are inter-connected using a multi-link aggregation group (LAG) trunk, based on industry standard protocols to provide adequate bandwidth between switches to support the resulting configuration. Because our goal is to help you deploy a switch-based SAN easily and quickly, we use a standard host configuration and a standard EqualLogic Group configuration, accept default switch settings wherever possible, and employ all accepted best-practice recommendations for EqualLogic SANs. When developing this reference architecture, we used the following guidelines: All hosts have two (2) Ethernet ports attached to the SAN. All SAN attached ports are configured based on default, out of the box settings where possible. The exceptions are the use of Jumbo Frames and Flow Control, both of which are enabled in the configuration. All hosts run Microsoft Windows Server. 6

8 We use EqualLogic s Host Integration Toolkit for all hosts. In particular, we use the MPIO Device Specific Module to provide EqualLogic-aware multipathing. Host connections to the SAN equal the number of active array ports connected to the SAN. o Since each PS6x10 series array has two (2) active array ports, each array in the test configuration has one (1) host connected to the SAN. o The ratio of arrays to hosts is based solely on the available number of ports when using two switches. o Although this approach might not meet every customer s requirements, it simplifies the testing, allowing us to ensure an adequate number of hosts are available to run the various workloads tested and to generate enough load to ensure that the Juniper EX4500 can manage the traffic for a maximum number of arrays. The SAN is on a single, non-routed subnet. The SAN uses standard IPv4 addressing. Our tests use three pre-defined standardized workloads that reflect various types of real-world SAN utilization. Note: For more information on EqualLogic SAN design common practice recommendations, consult the EqualLogic Configuration Guide that can be found at 5 Reference Architecture Details The reference architecture described in this paper consists of two (2) Juniper Networks EX4500 switches. The switches are inter-connected using a multi-link aggregation group (LAG) trunk, based on industry standard protocols to provide adequate bandwidth between switches to support the resulting configuration. Connected to these switches are a total of twelve (12) PS6010 arrays and twelve (12) PowerEdge servers, which make use of all fixed ports available on the two (2) EX4500 switches. The twelve (12) arrays are configured into four (4) storage pools. Each pool consists of three (3) arrays and is dedicated to a different workload type to ensure maximum performance of volumes for each workload profile. Figure 1 illustrates the resulting SAN reference architecture. 7

9 Figure 1- Two-Switch / 12-Array Reference Configuration 8

10 5.1 Server Configuration This reference architecture uses twelve Dell PowerEdge R610 rack servers. Table 1 provides the model specifications and configuration settings for each server. Table 1 Server Specifications and Configuration Settings Power Edge R610 Specifications BIOS Intel chipset OS A02 Windows Server 2008 R2 Host Network Interface Configuration (x2 per server) Model Dell Broadcom BCM57711 Dual port iscsi Offload Engine (isoe) TCP Offload Engine (ToE) NDIS Mode iscsi Initiator Disabled Disabled Enabled Microsoft Windows Server 2008 R2 BCM Software Firmware: isoe driver: NDIS driver: VBD driver: MPIO Configuration Dell EqualLogic Host Integration Toolkit Version Dell EqualLogic MPIO Device Specific Module Maximum Sessions per Slice: 2 Maximum Sessions per Volume: Array Configuration This reference architecture uses twelve (12) EqualLogic PS Series arrays. All arrays in the storage group are the same model (EqualLogic PS6010XV ) and use the same Array Software version (version 4.3.7). The arrays in the storage group are divided into four (4) storage pools, each of which consists of three (3) arrays. Each storage pool is assigned to process a single type of workload: Random Read/Write, Sequential Read, or Sequential Write. 9

11 Table 2 specifies the assignment of workloads to each pool in the group. Table 2 Workload to Storage Pool Mappings Storage Pool Workload Type Volumes Storage Pool 1 Storage Pool 2 Storage Pool 3 Storage Pool 4 Random Read/Write Sequential Read Sequential Write 4 x 100 GB volumes per server RAID Switch Configuration For the purpose of validating the switches for use with EqualLogic storage, the EX4500 is configured with the standard forty (40) fixed ports. Four (4) of these ports are configured into a dynamic LAG port channel using the IEEE Link Aggregation Control Protocol (LACP) standard. Our reference architecture does not make use of the Virtual Chassis feature. Based on our analysis of the switch architecture and lab testing, the following subsections specify the commands and settings recommended to configure the switches in this reference architecture Overview of the Switch Configuration Table 3 provides an overview of the switch configuration for this SAN reference architecture. Table 3 Switch Configuration Overview Juniper Networks EX 4500 Specifications Switch hardware revision Routing Engine 0 Rev 07 Switch firmware JUNOS 10.4R1.9 Cable Types Host switch cable type Array switch cable type Switch switch cable type Host: Dell SFP+SR Optical Transceiver (DP/NON743D) Switch: Juniper SFP+SR Optical Transceiver, LC-LC Fiber Optic Cable Array: Dell SFP+SR Optical Transceiver (DP/NON743D) Switch: Juniper SFP+SR Optical Transceiver, LC-LC Fiber Optic Cable SFP+Copper Twinax ( Rev A0) 1 meter cable 10

12 Configuration Settings Switch inter-connection Global switch setting Individual port settings Dynamic Link Aggregation Group (LACP LAG) Flow Control enabled on each port channel group MTU=9216 on each port channel group Spanning-tree mode (rstp) enabled MTU=9216 RSTP Edge mode (portfast) enabled Flow Control enabled Global Switch Setting: Enabling Spanning Tree Protocol Spanning Tree Protocol (RSTP) is a global configuration setting on the Juniper EX4500 that is disabled by default. Use the following commands from the VLAN configuration mode to enable spanning tree for all ports the switch: root@ex4500:re:0% cli {master:0} root@ex4500> configure Entering configuration mode root@ex4500# set protocols rstp root@ex4500# commit Array and Server Port Configuration All ports assigned to connect to server or array interfaces are configured identically. To simplify configuration of these ports, an interface-range is defined named xe0. For all ports in this range, Jumbo frames are enabled by setting mtu equal to 9216, flow control is enabled by setting etheroptions flow control, and the ports are placed in Layer 2 (switching) mode by setting unit 0 family Ethernet-switching. For this reference architecture, switch ports xe-0/0/0 through xe-0/0/35 on each switch are used for either server or array connections. Use the following commands to configure the array and server ports on each switch: {master:0} root@ex4500> configure Entering configuration mode root@ex4500# set interfaces interface-range xe0 member-range xe-0/0/0 to xe-0/0/35 root@ex4500# edit interfaces interface-range xe0 11

13 {master:0}[edit interfaces interface-range xe0] set mtu 9216 {master:0}[edit interfaces interface-range xe0] set ether-options flow-control {master:0}[edit interfaces interface-range xe0] set unit 0 family ethernet-switching {master:0}[edit interfaces interface-range xe0] root@ex4500# exit root@ex4500# commit Run show config to confirm the configuration. The results should be similar to the following: interfaces { interface-range xe0 { member-range xe-0/0/0 to xe-0/0/35; mtu 9216; ether-options { flow-control; } unit 0 { family ethernet-switching; } } Interfaces Spanning Tree Edge Mode Configuration Spanning tree should only be enabled for those ports assigned to inter-switch communication. Ports connected to servers and arrays should not be used in spanning tree calculation, during which time ports do not forward traffic. This could introduce latency for those ports, or could cause storage applications to time out. Spanning tree is a global parameter on the switch, and unless the switch is instructed otherwise, all ports are included in spanning tree recalculations. Placing ports into Edge Mode (portfast mode) instructs the switch not to include the ports in spanning tree calculation, enabling the ports to continue forwarding traffic during the time spanning tree responds to network changes. After you create the interface group used for ports connecting to servers and arrays, the interface group needs to be configured for RSTP Edge Mode. Use the following commands in the RSTP configuration context to configure interface range xe0 for RSTP Edge Mode: {master:0} root@ex4500> configure Entering configuration mode 12

14 set protocols rstp interface xe0 edge commit Run show config to confirm the configuration. The results should be similar to the following: protocols { rstp { interface xe0 { edge; } } Link Aggregation Group (LAG) Configuration The LAG consists of four (4) 10-Gb links providing an aggregate bandwidth between each switch of 40 Gbps. Ports of each switch are configured into a dynamic LAG port channel using the IEEE LACP standard. In our reference architecture, the single LAG used to interconnect the two Juniper EX4500 switches is configured by creating an interface-range named lag1, and the interfaces in that interface-range are placed into a LAG named ae0. Parameters to be configured for ae0 are as follows: mtu 9216 aggregated-ether-options lacp passive lacp messages interval fast The LAG is also placed in Layer 2 switching mode by configuring unit 0 family Ethernet-switching with port-mode access. As a security feature, the LAG cannot be enabled for traffic management until the Juniper EX4500 switches have been configured to allow aggregated Ethernet devices. This is achieved with the set chassis aggregated-devices Ethernet device-count 1 command. Note: The device-count parameter refers to the number of virtual devices, in this case the number of LAGs configured on this switch. On each switch, use the following commands to configure the single LAG used to interconnect the EX4500 switches: {master:0} root@ex4500> configure Entering configuration mode root@ex4500# set interfaces interface-range lag1 member-range xe-0/0/36 to xe-0/0/39 root@ex4500# set interfaces interface-range lag1 ether-options 802.3ad ae0 13

15 set interfaces ae0 mtu 9216 aggregated-ether-options lacp passive periodic fast set interfaces ae0 unit 0 family ethernet-switching port-mode access root@ex4500# set chassis aggregated-devices ethernet device-count 4 root@ex4500# commit 14

16 Appendix A Switch Port Mapping H1P1 H3P1 H5P1 A1C1P0 A2C1P0 A3C1P0 A4C1P0 A5C1P0 A6C1P0 A7C1P0 H7P1 H9P1 H11P1 A8C1P0 A9C1P0 A10C1P0 A11C1P0 A12C1P0 S2P36 S2P38 H2P1 H4P1 H6P1 A1C2P0 A2C2P0 A3C2P0 A4C2P0 A5C2P0 A6C2P0 A7C2P0 H8P1 H10P1 H12P1 A8C2P0 A9C2P0 A10C2P0 A11C2P0 A12C2P0 S2P37 S2P SW H1P2 H3P2 H5P2 A1C1P1 A2C1P1 A3C1P1 A4C1P1 A5C1P1 A6C1P1 A7C1P1 H7P2 H9P2 H11P2 A8C1P1 A9C1P1 A10C1P1 A11C1P1 A12C1P1 S1P36 S1P38 H2P2 H4P2 H6P2 A1C2P1 A2C2P1 A3C2P1 A4C2P1 A5C2P1 A6C2P1 A7C2P1 H8P2 H10P2 H12P2 A8C2P1 A9C2P1 A10C2P1 A11C2P1 A12C2P1 S1P37 S1P SW2 Array Connections: AxCyPz = Array x, Controller y, Port z (on that controller) Link Aggregation Connections: SxPy = Switch x, Port y Host Connections HxPy = Host x, NIC Port y 15

17 Appendix B TCP/IP Configuration B.1 Arrays Array Group IP Address: Array Port 0 IP Address Port 1 IP Address EQL EQL EQL EQL EQL EQL EQL EQL EQL EQL EQL EQL

18 B.2 Hosts Server NIC 1 IP Address NIC 2 IP Address SVR SVR SVR SVR SVR SVR SVR SVR SVR SVR SVR SVR

19 THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.