Position Paper The new data center edge Horizontal Stacking and Switch Clustering

Transcription

1 Position Paper The new data center edge Horizontal Stacking and Switch Clustering Data center requirements The enterprise data center is one of the most critical areas of the network. This is especially true when consolidating applications and services into a single or geo-redundant data center construct. The challenge to reduce overall costs is realized as servers, applications and network services are moved into centralized data centers. This consolidation of critical resources places a greater burden on network availability and performance. 24 x 7 access to applications and data stores is a common requirement in most enterprises. Any disruption in network service within the data center has repercussions throughout the business. These effects are different in nature, scope and severity depending on the type of enterprise, but the effects are all negative and likely impact the enterprise s revenue. The availability of applications begins with the availability of the network. The ability to nearly eliminate both planned and unplanned network downtime is an important factor when designing the next-generation data center. The need for continuous availability makes it nearly impossible to schedule any outages for network upgrades or routine maintenance. A data center architecture that provides the capability to schedule maintenance without interrupting the availability of the data center network gives the network administrator increased flexibility to maintain the network. The other major factor for design consideration is Total Cost of Ownership (TCO). A data center solution must be cost-effective without sacrificing performance or resiliency while still paying attention to the need to be as eco-friendly (Green IT) as possible. TCO is often measured in two areas capital expenditures (CAPEX) and operational expenditures (OPEX). The initial cost to purchase the equipment and services is considered CAPEX, while all other costs to maintain the network fall into OPEX. Over the lifetime of the network, OPEX costs can largely outweigh CAPEX. This can be especially true when looking at the

2 requirements for maintenance contracts and the energy costs associated with operating the equipment. The ongoing operation and management of the network are OPEX costs that aren t as easily visible. If the network is simple to operate and manage, then the amount of time network administrators spend on these tasks is reduced, thus freeing them up to do other work and increasing their overall productivity. The Server Access Layer evolution The data center Server Access Layer continues to evolve. This is a direct result of the types and numbers of servers being deployed in the enterprise data centers. When there were fewer physical servers, it made sense to consolidate the Server Access Layer into a few strategic locations within the data center and run cabling out to the servers. This allowed for the best usage of ports on the network equipment and therefore the most economical solution. As the trend moved away from large servers to single purpose 1U rackmounted devices, the amount of network connections increased dramatically. Many of these servers came equipped with multiple Network Interface Cards (NICs), and server administrators were taking advantage of multiple network connections. As an example, a common practice separates the production traffic from the data backup traffic across different network connections. This increased number of connections required many more cable runs from the servers to the network equipment. This initiated the trend to distribute the network equipment out to the server racks as opposed to running all those cables to a centralized network rack. To core To core To core To core To core To core To core To core Figure 1. Each server rack with two uplinks to the core (Gigabit or 10 Gigabit) Today, as the Ethernet switches are At the same time, as this increase in distributed to the server racks, it has the number of servers is happening, become common practice to dual No inter-rack stacking so is the need for high-speed, lowlatency connectivity between servers. home each switch back to the data center Distribution Layer or Core Many applications and services require Layer. The availability requirements extensive communications between the mandate a solution that supports the web, application and data base tiers best utilization of bandwidth with in order to service end-user requests. sub-second failover yet without adding These applications may be latencysensitive and intolerable to any network more complexity. This is the ultimate resiliency model of s Switch delay. High-speed computing and grid Clustering technology discussed later. computing environments also require ultra-fast server-to-server connectivity. Data centers continue to expand due A better solution for inter-server to consolidation efforts and increases connectivity is required as millisecond in the sheer number of applications latency between servers is no longer deployed within the enterprise. This sufficient; it must be reduced to microseconds. This can be accomplished with increases the number of server racks and the number of Ethernet switches s hardware-based forwarding and deployed within the Server Access Layer. stacking technology as described next. A significantly increased number of ports are required in the Distribution/ Core to collect all the Server Access Switch connections. In an effort to curb this sometimes uncontrolled growth and to better utilize compute resources, application virtualization and Blade Center technology have become increasingly popular. This is definitely a step in the right direction; however, it is not the total answer. 2Gbps or 2 x 10Gbps uplinks bandwidth per rack ($$$) 2

3 Stacking technology introduced one of the industry s first true resilient stacking technologies in 1998 with the BayStack 450 Ethernet Switch. This resilient stacking architecture allowed a stack to be managed as a single entity and continue to operate as normal even in the event of a failure of one of the units. Since that time, s Flexible Advanced Stack (FAST) architecture technology has matured considerably. This maturity is seen in functionality with many additional features such as: Ability to add or remove switches in the stack on the fly without any resets or reboots required Auto Unit Replacement performs configuration of the new unit and image upgrade/downgrade to match the existing stack Stack Monitor provides a warning if the size of the stack changes Stack Loopback test allows the network administrator to remotely test stack port and stack cable integrity Stack Counters provide information on traffic flowing across the stack ports Along with feature enhancements, there has also been an appreciable increase in stacking bandwidth. A 2.5 Gigabit stack was sufficient to handle 10/100 ports (BayStack 450), but with the addition of 10/100/1000 and 10 Gigabit ports, there came the need to increase the stacking bandwidth, which now stands at 144 Gigabits per switch and up to Terabits per stack of eight (Ethernet Routing Switch 5600). This increase in bandwidth becomes critical for server-to-server communications within a single stack as well as supporting shared access to common uplinks. Figure 2. Ethernet Routing Switch 5600 Stack The common deployment of stackable switches is vertically stacked within a single rack. These switches are normally installed in wiring closets as edge access ports for users. The stack is created by cabling one switch into another and then completing the stack with a return cable connecting the last switch in the stack to the first switch in the stack. This stacking technology not only provides high bandwidth between switches, but also provides a single management interface, so all are managed together as a single logical unit. Horizontal Stacking By leveraging s resilient stacking architecture with its FAST technology, Horizontal Stacking provides a unique solution to solve today s challenges in the data center Server Access Layer. The concept simply takes the normal vertical stacking of the Ethernet Routing Switches and cables the switches in a horizontal orientation. This now allows switches across multiple racks to be connected as a stack. No special software or configurations are required to take advantage of this unique physical solution. offers various lengths of stacking cables 42cm, 1m, 3m and 5m providing the needed flexibility to connect the switches across multiple server racks. The physical configuration of the horizontal stack is extremely flexible. Figure 3 depicts one permutation in which there is a single switch installed in each rack. However, if needed, multiple switches can be installed in the same rack and horizontally stacked with other switches in adjacent racks. This can be thought of as combining both vertical and horizontal stacking simultaneously. 3

4 Multilink Trunking into Switch Cluster Core provides the collection and analysis of the IP flow data. Figure 3. Extend a stack up to eight racks wide with scalable capacity A Horizontal Stack can range in size from two switches up to eight switches. Switches can be added to the stacks without affecting existing stack operations. This allows for pay-as-yougrow functionality and no need for network downtime to add capacity. The Horizontal Stack can handle up to /100/1000 and Gigabit connections using the Ethernet Routing Switch 5650TD. Any of the Ethernet Routing Switch 5000 series can be stacked together as required to meet the port count and interface requirements in the individual server racks. Ethernet Routing Switches provide a 160 GbE feature to identify their unit number by uplinks illuminating port LEDs on the front bandwidth of the switch that correlate to their unit Extend a stack up to 8 racks total stack distance up to 35 meters number (i.e., unit number 6 will illuminate ports 1 through 6 LEDs). Once the Horizontal Stack has been cabled up correctly, with every switch having two stacking connections, it is recommended to renumber the units in the stack. The base unit should be the leftmost or rightmost unit in the stack and identified as unit #1. Moving sequentially across the racks either left to right or right to left, number the units in the stack as #2, #3, #4, etc. Having the Horizontal Stack in sequential order makes it much easier for ongoing network management of the stack. This ensures that configuration changes are made to the appropriate switch in the appropriate rack. In the event there is uncertainty about the unit numbers in the stack, the By stacking the Ethernet switches, there is a reduction in the number of uplinks required to the Distribution/Core Layer. The number of uplinks and the speed of those uplinks (Gigabit or 10 Gigabit) are dependent on the amount of traffic expected to traverse the uplinks. There is no absolute formula for number of servers to uplink bandwidth as this is dependent on types of applications and amount of data flow expected. In order to help characterize data flow and to assist in troubleshooting, supports IPFIX (IP Flow Information export) which is the IETF standard for flow management based on NetFlow v9. Offering these capabilities at the Server Access Layer allows the network administrator to understand flows to and from users as well as between servers. IPFIX can easily be used for real-time traffic monitoring, troubleshooting, security and trending network traffic for capacity analysis. s IP Flow Manager is the network management software that IPFIX can help identify when there is a need to add uplink capacity. This proactive approach alleviates the uncertainty of uplink utilization and also mitigates the need for any outage to simply add uplink bandwidth when required. supports up to eight physical links within a single trunk group. Providing high-bandwidth and lowlatency connectivity between servers highlights another major advantage in stacking the Ethernet switches across the racks. The Ethernet Routing Switch 5600 supports 144Gbps stacking bandwidth per switch and up to 1.152Tbps of bandwidth in a stack of eight units. Using these capabilities allows a high-bandwidth pipe between servers on the same stack. The Ethernet Routing Switch 5600 also supports the FAST technology which uses a shortest path algorithm across the stack for the most efficient use of that stack bandwidth. The latency for server-to-server communication can be reduced from milliseconds to microseconds when utilizing s stacking technology. The hardware-based forwarding and unique stacking algorithm support a 9µs average port-to-port latency within a stack. This order of magnitude improvement can be critical to time-sensitive applications and is near mandatory when working in high-speed computing and grid computing environments. Another performance benefit realized by stacking is better resource utilization of the Ethernet switches. The more traffic that stays local to the stack suggests there is less traffic that must traverse the uplinks. In many cases, traffic from a server must traverse the uplinks and 4

5 core just to have it come right back to the server in the next cabinet. This is a waste of bandwidth on the uplinks and degrades overall server performance. It is common with many applications that there is significant data flow between servers, and therefore advantageous to engineer the server connectivity to remain within a Horizontal Stack. Horizontal Stacking with Switch Clustering s Switch Clustering technology was introduced in 2001 and offers a simplified alternative to Spanning Tree. Simply aggregate two like Ethernet Routing Switches or stacks (Ethernet Routing Switches 8600, 8300, 1600, 5000) with an Inter-Switch Trunk (IST). This virtualizes the two Switch Cluster Cores and allows them to act as a single logical entity for all devices that are dual attached. No ports or links are blocked and there are no proprietary protocols required from the Switch Cluster Core to the edge devices. Therefore, Switch Clustering is totally interoperable with all third-party devices that support any form of link aggregation. The technology supports an active/ active model for uplinks and sub-second failover, and eliminates the need for Spanning Tree, which in turn significantly reduces the complexity of the network. Likewise, Switch Clustering supports the same active/active connectivity for servers. This now extends the network resiliency from the data center Core all the way to the server NIC. Any server that supports link aggregation can take full advantage of s Switch Clustering technology. This active/active configuration provides added bandwidth to servers and added resiliency with sub-second failover, making it the perfect architecture to support virtualized applications with VmWare and Microsoft s Hyper-V technology. The Ethernet Routing Switch 5600 supports Switch Clustering with an Advanced Software License. By combining the resilient stacking architecture with the resiliency of Switch Clustering using Split Multi- Link Trunking (SMLT) or Single Link Trunking (SLT), the overall data center design provides the highest level of resiliency for all dual connected devices. The ideal design includes installing a switch from each Horizontal Stack in a server cabinet/rack. This would allow dual connections from the server to the Ethernet switches inside a rack making cable installation, maintenance and troubleshooting much easier as all network connections are contained in the server rack. Combining these two technologies provides several key advantages in the data center the most important being the ability to dual connect the servers into network infrastructure in such a way as to allow for any planned or unplanned network outages without interrupting server access. Switch Clustering provides sub-second failover and therefore any outage of a switch or stack will not affect traffic to and from Square or Full Mesh SMLT into Switch Cluster Core IST Figure 4. Resiliency all the way to the server interface Extend a stack up to 8 racks wide Up to Horizontal Stacking Horizontal Stacking with Switch 320Gb Clustering uplink bandwidth Scalability from 640Gbps to 1.152Tbps Scalability from 640Gbps to 1.152Tbps Potential single point of failure (stack) No single point of failure Requires RPSU (ERS 5500) No RPSU (redundant power from rack) Limited MLT scaling SLT scales to all ports in the stack Very cost-effective and energy-efficient Very cost-effective and energy-efficient 5

6 the server. Outages due to hardware failure happen, but are very rare. What is more common is the need to perform network maintenance on a regular basis. Switch Clustering provides the answer to the needed resiliency in either case as there will be no outages to applications, since the network connectivity will always be maintained. Switch Clustering allows the servers to be dual connected to the network in an active/active model, which increases availability and increases available network bandwidth to the server. There is no proprietary software required on the servers to take advantage of s Switch Clustering technology. Simply configure the servers for link aggregation. Terminology used by the various server vendors varies for this functionality, but all support some form whether called NIC teaming, static 802.3ad, Multi- Link Trunking, Sun Trunking, etc.). For network bandwidth-intensive applications, Switch Clustering can support up to eight physical links and speeds of 10, 100, 1000 and 10 Gigabit. It is required that all physical links be of the same speed and duplex setting. For details on NIC teaming and Switch Clustering configurations, please refer to the Data Center Server Access Solution Guide (NN ). The connectivity between the Horizontal Stack Switch Cluster and the data center Switch Cluster Core is comprised of a Square or Full Mesh Split Multi-Link Trunk topology. These topologies ensure resiliency between the Server Access Layer and the data center Core Layer along with being scalable for the number of uplinks and amount of bandwidth required. Figure 5 shows the logical view of the Horizontal Stack Switch Cluster with server connectivity and Square SMLT to the Distribution/Core. SMLT SMLT/SLT Server with NIC team Distribution/ Core Switch Cluster Horizontal Stack Switch Cluster Figure 5. Logical view Price, performance and the importance of Green IT The Ethernet Routing Switch 5600 offers top-of-the-line performance, at a reasonable price. When comparing the switching capacity, stacking bandwidth and feature set, the Ethernet Routing Switch 5000 family stands out as a prime offering. Its unique ability to combine switching performance with features such as Switch Clustering puts the Ethernet Routing Switch 5000 family in a class by itself. The addition of the ERS 5600 adds significant features and capabilities, especially when deployed in the Data Center. By supporting built-in N+1 power redundancy that is hot swappable, the ERS 5600 eliminates the need for any external redundant power supply devices. This saves valuable space in the rack and more importantly reduces the overall cost of the solution while still providing full power redundancy. The advanced chipsets of the ERS 5600 also provide additional features such as many-to-many port mirroring, hardware that is capable of IPv6 routing and integrated IPS-Lite functionality with its Denial of Service (DoS) Attack Prevention Package (DAPP). The advantages don t stop there. In an effort to help reduce OPEX costs specifically around network operations, the Ethernet Routing Switch 5000 delivers a common industry de facto CLI (command line interface) along with TACACS+, making it easy to learn and integrate into any network. now offers industry-leading warranty services for the portfolio of Stackable Ethernet Routing Switches the ERS 2500, ERS 4500 and the ERS 5000 Series product lines. The complimentary next-business-day shipment of failed hardware is now extended to run for the full life of the product. also offers complimentary basic technical support (Level 1) for the supported lifecycle of the product. This includes the software version shipped with or updated through optional SRS Basic, or a separate Service Contract over and above our normal Warranty. The Ethernet Routing Switch 5000 series is also making a critical impact with regard to Green IT. When compared to other industry leaders, the Ethernet Routing Switch 5000 family provides significantly lower power consumption (whether sitting at idle or loaded up to 100 percent). This lower power consumption translates into energy cost savings to power the device. Another benefit is lower BTU output, which in turn reduces the amount of cooling required in the data center, which also translates into energy cost savings. Last, but not least, is the carbon footprint and once again, the Ethernet Routing Switch 5000 family leads the way with a much lower value than comparable switches from other 6

7 vendors 1. For more information on Energy Efficiency, please visit and test out the Energy Efficiency Calculator tool to see how much savings are available with solutions. Conclusion The data center architecture provides significant value to enterprise customers. A key piece of this strategy is the Server Access Layer and the use of Horizontal Stacking and Switch Clustering. Horizontal Stacking provides: High bandwidth and low latency for server-to-server communication within the stack Reduces the number of ports required in the data center Core no need for an uplink per switch Simplifies cabling between server and Ethernet switch keeps all cabling within the server rack Easily add capacity (switches, uplinks) on the fly without any network downtime And when combined with Switch Clustering provides: Dual-homed, active/active server connectivity eliminates the network as a potential point of failure Increased network bandwidth to servers multiple Gigabit or 10 Gigabit connections Simple network design no need for spanning tree or Layer 3 load balancing of NICs Ability to perform network maintenance (i.e., switch software upgrades) with zero impact to server availability Whether used separately or combined together, these features provide added benefits of performance, resiliency, Green IT and lower overall TCO as compared to any other vendor s offering. Please refer to the Cambridge University Case Study posted on the website at: global/emea/cstudies/collateral/ nn123206emea.pdf. 1 The Tolly Group Report Number , July 2008, reported the results of a power consumption evaluation of six converged data network products consisting of large and medium core and wiring closet Ethernet Routing Switches, enterprise branch office routers and IP phones compared to similar Cisco products on the basis of power consumption, heat dissipation and five-year operational cost. See also The Tolly Group report Number , January 2008, and InfoTech Research Group Report, January 30th

8 In the United States: 35 Davis Drive Research Triangle Park, NC USA In Canada: 195 The West Mall Toronto, Ontario M9C 5K1 Canada In Caribbean and Latin America: 1500 Concorde Terrace Sunrise, FL USA In Europe: Maidenhead Office Park, Westacott Way Maidenhead Berkshire SL6 3QH, UK In Asia: United Square 101 Thomson Road Singapore Phone: (65) is a recognized leader in delivering communications capabilities that make the promise of Business Made Simple a reality for our customers. Our next-generation technologies, for both service provider and enterprise networks, support multimedia and business-critical applications. s technologies are designed to help eliminate today s barriers to efficiency, speed and performance by simplifying networks and connecting people to the information they need, when they need it. does business in more than 150 countries around the world. For more information, visit on the Web at For the latest news, visit For more information, contact your representative, or call NORTEL or from anywhere in North America., the logo, Business Made Simple and the Globemark are trademarks of Networks. All other trademarks are the property of their owners. Copyright 2009 Networks. All rights reserved. Information in this document is subject to change without notice. assumes no responsibility for any errors that may appear in this document. NN BUSINESS MADE SIMPLE