G00249268 Technology Overview for Ethernet Switching Fabric Published: 16 May 2013 Analyst(s): Caio Misticone, Evan Zeng The term "fabric" has been used in the networking industry for a few years, but confusion remains among networking professionals. This research goes beyond the "marketecture" to define data center Ethernet fabric, including the main use cases and alternative technologies. Key Findings Ethernet switching fabric is the next logical evolution from the traditional Ethernet network for very large service providers, carriers, large enterprises and highly virtualized data centers where virtual machine () mobility, on demand network services and cloud computing are high priorities. Gartner has seen customers adopting virtual chassis switch/clustering technologies to build fabric-like networks at a lower cost than full fabric implementations. Next, they may consider skipping current fabric approaches and move directly to software-defined networking (SDN). Server virtualization, mobility and cloud computing are driving the move toward a fabricbased network solution. Implementation of current data center switching fabric solutions will likely lead to vendor lockin, as most existing implementations are based on proprietary versions of standard protocols, such as TRILL and SPB. Recommendations Confirm that your vendor offers a migration path to a fabric-based architecture, if required. Evaluate short-term alternatives (such as virtual chassis switch/clustering solutions), if moving from a traditional LAN to a network fabric-based offering is not a required option. SDN and hybrid SDN implementations using overlay technologies, such as Virtual extensible LAN (VXLAN) and Network Virtualization using Generic Routing Encapsulation (NVGRE), should be considered as alternative future solutions to improve business agility. Expect to deploy a single-vendor data center switching fabric solution.
What You Need to Know Evolution of the Data Center Infrastructure The data center has undergone several major architectural changes in the past few decades, evolving from single monolithic compute and storage resource and distributed client/server architecture to cloud-based computing. In this decade, it is undergoing another big architectural transition, from a simple server virtualization environment to a highly virtualized data center. "Eight Key Impacts on Your Data Center LAN Network" outlines all the key trends in the data center LAN network; here, we look at the four primary drivers of the evolution to Ethernet switching fabric: Resource pooling and fabric-based infrastructure: As compute evolves into virtual resource pools, it is time for network and storage to follow. Only after that will end users be able to choose pieces from different resource pools to form their choice of self-built and on-demand infrastructure. See "Clearing the Confusion about Fabric-Based Infrastructure: A Taxonomy" for our definition of fabric-based infrastructure. Cloud computing: Compute on-demand isn't very useful unless the associated storage and network are also on-demand. This makes end users embrace compute, storage and network resources, and build on-demand capabilities. East-west traffic: East-west traffic is dominant in the data center network, due to a number of contributing factors: application deployment model changes- breaking down monolithic client/ server applications into tiered and modular Web-based service-oriented architecture (SOA) composite applications, big data, and network/san convergence. mobility: s are evolving from a static resource residing only on a fixed physical host to a dynamic resource that moves from server to server in near real time. This makes all associated resources, such as storage and network, follow the mobility requirements and be -policyaware to automate associated policies on the fly with s. Limitations of Current Data Center Network The main limitations of the current data center Network are: Traditional three-tier network architecture is optimized for north-south traffic, but not for the increasingly predominant east-west traffic in enterprise data centers. With the creation of a virtual switch layer for access, physical aggregation layer switches become less necessary and largely counterproductive. Spanning Tree Protocol (STP) is widely implemented at data centers to prevent potential network loops. It can block 50% or more of switch ports in a redundant data path; therefore, it makes very inefficient use of network infrastructure. Slow and frequent convergence results in degraded performance. Once any link gets removed/ added to the network or any network topology changes, STP halts all traffic through the network and must recalculate the single path among all switches in the network before flowing traffic again. This process generally takes tens of seconds to minutes on all switch links, based on various Page 2 of 10 Gartner, Inc. G00249268
network sizes. It is unacceptable for mission-critical workloads to lose connectivity in the data path for a few seconds. Its successors' Rapid Spanning Tree Protocol (RSTP) or Multiple Spanning Tree Protocol (MSTP) can be used to reconverge and recover in a more timely manner, but will still compromise data center efficiency and performance. Each switch has its own control and management planes, without shared common configuration and policy information between switches. This is very complex in terms of operation and maintenance. Once a migrates across physical servers, the policy on a specific switch port does not migrate on the fly, together with its, to the destination switch port. It always requires manual configuration. This approach is not agile and tolerant in today's network. Analysis The increasing adoption of virtualization, cloud computing and server/storage convergence is requiring a rearchitecture of the data center environment. It is also introducing new technological, operational and organizational challenges, while adding new requirements to the existing data center network, such as scalability, simplification and flexibility. To support such a data center vision, the whole infrastructure needs to evolve, especially the network. Because Ethernet is the most widely accepted network standard in the data center, it needs to evolve from interconnecting physical switches to one Ethernet fabric that is -aware and behaves just like one integrated switch to support data center transformation. Ethernet fabric can be one- or two-tier Ethernet architectures in the data center (see Figure 1). The differences between those two forms are one- or two-hop forwarding across the fabric, and how edge ports connect together into one switch. In general, Ethernet fabric is composed of hardware and software components to make multiple interconnecting switches behave like a single switch. Gartner, Inc. G00249268 Page 3 of 10
Figure 1. Ethernet Fabric Architecture Management Tier Ethernet Fabric Core Switches Edge Switches Server Server Storage Virtual Storage Virtual Storage Source: Gartner (May 2013) Evolving the data center network into a network fabric requires changes in current network design practices moving from the conventional three-tier architecture to a one- or two-tier architecture, as well as making use of new technologies such as TRILL and SPB, and possibly new types of switches with larger Media Access Control (MAC) address tables. The technology and product evolution will be significant, and will not, in all cases, be backward-compatible, which will require a planned road map to ensure a smooth evolution or an eventual "rip and replace" of the current infrastructure. There are vendors that may allow some components of their hierarchical design to be reused for an Ethernet fabric architecture (for example, with a software upgrade). Enterprises should consider the migration paths for any upgrade in the data center, because there are several major vendor solutions that may require a rip and replace, even if the vendor is the incumbent. Ethernet fabric should also offer standard base interfaces and protocols to interoperate with networks running on legacy protocols such as STP. Technology Description Historically, the term "switching fabric" was used to describe the backplane and switching capacity of a single LAN switch. In the data center, fabric meant Fibre Channel or InfiniBand. Today, the term is generally used by vendors to describe the whole set of interconnected switches and their Page 4 of 10 Gartner, Inc. G00249268
interconnections with the implementation of Layer 2 multipathing technologies based on TRILL, SPB and other proprietary versions. As shown in Figure 1, Ethernet fabric is a logical entity composed of a number of hardware or virtual switches, various software components, and a single management platform. It is not only designed to interconnect physical servers and storage, but also s and virtual storage resources. Ethernet fabric should meet the following requirements: Control plane: The fabric can have either a centralized or distributed control plane in every switch node. An intelligent forwarding protocol is used that addresses the limitations of STP, including inefficient use of bandwidth and slow convergence. Also, it can support an integrated Layer 2/Layer 3 forwarding across all nodes in the fabric. Data plane: The fabric data path supports equal cost multipath forwarding at Layer 2 and data always takes the shortest path using multiple Inter-Switch Link (ISL) connections without loops. Also, all the server-facing ports in edge switches focus on user profile discovery and policy automation, and there should be a single shared-state table across the fabric for these ports. For the network-facing ports, the focus is on bandwidth aggregation, and the ports are transparent to user profile and policy. Redundancy: The fabric supports fast convergence if any network topology changes or links go up or down. Automation: The fabric should require a very simple setup or zero-touch deployment, and is able to automate network configuration and policy on a large scale. Management: The fabric supports a single point of management. Flat architecture: The fabric supports one- or two-tier architecture with deterministic latency from any port to any port within the fabric. Ethernet fabric may or may not meet the following requirements, depending on your business and technical requirements: Low latency: High-frequency trading (HFT), cluster/computing modeling or deep analytics applications require the lowest possible latency. Lossless: Convergence of Fibre Channel and Ethernet over a single network requires a guaranteed delivery of FCoE frames using data center bridging (DCB) to ensure a lossless operation. Interoperability: The fabric should provide a standard base interface to interoperate with thirdparty devices and networking equipment. awareness: The fabric is a -aware network and supports seamless mobility across physical servers with automated network policy and services on the fly, together with the moving s. Gartner, Inc. G00249268 Page 5 of 10
Technology Definition Ethernet fabric is an Ethernet construct in which multiple physical switches interconnect with each other and combine with hardware and software components to form a more elastic, automated and simpler switch network that behaves like a single switch. Ethernet fabric is not a switch cluster technology, and runs integrated Layer 2/Layer 3 standard base or proprietary protocols across all its switching fabric member nodes. Besides the classic Ethernet functions, Ethernet fabric has the following new characteristics: Support equal-cost multipath forwarding at Layer 2/Layer 3 with fine-grained, flow-based load balancing. Replace STP with Layer 2 Multipathing protocols that can be either standard base or proprietary. These protocols do not have the same limitations as STP and can support Layer 2 Equal Cost Multipath (ECMP) and faster convergence once the topology changes or any switch link is up or down. The whole Ethernet fabric collapses the legacy three-tier network into a one- or two-tier architecture across the full data center and with deterministic latency from any port to any port. All the server-facing switch ports can be either fully -aware or transparent with a single shared, scalable-state table for all ports. All devices and ports in the fabric can be accessed and managed by a single point. Ethernet fabric can be integrated with a provisioning and management system to provide multitenancy and policy automation for s, which can lead to a worse vendor lock-in scenario. Standards There are some standards and vendor-specific technologies that are usually involved and should be considered in a data center Ethernet switching fabric ecosystem. Some of them include: Layer 2 Multipathing: TRILL, SPB, Cisco FabricPath, Juniper Networks QFabric System, Brocade VCS Multichassis Link Aggregation (MC-LAG): Cisco Virtual Switching System (VSS)/vPC, HP Intelligent Resilient Framework (IRF), Juniper Virtual Chassis Storage/Network Convergence Protocols: FCoE Lossless Ethernet for Converged Infrastructure: DCB Uses In reviewing Gartner client inquiries, we have identified three broad use cases for Ethernet switching fabrics: The service provider's data center, where network automation, scalability, performance and manageability are critical Page 6 of 10 Gartner, Inc. G00249268
The enterprise's data center, with a highly virtualized and cloud-oriented environment that requires deterministic end-to-end latency, network automation and on-demand services Any sizable data center that wants to enhance network automation, requires on-demand network services, and gets rid of STP and facilitates network provisioning Benefits and Risks Benefits: Fabrics can simplify network operations by moving from a physical three-tier architecture to one or two tiers, providing a single point of management with high automation and treating the network as one logical switch. Fabrics provide deterministic latency for east-west traffic: In a traditional three-tier network, latency increases as traffic flows through each individual hop. This latency may not impact the movement of traditional best-effort applications, but it certainly impacts latency-sensitive applications (such as workload mobility and enterprise storage). Fabric provides multipath technology that increases the bandwidth by allowing all paths to a certain destination to be active at the same time, improving overall performance and reliability while eliminating downtime due to slow reconvergence times. Automation port profile migration for mobility makes Ethernet fabric a very convenient solution to operate in highly virtualized and cloud-oriented data centers, where agility is crucial. Risks: Because every fabric vendor solution has some sort of proprietary feature, vendor lock-in is a real possibility in the data center switching fabric market. Selecting a single vendor's proprietary fabric solution is similar to the way most organizations standardize on a single core data center network vendor today. Implementing a switching fabric can make your data center more complex, if your vendor doesn't offer you a clear migration path from a traditional data center network. This is because the Ethernet fabric runs over specific proprietary protocols and has a clear boundary with traditional data center networks. Technology Alternatives Gartner sees two main technology alternatives to fabric-based solutions: Clustering Solutions: This technology has been used by many enterprises that want to achieve resiliency, redundancy and management simplification without necessarily moving toward the deployment of a fabric solution. Most vendors have their own proprietary versions of clustering or MC-LAG solutions (for example, Cisco VSS/vPC, HP IRF and Juniper Virtual Chassis). Gartner, Inc. G00249268 Page 7 of 10
SDN: Gartner expects emerging technologies such as SDN to become real and mature alternatives to data-center-switch-fabric-based solutions in two to three years. As defined in "Ending the Confusion About Software-Defined Networking: A Taxonomy," SDN is considered a new way to design, build and operate networks. It decouples the data and control plane from every network element and moves the control plane to an SDN controller. This controller may be deployed as a cluster for high availability and scalability, and, over time, east-west APIs will emerge that enable multicontroller federation. One of the deployment models of SDN is one in which the controller communicates with each network element via the OpenFlow protocol. By using overlay technologies (such as VXLAN and NVGRE), network administrators can simplify the mobility of s by running on top of any Layer 3 network topology. With SDN, network intelligence and state are logically centralized, allowing the network to be abstracted from the applications, and the configuration of the whole network can be done from one place, instead of from each individual device. SDN promises the ability to leverage low-cost and commoditized hardware and will be a key element of real-time infrastructure (RTI), which allows the network to adapt itself according to application needs on-demand. Selection Guidelines When selecting among the different data center Ethernet switching fabric solutions, it's very important to compare the various vendors' architectures and features, and align them with your specific use cases: Vendors' architectures: One- or two-tier architecture: Either fully or partially meshed solutions, these are the two most common physical architectures. Select a vendor based on needs such as scalability, oversubscription rate, latency and automation, instead of the actual number of network tiers. Prefer vendors that allow scaling of the switching fabric with minimum intervention, while keeping the desired oversubscription rate and latency across the switching fabric. From an operational perspective, some vendors have a solid solution for defining and managing the whole fabric as a single entity one that is self-learning and adaptive. Other vendors have a box-by-box management approach and still require configuration changes when fabric topology changes. All vendors somehow still require some sort of manual command line interface (CLI) configuration; however, for operational simplification, select a vendor that will help diminish this problem via the use of intelligent and self-learning techniques. Features and use cases: DCB, FCoE, awareness and extreme low latency are all features that should be selected based on use cases and business needs. For example, if your environment is highly virtualized and cloud-oriented, then an architecture that provides an ability to be elastic, scalable and virtualization-aware would prevail over an architecture where every nanosecond can make a difference with latency as the prime factor. For example, if you run a highly virtualized enterprise data center, then live migration is an important function of data center capacity management and for operations efficiency. In this case, the amount of manual configuration of network policy to support the moving is a good business case for saving operational labor costs. Page 8 of 10 Gartner, Inc. G00249268
Interoperability: Ensure that your vendor also offers standard base interfaces and protocols to interoperate with networks running on legacy protocols like STP. Technology Providers Networking vendors have different branded names for their data center switching fabric solutions. The main ones, listed in alphabetical order, are: Alcatel-Lucent: Alcatel-Lucent Mesh Arista: No specific name Avaya: Avaya Virtual Enterprise Network Architecture (VENA) Fabric Connect Brocade: VCS Fabric Cisco: Unified Fabric Dell: No specific name Enterasys: OneFabric Extreme Networks: Open Fabric Huawei: Cloud Fabric HP: FlexFabric IBM: No specific name Juniper: QFabric System Recommended Reading Some documents may not be available as part of your current Gartner subscription. "Magic Quadrant for Data Center Network Infrastructure" "Eight Key Impacts on Your Data Center LAN Network" "Ending the Confusion About Software-Defined Networking: A Taxonomy" "Clearing the Confusion About Fabric-Based Infrastructure: A Taxonomy" Gartner, Inc. G00249268 Page 9 of 10
GARTNER HEADQUARTERS Corporate Headquarters 56 Top Gallant Road Stamford, CT 06902-7700 USA +1 203 964 0096 Regional Headquarters AUSTRALIA BRAZIL JAPAN UNITED KINGDOM For a complete list of worldwide locations, visit http://www.gartner.com/technology/about.jsp 2013 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This publication may not be reproduced or distributed in any form without Gartner s prior written permission. If you are authorized to access this publication, your use of it is subject to the Usage Guidelines for Gartner Services posted on gartner.com. The information contained in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. This publication consists of the opinions of Gartner s research organization and should not be construed as statements of fact. The opinions expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company, and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartner s Board of Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartner research, see Guiding Principles on Independence and Objectivity. Page 10 of 10 Gartner, Inc. G00249268