WHITE PAPER www.brocade.com Data Center Ethernet Fabrics: An Architecture for Cloud Networking As data centers evolve to a world where information and applications can move anywhere in the cloud, classic Ethernet with its hierarchical architecture also has to evolve to a new data center Ethernet fabric.
Data center networks rely on Ethernet. Over the decades, Ethernet has evolved as new application architectures emerged. Today, data center networks carry traffic for a diverse set of applications including client/server, Web services, and unified communications each with different traffic patterns and network service requirements. Applications are increasingly deployed within virtual machines hosted on server clusters. And Ethernet is used to build shared storage pools, which places stringent demands on the network, demands for lossless packet delivery, deterministic latency, and high bandwidth. All together, these changes are the forces behind the next evolutionary step in Ethernet: the Ethernet fabric. The Classic Ethernet Network Since data centers need more ports than are available in a single Ethernet switch, multiple switches are connected to form a network with increased connectivity. For example, server racks often include a switch at the Top of the Rack (ToR) or servers in several racks connect to a Middle of Row (MoR) or the End of Row (EoR) switch. All these Ethernet switches are connected, forming a hierarchical, or Ethernet tree topology, as shown in Figure 1. Limitations of Classic Ethernet In a classic Ethernet network, the connections between switches, or Inter-Switch Links (ISLs, shown as solid blue lines in Figure 1), are not allowed to form a loop or frames aren t delivered. Spanning Tree Protocol (STP) prevents loops by creating a tree topology with only one active path between any two switches. (In Figure 1, inactive paths are shown as dotted lines.) This means that ISL bandwidth is limited to a single connection, since multiple paths between switches are prohibited. The tree topology requires traffic to move up and down the tree, or north-south, to get to an adjacent rack. When most of the access traffic is between servers in a rack, this is not a problem. But server clusters, such as those required for server virtualization, have traffic between servers in multiple racks, travelling east-west, so the tree topology increases latency with multiple hops and restricts bandwidth with single links between switches. 2
Core Access Aggregation ISLs Inactive links Traffic flow Figure 1. Classic Ethernet network. Server rack STP automatically recovers when a link is lost. However, it halts all traffic through the network and must reconfigure the single path between all switches in the network before allowing traffic to flow again. Halting all traffic for tens of seconds up to minutes on all links limits scalability and constrains traffic to applications that can tolerate data path blocking to achieve link resiliency. In the past, traffic relied on TCP to handle this interruption in service, but today, with almost all data center applications running in a 24 x 7 high availability mode and storage traffic growing on the Ethernet network, loss of connectivity in the data path for even a few seconds is unacceptable. Finally, the classic Ethernet switch architecture presents other limitations. Each switch has its own control and management planes. Each switch has to discover and process the protocol of each frame as it arrives on an ingress port. As more switches are added, protocol processing time increases adding latency. Each switch and each port in the switch has to be configured individually, since there is no sharing of common configuration and policy information between switches. Complexity increases, configuration mistakes increase, and the total cost of operations and management does not scale well. Enhancements to Ethernet An additional Ethernet standard, Link Aggregation Groups (LAGs), was defined so that multiple links between switches were treated as a single connection without forming loops. But a LAG must be manually configured on each port in the LAG and is not very flexible. Flatter networks with self-aggregating ISL connections eliminate manual configuration. Spanning tree has been modified to support multiple logical trees in a single physical tree (MSTP) and to reduce convergence time when links are added or removed (RSTP). But the topology is still a hierarchical tree in which convergence is measured in seconds. Switch stacking was introduced so that multiple switches could be managed as one logical switch. But topologies are restricted, master switches and special ISL ports are required, and ISL oversubscription increases as the stack grows. Stacking does provide a single point of management for multiple switches, but adds complexity, restricts topology choices, and limits scalability. Properties of Classic Ethernet Networks Control plane on the switch, policy configuration and protocol processing required for every port. Data path interconnect relies on STP to prevent loops. Link aggregation on interconnects is manually configured on multiple switches. Management is at the switch level, limiting scalability. Topology is restricted to a hierarchical tree. Scalability is limited by single-path interconnects Link resiliency is automatic but disruptive to all traffic flows in the network. Adding network services requires careful placement to avoid Layer 2 bottlenecks. 3
The Ethernet Fabric Architecture Figure 2 shows the architecture of a classic Ethernet switch. The control, data, and management planes are logically connected to every port via a back plane. Control and management planes operate at the switch level not a network level. Figure 2. Ethernet switch architecture. Ethernet switches Control plane Management plane Data plane Ethernet fabrics can be thought of as extending the control and management planes beyond the physical switch into the fabric. As shown in Figure 3, they now operate at a fabric level rather than at a switch level. Control plane Figure 2. Ethernet fabric architecture. Self-aggregating, scalable, flat Ethernet fabric Management plane Data plane Switch Primary Functions: Planes Switches are designed using three primary functions, or planes, called the control, management, and data planes, ach providing specific services. The control plane is responsible for setting up the data plane forwarding tables that record the optimal path over which to send a frame. The control plane also decides if an incoming frame should be dropped based on security settings and what QoS level to assign to the frame based on network policies. The data plane is responsible for moving frames between the ingress and egress ports as quickly as possible. The management plane is responsible for managing the switch including switch and port configuration, monitoring and sending alerts to the management console. Control and management become scalable, that is, distributed services integrated into the network rather than restricted by the switch chassis, which provides a number of benefits. For example, a fabric scales automatically when another fabric-enabled switch is added. The new switch logically plugs into the distributed control plane, allowing control plane state and configuration parameters to be shared across all switches and ports in the fabric. Since the control plane is distributed, protocol packet processing is done once at the edge port, reducing overall packet latency; and policy and security settings are created once, ensuring consistent policy and security configuration across all switches in the fabric. When a switch is added, it automatically joins a logical chassis, similar to adding a port card to a chassis switch. This simplifies management, monitoring, and operations since policy and security configuration parameters are automatically inherited by the new switch. Configuration and management are performed once for the fabric, not multiple times for each switch and switch port. In addition, information about device connections to servers and storage is known to all switches in the fabric, enabling Automated Migration of Port Profiles (AMPP) within the fabric. When virtual machines or servers are moved to another fabric port, AMPP ensures that all network policies and security settings continue to be applied to its traffic without reconfiguring the entire network. The control path replaces STP with link state routing, while the data path provides equal-cost multipath forwarding at Layer 2 so data always takes the shortest path using multiple ISL connections without loops. Combined with the distributed control plane, scaling bandwidth is made simple. For example, when a new switch connects to any other switch in the fabric, ISLs come up automatically. When multiple ISLs connect two switches, trunks automatically 4
form with frame-based load balancing and automatic link failover. If a trunk link fails or is removed, traffic is rebalanced on the existing links non-disruptively. Finally, if an ISL is added or removed anywhere in the fabric, traffic on other ISLs continues to flow instead of halting as with STP. In summary, the Ethernet fabric architecture requires: A control plane distributed across all switches so that packet processing is performed once and common attributes (policies and security settings) are configured once ISLs that support any topology (ring, mesh, or core/edge) Equal-cost multipath routing at Layer 2 replacing STP on ISLs, removing loops, automatically scaling bandwidth, and ensuring shortest path is always used in the fabric Flatter network so that traffic flows scale out, have low latency, and are not congested Automatic link resiliency that is non-disruptive to traffic flowing on any other link Automatic link aggregation, with trunks delivering line-rate load balancing across all links and automatic link failover within a trunk A logical management architecture that is at a fabric level so that adding a new switch does not require configuration of parameters as they are inherited and switches are managed like port cards in a chassis Fabric support for traffic redirection to simplify deployment of network services such as security, Layer 4 7 application control, and native Fibre Channel services Ethernet Fabrics Compared to classic hierarchical Ethernet architectures, Ethernet fabrics provide higher levels of performance, utilization, availability and simplicity. They have the following characteristics: Flatter. Ethernet fabrics are selfaggregating, enabling a flatter network. Intelligent. Switches in the fabric know about each other and all connected devices. Scalable. All paths are available for high performance and high reliability. Efficient. Traffic automatically travels along the shortest path. Simple. The fabric is managed as a single logical entity. Brocade Virtual Cluster Switching Architecture Brocade is the first vendor to deliver products that meet the all the requirements of a true Ethernet fabric. Brocade Virtual Cluster Switching (VCS ) architecture redesigns traditional Ethernet networks, removing the limitations of classic Ethernet. The Brocade VCS architecture is ideally suited for both private and public cloud computing deployments. Figure 4 shows the key capabilities of the Brocade VCS capabilities that are responsible for implementing an Ethernet fabric: VCS Ethernet fabric in the data plane VCS Distributed Intelligence in the control plane VCS Logical Chassis in the management plane Ethernet Fabric Brocade VCS Technology Distributed Intelligence Logical Chassis Figure 2. Brocade VCS architecture. No STP Multi-path, deterministic Auto-healing, non-disruptive Lossless, low latency Convergence ready Self-forming Arbitrary topology Network aware of all members, devices, VMs Masterless control, no reconfiguration VAL interaction Logically flattens and collapses network layers Scale edge and manage as if single switch Auto-configuration Centralized or distributed management, end-to-end Dynamic Services Connectivity over distance, Native Fibre Channel, Security Services, Layer 4-7, and so on 5
In addition, Brocade VCS Dynamic Services support simple scaling of network services such as security, Layer 4 7 application delivery control, extended Layer 2 networks between data centers, and native Fibre Channel storage services for converged networks. Combined, these VCS capabilities transform classic Ethernet networks into Ethernet fabrics. Brocade VCS Implementation of an Ethernet Fabric Figure 5 shows a logical model of how VCS implements an Ethernet fabric. Figure 5. Brocade VCS control and management plane architecture. Core Distributed Intelligence Logical Chassis VCS Distributed Intelligence Instead of independent control planes for each switch, Brocade VCS Distributed Intelligence service uses a distributed control plane so that protocol processing occurs once at an edge port and common configuration parameters are set once and available to every switch in the fabric. For example, when a new switch is added to the fabric, it logically plugs into the distributed control plane, adding additional control path processing for scalability. At the fabric edge, packet processing occurs on any port of any switch, but is not required on every ISL port, greatly reducing latency and making configuration simple and scalable. Policies can be configured once and are available to all switch ports. With Automated Migration of Port Profiles (AMPP), virtual servers can migrate between switch ports without risking conflicts with network policies and security settings at edge ports; these settings automatically follow the MAC address of the virtual machine. VCS Logical Chassis Instead of managing each switch and its ports independently, the Brocade VCS Logical Chassis service creates a virtual management plane. A Logical Chassis has common configuration parameters for policies and can bind MAC address to policies with AMPP and apply them to all switches in the fabric. The Logical Chassis service creates a single point of management for monitoring all switches, ports and traffic in the fabric. Adding a new switch is as simple as adding a new port blade to a chassis switch. The switch and its ports are uniquely identified, share common configuration parameters, and are monitored from a single management connection. With the Logical Chassis service, management can scale with the network, since as adding switches and ports does not add complexity. VCS Ethernet Fabric The Brocade VCS Ethernet fabric provides equal-cost multipath forwarding, simplifies interconnect configuration, provides automatic link failover with no interruption to traffic on unaffected links, and provides plug-and-play fabric scalability. Instead of manually configuring LAG on individual ports on multiple switches, Brocade VCS Ethernet fabric lets the data 6
plane automatically form trunks when multiple ISL connections are added between switches. Simply adding another cable increases Layer 2 bandwidth, providing linear scalability of server to server and server to storage traffic. ISL connections between switches form automatically, as long as the fabric identifier of the switch is set to the identifier of the fabric to which it s connected. The data plane topology is user configurable; it is not dictated by the underlying technology nor the limitations inherent in STP and hierarchical Ethernet tree topologies. The flexibility of this topology allows easy specification of oversubscription ratios. For example, High Performance Computing (HPC) workloads may require 1:1 subscription, virtual servers 4:1, and client/server 10:1 or higher. The Brocade VDX 6720 Family of Data Center Switches The introduction of the Brocade VDX 6720 family of 10 GbE-capable fabric switches delivers a new category of Ethernet switching, the Ethernet fabric switch. The Brocade VDX 6720 includes Brocade VCS architecture, offering revolutionary Ethernet fabrics, Distributed Intelligence, and Logical Chassis technology for the data center. The Brocade VDX 6720 is available in 24- or 60-port rack-mount form factors and can be deployed in 16-, 24-, 40-, 50-, or 60-port configurations with Brocade exclusive Ports-on- Demand (POD) licensing. At launch, the Brocade VDX family interoperates with existing Ethernet switching products, providing an evolutionary path from legacy Ethernet networks to Ethernet fabrics. Existing legacy switches do not have to be replaced until the end of their useful life; Brocade VDX 6720 switches can be added at the top of rack or middle or end of row forming a scalable, simple-to-manage Ethernet fabric. Figure 7. Brocade VDX 6720-24 Data Center Switch (top) and Brocade VDX6720-60 Data Center Switch (bottom). For example, several Brocade VDX 6720 switches deployed in a ToR configuration create a single Logical Chassis with a single distributed control plane across multiple racks of servers, delivering compelling reductions in capital and operating costs, while simplifying virtual machine migration. Automated Migration of Port Profiles (AMPP) is hypervisor agnostic, which is an important feature, as most data centers deploy different hypervisor stacks based on application and server requirements. And, since the Brocade VDX 6720 family supports the emerging Ethernet Data Center Bridging (DCB), TRILL, and Fiber Channel over Ethernet (FCoE) standards, the Ethernet fabric is lossless, low latency, and convergence ready. Learn More Brocade networking solutions help the world s leading organizations transition smoothly to a virtualized world where applications and information reside anywhere. This approach is based on the Brocade One unified network strategy, which enables a wide range of consolidation, convergence, virtualization, and cloud computing initiatives Learn more at www.brocade.com. 7
WHITE PAPER www.brocade.com Corporate Headquarters San Jose, CA USA T: +1-408-333-8000 info@brocade.com European Headquarters Geneva, Switzerland T: +41-22-799-56-40 emea-info@brocade.com Asia Pacific Headquarters Singapore T: +65-6538-4700 apac-info@brocade.com 2010 Brocade Communications Systems, Inc. All Rights Reserved. 11/10 GA-WP-1551-00 Brocade, the B-wing symbol, BigIron, DCFM, DCX, Fabric OS, FastIron, IronView, NetIron, SAN Health, ServerIron, TurboIron, and Wingspan are registered trademarks, and Brocade Assurance, Brocade NET Health, Brocade One, Extraordinary Networks, MyBrocade, VCS, and VDX are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned are or may be trademarks or service marks of their respective owners. Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.