Cisco Nexus 9000 Data Center Service Provider

Transcription

1 Guide Cisco Nexus 9000 Data Center Service Provider Guide March Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 80

2 Contents 1. Introduction What You Will Learn Disclaimer Why Choose Cisco Nexus 9000 Series Switches About Cisco Nexus 9000 Series Switches ACI-Readiness What Is ACI? Converting Nexus 9000 NX-OS Mode to ACI Mode Evolution of Data Center Design Evolution of Data Center Operation Key Features of Cisco NX-OS Software Data Center Design Considerations Traditional Data Center Design Leaf-Spine Architecture Spanning Tree Support Layer 2 versus Layer 3 Implications Virtual Port Channels (vpc) Overlays Virtual Extensible LAN (VXLAN) BGP EVPN Control Plane for VXLAN VXLAN Data Center Interconnect (DCI) with a BGP Control Plane Integration into Existing Networks Pod Design with vpc Fabric Extender Support Pod Design with VXLAN Traditional Three-Tier Architecture with 1/10 Gigabit Ethernet Server Access Traditional Cisco Unified Computing System and Blade Server Access Integrating Layer 4 - Layer 7 Services Cisco Nexus 9000 Data Center Topology Design and Configuration Hardware and Software Specifications Leaf-Spine-Based Data Center Topology Traditional Data Center Topology Managing the Fabric Adding Switches and Power-On Auto Provisioning Software Upgrades Guest Shell Container Virtualization and Cloud Orchestration VM Tracker OpenStack Cisco UCS Director Cisco Prime Data Center Network Manager Cisco Prime Services Catalog Automation and Programmability Support for Traditional Network Capabilities Programming Cisco Nexus 9000 Switches through NX-APIs Chef, Puppet, and Python Integration Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 80

3 12.4 Extensible Messaging and Presence Protocol Support OpenDay Light Integration and OpenFlow Support Troubleshooting Appendix Products Cisco Nexus 9500 Product Line Cisco Nexus 9300 Product Line NX-API About NX-API Using NX-API NX-API Sandbox NX-OS Configuration Using Postman Configuration Configuration of Interfaces and VLAN Configuration of Routing - EIGRP BGP Configuration for DCI vpc Configuration at the Access Layer Multicast and VXLAN Firewall ASA Configuration F5 LTM Load Balancer Configuration References Design Guides Nexus 9000 platform Network General Migration Analyst Reports Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 80

4 1. Introduction 1.1 What You Will Learn The Cisco Nexus 9000 Series Switch product family makes the next generation of data center switching accessible to customers of any size. This white paper is intended for the commercial customer new to the Cisco Nexus 9000 who is curious how the Cisco Nexus 9000 might be feasibly deployed in its data center. This white paper will highlight the benefits of the Cisco Nexus 9000, outline several designs suitable for small-tomidsize customer deployments, discuss integration into existing networks, and walk through a Cisco validated Nexus 9000 topology, complete with configuration examples. The featured designs are practical for both entry-level insertion of Cisco Nexus 9000 switches, and for existing or growing organizations scaling out the data center. The configuration examples transform the logical designs into tangible, easy-to-use templates, simplifying deployment and operations for organizations with a small or growing IT staff. Optionally, readers can learn how to get started with the many powerful programmability features of Cisco NX-OS from a beginner s perspective by referencing the addendum at the end of this document. This white paper also provides many valuable links for further reading on protocols, solutions, and designs discussed. A thorough discussion on the programmability features of the Cisco Nexus 9000 is out of the scope of this document. 1.2 Disclaimer Always refer to the Cisco website for the most recent information on software versions, supported configuration maximums, and device specifications at Why Choose Cisco Nexus 9000 Series Switches Cisco Nexus 9000 Series Switches (Figure 1) are ideal for small-to-medium-sized data centers, offering five key benefits: price, performance, port-density, programmability, and power efficiency. Figure 1. Cisco Nexus 9000 Series Switch Product Family Cisco Nexus 9000 Series Switches are cost-effective by taking a merchant-plus approach to switch design. Both Cisco developed, plus merchant silicon application-specific integrated circuits (ASICs; Trident II, sometimes abbreviated as T2) power Cisco Nexus 9000 switches. The T2 ASICs also deliver power efficiency gains. Additionally, Cisco Nexus 9000 Series Switches lead the industry in 10 GE and 40 GE price-per-port densities. The cost-effective design approach, coupled with a rich feature set, make the Cisco Nexus 9000 a great fit for the commercial data center Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 80

5 Licensing is greatly simplified on the Cisco Nexus At the time of writing, there are two licenses available: the Enterprise Services Package license can enable dynamic routing protocol and Virtual Extensible LAN (VXLAN) support, and the Data Center Network Manager (DCNM) license provides a single-pane-of-glass GUI management tool for the entire data center network. Future licenses may become available as new features are introduced. Lastly, the Cisco Nexus 9000 offers powerful programmability features to drive emerging networking models, including automation and DevOps, taking full advantage of tools like the NX-API, Python, Chef, and Puppet. For the small-to-midsize commercial customer, the Cisco Nexus 9000 Series Switch is the best platform for 1-to-10 GE migration or 10-to-40 GE migration, and is an ideal replacement for aging Cisco Catalyst switches in the data center. The Cisco Nexus 9000 can easily be integrated with existing networks. This white paper will introduce you to a design as small as two Cisco Nexus 9000 Series Switches, and provide a path to scale out as your data center grows, highlighting both access/aggregation designs, and spine/leaf designs. 1.4 About Cisco Nexus 9000 Series Switches The Cisco Nexus 9000 Series consists of larger Cisco Nexus 9500 Series modular switches and smaller Cisco Nexus 9300 Series fixed-configuration switches. The product offerings will be discussed in detail later in this white paper. Cisco provides two modes of operation for the Cisco Nexus 9000 Series. Customers can use Cisco NX-OS Software to deploy the Cisco Nexus 9000 Series in standard Cisco Nexus switch environments. Alternately, customers can use the hardware-ready Cisco Application Centric Infrastructure (ACI) to take full advantage of an automated, policy-based, systems management approach. In addition to traditional NX-OS features like virtual PortChannel (vpc), In-Service Software Upgrades (ISSU - future), Power-On Auto-Provisioning (POAP), and Cisco Nexus 2000 Series Fabric Extender support, the singleimage NX-OS running on the Cisco Nexus 9000 introduces several key new features: The intelligent Cisco NX-OS API (NX-API) provides administrators a way to manage the switch through remote procedure calls (JSON or XML) over HTTP/HTTPS, instead of accessing the NX-OS command line directly. Linux shell access can enable the switch to be configured through Linux shell scripts, helping automate the configuration of multiple switches and helping to ensure consistency among multiple switches. Continuous operation through cold and hot patching provides fixes between regular maintenance releases or between the final maintenance release and the end-of-maintenance release in a non-disruptive manner (for hot patches). VXLAN bridging and routing in hardware at full line rate facilitates and accelerates communication between virtual and physical servers. VXLAN is designed to provide the same Layer 2 Ethernet services as VLANs, but with greater flexibility and at a massive scale. For more information on upgrades, refer to the Cisco Nexus 9000 Series NX-OS Software Upgrade and Downgrade Guide. This white paper will focus on basic design, integration of features like vpc, VXLAN, access layer device connectivity, and Layer 4-7 service insertion. Refer to the addendum for an introduction to the advanced programmability features of NX-OS on Cisco Nexus 9000 Series Switches. Other features are outside the scope of this white paper Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 80

6 2. ACI-Readiness 2.1 What Is ACI? The future of networking with Cisco Application Centric Infrastructure (ACI) is about providing a network that is deployed, monitored, and managed in a fashion that supports rapid application change. ACI does so through the reduction of complexity and a common policy framework that can automate provisioning and management of resources. Cisco ACI works to solve the business problem of slow application deployment due to focus on primarily technical network provisioning and change management problems, by enabling rapid deployment of applications to meet changing business demands. ACI provides an integrated approach by providing application-centric, end-to-end visibility from a software overlay down to the physical switching infrastructure. At the same time it can accelerate and optimize Layer 4-7 service insertion to build a system that brings the language of applications to the network. ACI delivers automation, programmability, and centralized provisioning by allowing the network to be automated and configured based on business-level application requirements. ACI provides accelerated, cohesive deployment of applications across network and Layer 4-7 infrastructure and can enable visibility and management at the application level. Advanced telemetry for visibility into network health and simplified day-two operations also opens up troubleshooting to the application itself. ACI s diverse and open ecosystem is designed to plug into any upper-level management or orchestration system and attract a broad community of developers. Integration and automation of both Cisco and third-party Layer 4-7 virtual and physical service devices can enable a single tool to manage the entire application environment. With ACI mode customers can deploy the network based on application requirements in the form of policies, removing the need to translate to the complexity of current network constraints. In tandem, ACI helps ensure security and performance while maintaining complete visibility into application health on both virtual and physical resources. Figure 2 highlights how the network communication might be defined for a three-tier application from the ACI GUI interface. The network is defined in terms of the needs of the application by mapping out who is allowed to talk to whom, and what they are allowed to talk about by defining a set of policies, or contracts, inside an application profile, instead of configuring lines and lines of command-line interface (CLI) code on multiple switches, routers, and appliances Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 80

7 Figure 2. Sample Three-Tier Application Policy 2.2 Converting Nexus 9000 NX-OS Mode to ACI Mode This white paper will feature Cisco Nexus 9000 Series Switches in NX-OS (standalone) mode. However, Cisco Nexus 9000 hardware is ACI-ready. Cisco Nexus 9300 switches and many Cisco Nexus 9500 line cards can be converted to ACI mode. Cisco Nexus 9000 switches are the foundation of the ACI architecture, and provide the network fabric. A new operating system is used by Cisco Nexus 9000 switches running in ACI mode. The switches are then coupled with a centralized controller called the Application Policy Infrastructure Controller (APIC) and its open API. The APIC is the unifying point of automation, telemetry, and management for the ACI fabric, helping to enable an application policy model approach to the data center. Conversion from standalone NX-OS mode to ACI mode on the Cisco Nexus 9000 is outside the scope of this white paper. For more information on ACI mode on Cisco Nexus 9000 Series Switches, visit the Cisco ACI website. 3. Evolution of Data Center Design This section describes the key considerations that are driving data center design and how these are addressed by Cisco Nexus 9000 Series Switches. Flexible Workload Placement Virtual machines may be placed anywhere in the data center, without considerations of physical boundaries of racks. After initial placement, virtual machines may be moved for optimization, consolidation, or other reasons, which could include migrating to other data centers or to public cloud. The solution should provide mechanisms to allow such movements in a seamless manner to the virtual machines. The desired functionality is achieved using VXLAN and a distributed any-cast gateway Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 80

8 East-West Traffic within the Data Center Data center traffic patterns are changing. Today, more traffic moves east-west from server to server through the access layer, as servers need to talk to each other and consume services within the data center. This shift is primarily driven by consolidation of data centers, evolution of clustered applications such as Hadoop, virtual desktops and multi-tenancy. Traditional three-tier data center design is not optimal, as east-west traffic is often forced up through the core or aggregation layer, taking a suboptimal path. The requirements of east-west traffic are addressed by a two-tier flat data center design that takes full advantage of a Leaf-and-Spine architecture, achieving most specific routing at the first hop router at the access layer. Host routes are exchanged to help ensure most specific routing to and from servers and hosts. And virtual machine mobility is supported through detection of virtual machine attachment and signaling of a new location to the rest of the network so that routing to the virtual machine continues to be optimal. The Cisco Nexus 9000 can be used as an end-of-row or top-of-rack access layer switch, as an aggregation or core switch in a traditional, hierarchical two- or three-tier network design, or deployed in a modern Leaf-and-Spine architecture. This white paper will discuss both access/aggregation and Leaf/Spine designs. Multi-Tenancy and Segmentation of Layer 2 and 3 Traffic Large data centers, especially service provider data centers, need to host multiple customers and tenants who would have overlapping private IP address space. The data center network must allow co-existence of traffic from multiple customers on shared network infrastructure and provide isolation between those unless specific routing policies allow. Traffic segmentation is achieved by a) using VXLAN encapsulation, where VNI (VXLAN Network Identifier) acts as segment identifier, and b) through Virtual Route Forwarding (VRFs) to de-multiplex overlapping private IP address space. Eliminate or Reduce Layer 2 Flooding in the Data Center In the data center, unknown unicast traffic and broadcast traffic in Layer 2 is flooded, with Address Resolution Protocol (ARP) and IPv6 neighbor solicitation being the most significant part of broadcast traffic. While a VXLAN can enable distributed switching domains which allow virtual machines (VMs) to be placed anywhere within a data center, this comes at the cost of having to broadcast traffic across the distributed switching domains that are now spread across the data center. Therefore VXLAN implementation has to be complemented with a mechanism that would reduce the broadcast traffic. The desired functionality is achieved by distributing MAC reachability information through Border Gate Protocol Ethernet VPN (BGP EVPN) to optimize flooding relating to unknown Layer 2 unicast traffic. Optimization of reducing broadcasts associated with ARP and IPv6 neighbor solicitation is achieved by distributing the necessary information through BGP EVPN and caching it at the access switches. An address solicitation request can then locally respond without sending a broadcast. Multi-Site Data Center Design Most service provider data centers, as well as large enterprise data centers, are multi-site to support requirements such as geographical reach, disaster recovery, etc. Data center tenants require the ability to build their infrastructure across different sites as well as operate across private and public clouds Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 80

9 4. Evolution of Data Center Operation Data center operation has rapidly evolved in the last few years, driven by some of these needs: Virtualization - All popular virtualization platforms - ESXi, OpenStack, and HyperV - have built-in virtual networking. Physical networking elements have to interface and optimize across these virtual networking elements. Convergence of compute, network and storage elements - Users want to implement full provisioning use cases (that is, create tenants, create tenant networks, provision VMs, bind VMs to a tenant network, create vdisks, bind vdisks to a VM) through integrated orchestrators. In order for this to be achieved, all infrastructure elements need to provide APIs to allow orchestrators to provision and monitor the devices. DevOps - As scale of data centers has grown, data center management is increasingly through programmatic frameworks such as Puppet and Chef. These frameworks have a Master that controls the target devices through an Agent that runs locally on the target devices. Puppet/Chef allows users to define their intent through a manifest/recipe. A reusable set of configuration or management tasks - and allows the recipe to be deployed on numerous devices which are executed by the Agent. Dynamic application provisioning - Data center infrastructure is quickly transitioning from an environment that supports relatively static workloads confined to specific infrastructure silos to a highly dynamic cloud environment in which any workload can be provisioned anywhere and can scale on demand according to application needs. As the applications are provisioned, scaled, and migrated, their corresponding infrastructure requirements, IP addressing, VLANs, and policies need to be seamlessly enforced. Software Define Networking (SDN) - SDN separates the control plane from data plane within a network, allowing intelligence and the state of the network to be managed centrally while abstracting the complexity of the underlying physical network. The industry has standardized around protocols such as OpenFlow. SDN would allow users to adapt the network dynamically to application needs for applications such as Hadoop, Video Delivery, and others. The Cisco Nexus 9000 has been designed to address the operational needs of evolving data centers. Refer to Section 6 in this document for a detailed discussion on the features. Programmability and Automation NX-APIs: Intelligent Cisco NX-OS API (NX-API) provides administrators a way to manage the switch through remote procedure calls (JSON or XML) over HTTP/HTTPS, instead of accessing the NX-OS command line directly. Integration with Puppet and Chef: Cisco Nexus 9000 switches provide agents a way to integrate with Puppet and Chef, as well as recipes that allow automated configuration and management of a Cisco Nexus 9000 Series Switch. The recipe, when deployed on a Cisco Nexus 9000 Series Switch, translates into network configuration settings and commands for collecting statistics and analytics information. OpenFlow: Cisco supports OpenFlow through its OpenFlow plug-ins that are installed on NX-OSpowered devices and through the Cisco ONE Enterprise Network Controller that is installed on a server and manages the devices using the OpenFlow interface. ONE Controller, in turn, provides Java abstractions to applications to abstract and manage the network Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 80

10 Cisco OnePK : OnePK is an easy-to-use toolkit for development, automation, rapid service creation, and more. It supports C, Java, and Python, and integrates with PyCharm, PyDev, Eclipse, IDLE, NetBeans, and more. With its rich set of APIs, you can easily access the valuable data inside your network and perform functions. Examples include customizing route logic; creating flow-based services such as quality of service (QoS); adapting applications for changing network conditions such as bandwidth; automating workflows spanning multiple devices; and empowering management applications with new information. Guest Shell: Starting with Cisco NX-OS Software Release 6.1(2)I3(1), the Cisco Nexus 9000 Series devices support access to a decoupled execution space called the Guest Shell. Within the guest shell the network-admin is given Bash access and may use familiar Linux commands to manage the switch. The guest shell environment has: Access to the network, including all VRFs known to the NX-OS Software Read and write access to host Cisco Nexus 9000 bootflash Ability to execute Cisco Nexus 9000 CLI Access to Cisco onepk APIs The ability to develop, install, and run Python scripts The ability to install and run 64-bit Linux applications A root file system that is persistent across system reloads or switchovers Integration with Orchestrators & Management Platforms Cisco UCS Director: Cisco UCS Director provides easy management of Cisco converged infrastructure platforms, vblock, and FlexPod, and provides end-to-end provisioning and monitoring of uses cases around Cisco UCS Servers, Nexus switches, and Storage Arrays. Cisco Prime Data Center Network Manager (DCNM) is a very powerful tool for centralized data center monitoring, managing, and automation of Cisco data center compute, network, and storage infrastructure. A basic version of DCNM is available for free, with more advanced features requiring a license. DCNM allows centralized management of all Cisco Nexus switches, Cisco UCS, and Cisco MDS devices. OpenStack: OpenStack is the leading open-source cloud management platform. The Cisco Nexus 9000 Series includes plug-in support for OpenStack s Neutron. The Cisco Nexus plug-in accepts OpenStack Networking API calls and directly configures Cisco Nexus switches as well as Open vswitch (OVS) running on the hypervisor. Not only does the Cisco Nexus plug-in configure VLANs on both the physical and virtual network, but it also intelligently allocates VLAN IDs, de-provisioning them when they are no longer needed and reassigning them to new tenants whenever possible. VLANs are configured so that virtual machines running on different virtualization (computing) hosts that belong to the same tenant network transparently communicate through the physical network. Moreover, connectivity from the computing hosts to the physical network is trunked to allow traffic only from the VLANs configured on the host by the virtual switch. For more information, visit OpenStack at Cisco Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 80

11 Policy-based networking to address dynamic application provisioning: Nexus 9000 can operate in standalone and ACI mode. ACI mode provides rich-featured, policy-based networking constructs and a framework to holistically define infrastructure needs of applications as well as provision and manage the infrastructure. Some of the constructs supported by ACI include definition of Tenants, Applications, End Point Groups (EPGs), Contracts & Policies and association of EPGs to the network fabric. For more details, refer to the Cisco Application Centric Infrastructure Design Guide: 5. Key Features of Cisco NX-OS Software Cisco NX-OS Software for Cisco Nexus 9000 Series Switches works in two modes: Standalone Cisco NX-OS deployment Cisco ACI deployment The Cisco Nexus 9000 Series uses an enhanced version of Cisco NX-OS Software with a single binary image that supports every switch in the series to simplify image management. Cisco NX-OS is designed to meet the needs of a variety of customers, including midmarket, enterprise, service providers, and a range of specific industries. Cisco NX-OS allows customers to create a stable and standard switching environment in the data center for the LAN and SAN. Cisco NX-OS is based on a highly secure, stable, and standard Linux core, providing a modular and sustainable base for the long term. Built to unify and simplify the data center, Cisco NX-OS provides the networking software foundation for the Cisco Unified Data Center. Its salient features include: Modularity - Cisco Nx-OS provides isolation between control and data forwarding planes within the device and between software components, so that a failure within one plane or process does not disrupt others. Most system functions, features, and services are isolated so that they can be started as well as restarted independently in case of failure while other services continue to run. Most system services can perform stateful restarts, which allow the service to resume operations transparently to other services. Resilience - Cisco NX-OS is built from the foundation to deliver continuous, predictable, and highly resilient operations for the most demanding network environments. With fine-grained process modularity, automatic fault isolation and containment, and tightly integrated hardware resiliency features, Cisco NX-OS delivers a highly reliable operating system for operation continuity. Cisco NX-OS resiliency includes several features. Cisco In-Service Software Upgrade (ISSU) provides problem fixes, feature enhancements, and even full OS upgrades without interrupting operation of the device. Per-process modularity allows customers to restart individual processes or update individual processes without disrupting the other services on the device. Cisco NX- OS also allows for separation of the control plane from the data plane; data-plane events cannot block the flow of control commands, helping to ensure uptime. Efficiency - Cisco NX-OS includes a number of traditional and advanced features to ease implementation and ongoing operations. Monitoring tools, analyzers, and clustering technologies are integrated into Cisco NX-OS. These features provide a single point of management that simplifies operations and improves efficiency Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 80

12 Having a single networking software platform that spans all the major components of the data center network creates a predictable, consistent environment that makes it easier to configure the network, diagnose problems, and implement solutions. Cisco Data Center Network Manager (DCNM) is a centralized manager that can handle all Cisco NX-OS devices, allowing centralization of all the monitoring and analysis performed at the device level and providing a high level of overall control. Furthermore, Cisco NX- OS offers the same industry-standard command-line environment that was pioneered in Cisco IOS Software, making the transition from Cisco IOS Software to Cisco NX-OS Software easy. Virtualization - Cisco NX-OS is designed to deliver switch-level virtualization. With Cisco NX-OS, switches can be virtualized in many logical devices, each operating independently. Device partitioning is particularly useful in multi-tenant environments and in environments in which strict separation is necessary due to regulatory concerns. Cisco NX-OS provides VLANs and VSANs and also supports newer technologies such as VXLAN, helping enable network segregation. The technologies incorporated into Cisco NX-OS provide tight integration between the network and virtualized server environments, enabling simplified management and provisioning of data center resources. For additional information about Cisco NX-OS refer to the Cisco Nexus 9500 and 9300 Series Switches NX-OS Software data sheet. 6. Data Center Design Considerations 6.1 Traditional Data Center Design Traditional data centers are built on a three-tier architecture with core, aggregation, and access layers (Figure 3), or a two-tier collapsed core with the aggregation and core layers combined into one layer (Figure 4). This architecture accommodates a north-south traffic pattern where client data comes in from the WAN or Internet to be processed by a server in the data center, and is then pushed back out of the data center. This is common for applications like web services, where most communication is between an external client and an internal server. The north-south traffic pattern permits hardware oversubscription, since most traffic is funneled in and out through the lower-bandwidth WAN or Internet bottleneck. A classic network in the context of this document is the typical three-tier architecture commonly deployed in many data center environments. It has distinct core, aggregation, and access layers, which together provide the foundation for any data center design. Table 1 outlines the layers of a typical tier-three design, and the functions of each layer. Table 1. Layer Core Aggregation Classic Three-Tier Data Center Design Description This tier provides the high-speed packet switching backplane for all flows going in and out of the data center. The core provides connectivity to multiple aggregation modules and provides a resilient, Layer 3 -routed fabric with no single point of failure (SPOF). The core runs an interior routing protocol, such as Open Shortest Path First (OSPF) or Border Gateway Protocol (BGP), and load-balances traffic between all the attached segments within the data center. This tier provides important functions, such as service module integration, Layer 2 domain definitions and forwarding, and gateway redundancy. Server-to-server multitier traffic flows through the aggregation layer and can use services, such as firewall and server load balancing, to optimize and secure applications. This layer provides the Layer 2 and 3 demarcations for all northbound and southbound traffic, and it processes most of the eastbound and westbound traffic within the data center Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 80

13 Layer Access Description This tier is the point at which the servers physically attach to the network. The server components consist of different types of servers: Blade servers with integral switches Blade servers with pass-through cabling Clustered servers Possibly mainframes The access-layer network infrastructure also consists of various modular switches and integral blade server switches. Switches provide both Layer 2 and Layer 3 topologies, fulfilling the various server broadcast domain and administrative requirements. In modern data centers, this layer is further divided into a virtual access layer using hypervisor-based networking, which is beyond the scope of this document. Figure 3. Traditional Three-Tier Design Figure 4. Traditional Two-Tier Medium Access-Aggregation Design Customers may also have the small collapsed design replicated to another rack or building, with a Layer 3 connection between the pods (Figure 5) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 80

14 Figure 5. Layer 2 Access, Layer 3 Connection between Pods Some of the considerations used in making a decision on the Layer 2 and Layer 3 boundary are listed in Table 2. Table 2. Layer 2 and Layer 3 Boundaries Consideration Layer 3 at Core Layer 3 at Access Multipathing Spanning Tree Layer 2 reach Convergence time One active path per VLAN due to Spanning Tree between access and core switches More Layer 2 links, therefore more loops and links to block Greater Layer 2 reachability for workload mobility and clustering applications Spanning Tree generally has a slower convergence time than a dynamic routing protocol Equal cost multipathing using dynamic routing protocol between access and core switches No Spanning Tree Protocol (STP) running north of the access layer Layer 2 adjacency is limited to devices connected to the same access layer switch Dynamic routing protocols generally have a faster convergence time than Spanning Tree 6.2 Leaf-Spine Architecture Spine-Leaf topologies are based on the Clos network architecture. The term originates from Charles Clos at Bell Laboratories, who published a paper in 1953 describing a mathematical theory of a multipathing, non-blocking, multiple-stage network topology in which to switch telephone calls. Today, Clos original thoughts on design are applied to the modern Spine-Leaf topology. Spine-leaf is typically deployed as two layers: spines (like an aggregation layer), and leaves (like an access layer). Spine-leaf topologies provide high-bandwidth, low-latency, non-blocking server-to-server connectivity. Leaf (aggregation) switches are what provide devices access to the fabric (the network of Spine and Leaf switches) and are typically deployed at the top of the rack. Generally, devices connect to the Leaf switches. Devices may include servers, Layer 4-7 services (firewalls and load balancers), and WAN or Internet routers. Leaf switches do not connect to other leaf switches (unless running vpc in standalone NX-OS mode). However, every leaf should connect to every spine in a full mesh. Some ports on the leaf will be used for end devices (typically 10 Gigabits), and some ports will be used for the spine connections (typically 40 Gigabits). Spine (aggregation) switches are used to connect to all Leaf switches, and are typically deployed at the end or middle of the row. Spine switches do not connect to other spine switches. Spines serve as backbone interconnects for Leaf switches. Generally, spines only connect to leaves, but when integrating a Cisco Nexus 9000 switch into an existing environment it is perfectly acceptable to connect other switches, services, or devices to the spines. All devices connected to the fabric are an equal number of hops away from one another. This delivers predictable latency and high bandwidth between servers. The diagram in Figure 6 shows a simple two-tier design Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 80

15 Figure 6. Two-Tier Design and Connectivity Another way to think about the Spine-Leaf architecture is by thinking of the Spines as a central backbone, with all leaves branching off the spine like a star. Figure 7 depicts this logical representation, which uses identical components laid out in an alternate visual mapping. Figure 7. Logical Representation of a Two-Tier Design 6.3 Spanning Tree Support The Cisco Nexus 9000 supports two Spanning Tree modes: Rapid Per-VLAN Spanning Tree Plus (PVST+), which is the default mode, and Multiple Spanning Tree (MST). The Rapid PVST+ protocol is the IEEE 802.1w standard, Rapid Spanning Tree Protocol (RSTP), implemented on a per-vlan basis. Rapid PVST+ interoperates with the IEEE 802.1Q VLAN standard, which mandates a single STP instance for all VLANs, rather than per VLAN. Rapid PVST+ is enabled by default on the default VLAN (VLAN1) and on all newly created VLANs on the device. Rapid PVST+ interoperates with devices that run legacy IEEE 802.1D STP. RSTP is an improvement on the original STP standard, 802.1D, which allows faster convergence. MST maps multiple VLANs into a Spanning Tree instance, with each instance having a Spanning Tree topology independent of other instances. This architecture provides multiple forwarding paths for data traffic, enables load balancing, and reduces the number of STP instances required to support a large number of VLANs. MST improves the fault tolerance of the network because a failure in one instance does not affect other instances. MST provides rapid convergence through explicit handshaking because each MST instance uses the IEEE 802.1w standard. MST improves Spanning Tree operation and maintains backward compatibility with the original 802.1D Spanning Tree and Rapid PVST Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 80

16 6.4 Layer 2 versus Layer 3 Implications Data center traffic patterns are changing. Today, more traffic moves east-west from server to server through the access layer, as servers need to talk to each other and consume services within the data center. The oversubscribed hardware now isn t sufficient for the east-west 10 Gigabit-to-10 Gigabit communications. Additionally, east-west traffic is often forced up through the core or aggregation layer, taking a suboptimal path. Spanning Tree is another hindrance in the traditional three-tier data center design. Spanning Tree is required to block loops in flooding Ethernet networks so that frames are not forwarded endlessly. Blocking loops means blocking links, leaving only one active path (per VLAN). Blocked links severely impact available bandwidth and oversubscription. This can also force traffic to take a suboptimal path, as Spanning Tree may block a more desirable path (Figure 8). Figure 8. Suboptimal Path between Servers in Different Pods Due to Spanning Tree Blocked Links Addressing these issues could include: a. Upgrading hardware to support 40- or 100-Gigabit interfaces b. Bundling links into port channels to appear as one logical link to Spanning Tree c. Or, moving the Layer 2/Layer 3 boundary down to the access layer to limit the reach of Spanning Tree (Figure 9). Using a dynamic routing protocol between the two layers allows all links to be active, fast reconvergence, and equal cost multipathing (ECMP). Figure 9. Two-Tier Routed Access Layer Design 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 80

17 The tradeoff in moving Layer 3 routing to the access layer in a traditional Ethernet network is that it limits Layer 2 reachability. Applications like virtual-machine workload mobility and some clustering software require Layer 2 adjacency between source and destination servers. By routing at the access layer, only servers connected to the same access switch with the same VLANs trunked down would be Layer 2-adjacent. This drawback is addressed by using VXLAN, which will be discussed in subsequent section. 6.5 Virtual Port Channels (vpc) Over the past few years many customers have sought ways to move past the limitations of Spanning Tree. The first step on the path to Cisco Nexus-based modern data centers came in 2008 with the advent of Cisco virtual Port Channels (vpc). A vpc allows a device to connect to two different physical Cisco Nexus switches using a single logical port-channel interface (Figure 10). Prior to vpc, port channels generally had to terminate on a single physical switch. vpc gives the device activeactive forwarding paths. Because of the special peering relationship between the two Cisco Nexus switches, Spanning Tree does not see any loops, leaving all links active. To the connected device, the connection appears as a normal port-channel interface, requiring no special configuration. The industry standard term is called Multi- Chassis EtherChannel; the Cisco Nexus-specific implementation is called vpc. Figure 10. vpc Physical Versus Logical Topology vpc deployed on a Spanning Tree Ethernet network is a very powerful way to curb the number of blocked links, thereby increasing available bandwidth. vpc on the Cisco Nexus 9000 is a great solution for commercial customers, and those satisfied with current bandwidth, oversubscription, and Layer 2 reachability requirements. Two of the sample small-to-midsized traditional commercial topologies are depicted using vpcs in Figures 11 and 12. These designs leave the Layer 2/Layer 3 boundary at the aggregation to permit broader Layer 2 reachability, but all links are active as Spanning Tree does not see any loops to block Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 80

18 Figure 11. Traditional One-Tier Collapsed Design with vpc Figure 12. Traditional Two-Tier Design with vpc 6.6 Overlays Overlay technologies are useful in extending the Layer 2 boundaries of a network across a large data center as well as across different data centers. This section discusses some of the important overlay technologies, including: An overview of VXLAN VXLAN with BGP EVPN as the control plane VXLAN Data Center Interconnect with a BGP control plane For more information on overlays in general, read the Data Center Overlay Technologies white paper Virtual Extensible LAN (VXLAN) VXLAN is a Layer 2 overlay scheme over a Layer 3 network. It uses an IP/User Datagram Protocol (UDP) encapsulation so that the provider or core network does not need to be aware of any additional services that VXLAN is offering. A 24-bit VXLAN segment ID or VXLAN network identifier (VNI) is included in the encapsulation to provide up to 16 million VXLAN segments for traffic isolation and segmentation, in contrast to the 4000 segments achievable with VLANs. Each of these segments represents a unique Layer 2 broadcast domain and can be administered in such a way that it can uniquely identify a given tenant's address space or subnet (Figure 13) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 80

19 Figure 13. VXLAN Frame Format VXLAN can be considered a stateless tunneling mechanism, with each frame encapsulated or de-encapsulated at the VXLAN tunnel endpoint (VTEP) according to a set of rules. A VTEP has two logical interfaces: an uplink and a downlink (Figure 14). Figure 14. VTEP Logical Interfaces 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 80

20 The VXLAN draft standard does not mandate a control protocol for discovery or learning. It offers suggestions for both control-plane source learning (push model) and central directory-based lookup (pull model). At the time of this writing, most implementations depend on a flood-and-learn mechanism to learn the reachability information for end hosts. In this model, VXLAN establishes point-to-multipoint tunnels to all VTEPs on the same segment as the originating VTEP to forward unknown and multi-destination traffic across the fabric. This forwarding is accomplished by associating a multicast group for each segment, so it requires the underlying fabric to support IP multicast routing. Cisco Nexus 9000 Series Switches provide Layer 2 connectivity extension across IP transport within the data center, and easy integration between VXLAN and non-vxlan infrastructures. The section, 9.3.2, further in this document provides detailed configurations to configure the VXLAN fabric using both a Multicast/Internet Group Management Protocol (IGMP) approach as well as a BGP EVPN control plane. Beyond a simple overlay sourced and terminated on the switches, the Cisco Nexus 9000 can act as a hardwarebased VXLAN gateway. VXLAN is increasingly popular for virtual networking in the hypervisor for virtual machineto-virtual machine communication, and not just switch to switch. However, many devices are not capable of supporting VXLAN, such as legacy hypervisors, physical servers, and service appliances like firewalls, load balancers, and storage devices. Those devices need to continue to reside on classic VLAN segments. It is not uncommon that virtual machines in a VXLAN segment need to access services provided by devices in a classic VLAN segment. The Cisco Nexus 9000 acting as a VXLAN gateway can provide the necessary translation, as shown in Figure 15. Figure 15. Cisco Nexus 9000 Leaf Switches Translate VLAN to VXLAN 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 80

21 6.6.2 BGP EVPN Control Plane for VXLAN The EVPN overlay draft specifies adaptations to the BGP Multiprotocol Label Switching (MPLS)-based EVPN solution to enable it to be applied as a network virtualization overlay with VXLAN encapsulation where: The Provider Edge (PE) node role described in BGP MPLS EVPN is equivalent to the VTEP or network virtualization edge (NVE) device VTEP endpoint information is distributed using BGP VTEPs use control-plane learning and distribution through BGP for remote MAC addresses instead of data plane learning Broadcast, unknown unicast, and multicast data traffic is sent using a shared multicast tree or with ingress replication BGP Route Reflector (RR) is used to reduce the full mesh of BGP sessions among VTEPs to a single BGP session between a VTEP and the RR Route filtering and constrained route distribution are used to help ensure that control plane traffic for a given overlay is only distributed to the VTEPs that are in that overlay instance The Host MAC mobility mechanism is in place to help ensure that all the VTEPs in the overlay instance know the specific VTEP associated with the MAC Virtual network identifiers are globally unique within the overlay The EVPN overlay solution for VXLAN can also be adapted to be applied as a network virtualization overlay with VXLAN for Layer 3 traffic segmentation. The adaptations for Layer 3 VXLAN are similar to Layer 2 VXLAN except: VTEPs use control plane learning and distribution using the BGP of IP addresses (instead of MAC addresses) The virtual routing and forwarding instance is mapped to the VNI The inner destination MAC address in the VXLAN header does not belong to the host, but to the receiving VTEP that does the routing of the VXLAN payload. This MAC address is distributed using a BGP attribute along with EVPN routes Note that since IP hosts have an associated MAC address, coexistence of both Layer 2 VXLAN and Layer 3 VXLAN overlays will be supported. Additionally, the Layer 2 VXLAN overlay will also be used to facilitate communication between non-ip based (Layer 2-only) hosts VXLAN Data Center Interconnect (DCI) with a BGP Control Plane The BGP EVPN control plane feature of the Cisco Nexus 9000 can be used to extend the reach of Layer 2 domains across data center pods, domains, and sites. Figure 16 shows two overlays in use: Overlay Transport Virtualization (OTV) on a Cisco ASR 1000 for data centerto-data center connectivity, and VXLAN within the data center. Both provide Layer 2 reachability and extension. OTV is also available on the Cisco Nexus 7000 Series Switch Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 80

22 Figure 16. Virtual Overlay Networks Provide Dynamic Reachability for Applications 7. Integration into Existing Networks In migrating your data center to the Cisco Nexus 9000 Series, you need to consider, not only compatibility with existing traditional servers and devices; you also need to consider the next-generation capabilities of Cisco Nexus 9000 switches, which include: VXLAN with added functionality of BGP control plane for scale Fabric Extender (FEX) vpc 10/40 Gbps connectivity Programmability With their exceptional performance and comprehensive feature set, Cisco Nexus 9000 Series Switches are versatile platforms that can be deployed in multiple scenarios, including: Layered access-aggregation-core designs Leaf-and-spine architecture Compact aggregation-layer solutions Cisco Nexus 9000 Series Switches deliver a comprehensive Cisco NX-OS Software data center switching feature set. Table 2 lists the current form factors. Visit for latest updates to Cisco Nexus 9000 portfolio Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 80

23 Table 3. Cisco Nexus 9000 Series Switches Device Model Line Cards and Expansion Modules Description Deployment Cisco Nexus 9500 Modular Switch N9K-X9636PQ 36-port 40-Gbps Enhanced Quad Small Form-Factor Pluggable (QSFP+) End of row (EoR), middle of row (MoR), aggregation layer, and core N9K-X9564TX 48-port 1/10GBASE-T plus 4-port 40-Gbps QSFP+ Cisco Nexus 9396PX Switch Cisco Nexus 93128TX Switch N9K-X9564PX 48-port 1/10-Gbps SFP+ plus 4-port 40-Gbps QSFP+ N9K-C9396PX Cisco Nexus 9300 platform with 48-port 1/10- Gbps SFP+ N9K-C93128TX Cisco Nexus 9300 platform with 96-port 1/10GBASE-T Top of rack (ToR), EoR, MoR, aggregation layer, and core ToR, EoR, MoR, aggregation layer, and core 7.1 Pod Design with vpc A vpc allows links physically connected to two different Cisco Nexus 9000 Series Switches to appear as a single PortChannel to a third device. A vpc can provide Layer 2 multipathing, which allows the creation of redundancy by increasing bandwidth, supporting multiple parallel paths between nodes, and load-balancing of traffic where alternative paths exist. The vpc design remains the same as described in the vpc design guide, with the exception that the Cisco Nexus 9000 Series does not support vpc active-active or two-layer vpc (evpc). Refer to the vpc design and best practices guide for more information: pdf. Figure 17 shows a next-generation data center with Cisco Nexus switches and vpc. There is a vpc between the Cisco Nexus 7000 Series Switches and the Cisco Nexus 5000 Series Switches, a dual-homed vpc between the Cisco Nexus 5000 Series Switches and the Cisco Nexus 2000 Series FEXs, and a dual-homed vpc between the servers and the Cisco Nexus 2000 Series FEXs. Figure 17. vpc Design Considerations with Cisco Nexus 7000 Series in the Core 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 80