VMware vsphere and Cumulus Linux Validated Design Guide. Deploying VMware vsphere with Network Switches Running Cumulus Linux

VMware vsphere and Cumulus Linux Validated Design Guide Deploying VMware vsphere with Network Switches Running Cumulus Linux

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Contents Contents... 2 VMware vsphere with Cumulus Linux... 4 Objective... 4 Enabling Choice of Hardware in the Data Center... 4 Combined Solution Using VMware vsphere and Cumulus Linux... 4 Driving Towards Operational Efficiencies... 5 Intended Audience for Network Design and Build... 6 VMware vsphere in a Traditional Layer 2 Enterprise Data Center... 6 Network Architecture and Design Considerations... 6 Management and Out-of-Band Networking Considerations... 9 Storage Considerations... 9 NFS NAS Configuration Considerations... 10 Hyper-Converged Storage... 10 iscsi SAN Configuration Considerations... 11 Scaling Out... 12 External Connectivity... 13 External Connectivity at Layer 2... 13 External Connectivity at Layer 3... 14 Building a Data Center Network with VMware vsphere and Cumulus Linux... 15 Network Architecture for Hierarchical Layer 2 Leaf, Spine, and Core... 15 Build Steps... 18 1. Set Up Physical Network and Basic Configuration of All Switches... 19 2. Configure Leaf Switches... 20 3. Configure Spine Switches... 26 4. Set Up Spine/Leaf Network Fabric... 29 5. Configure VLANs... 31 6. Connect vsphere ESXi Hosts... 33 7. Configure Virtual Networking... 35 8. Connect Spine Switches to Core... 35 Automation Considerations... 38 Automation and Converged Administration... 38 Automated Switch Provisioning with Zero Touch Provisioning... 38 Network Configuration Templates Using Mako... 39 Automated Network Configuration Using Ansible... 40 Automated Network Configuration Using Puppet... 43 2

CONTENTS Operational and Management Considerations... 47 Authentication and Authorization... 47 Security Hardening... 47 Accounting and Monitoring... 48 Quality of Service (QoS) Considerations... 48 Link Pause... 49 Conclusion... 50 Summary... 50 References... 50 Appendix A: Example /etc/network/interfaces Configurations... 52 leaf01... 52 leaf02... 54 spine01... 56 spine02... 58 oob-mgmt... 60 Appendix B: Configuring a vds... 61 Appendix C: Network Design and Setup Checklist... 65 Version 1.0.1 February 6, 2015 About Cumulus Networks Unleash the power of Open Networking with Cumulus Networks. Founded by veteran networking engineers from Cisco and VMware, Cumulus Networks makes the first Linux operating system for networking hardware and fills a critical gap in realizing the true promise of the software-defined data center. Just as Linux completely transformed the economics and innovation on the server side of the data center, Cumulus Linux is doing the same for the network. It is radically reducing the costs and complexities of operating modern data center networks for service providers and businesses of all sizes. Cumulus Networks has received venture funding from Andreessen Horowitz, Battery Ventures, Sequoia Capital, Peter Wagner and four of the original VMware founders. For more information visit cumulusnetworks.com or @cumulusnetworks. 2015 Cumulus Networks. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the Marks ) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their respective owners. VMware products are covered by one or more patents listed at http://vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. www.cumulusnetworks.com 3

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE VMware vsphere with Cumulus Linux Objective This Validated Design Guide presents a design and implementation approach for deploying VMware vsphere on network switches running Cumulus Linux. Enabling Choice of Hardware in the Data Center Server virtualization revolutionized the data center by giving IT the freedom and choice of industry-standard server hardware, thereby lowering CapEx costs, as well as a rich ecosystem of automation tools and services to provision and manage compute and storage infrastructure to achieve lower OpEx costs. This same benefit of choice is now available for networking in the data center. With Cumulus Linux, network administrators have a multi-platform, multi-vendor network OS that provides the freedom of choice with network switch hardware. Because Cumulus Linux is Linux, data center administrators can tap the already established wealth of Linux knowledge and vast application ecosystem for administering servers and extend them to network switches for converged administration. VMware s vsphere hypervisor platform enables data center architects to use industry-standard server hardware and provides high availability and disaster recovery functions that come with top tier hardware platforms. The use of a hypervisor ensures the application can be fully portable on top of a commodity server and, in the event of a system failure or capacity expansion, the workload can be moved to another server, rack or row within the data center. The end result is radical CapEx and OpEx savings, while ensuring the same level of application availability, which is why vsphere is found in many enterprise and commercial IT environments. Cumulus Linux can help you achieve the same CapEx and OpEx efficiencies for your networks by enabling an open market approach for switching platforms, and by offering a radically simple lifecycle management framework built on the industry s best open source tools. By using bare metal servers and network switches, you can achieve cost savings that would have been impossible just a few years ago Combined Solution Using VMware vsphere and Cumulus Linux Cumulus Linux and VMware vsphere are software solutions run on top of bare metal, industry-standard hardware. This allows customers to build compute, storage, and now network platforms from a wide array of suppliers who often employ highly competitive pricing models. The software defines the performance and behavior of the environment and allows the administrator to exercise version control and programmatic approaches that are already in use by DevOps teams. Refer to the Cumulus Linux Hardware Compatibility List (HCL) at cumulusnetworks.com/hcl for a list of hardware vendors and their supported model numbers, descriptions, switch silicon, and CPU type. 4

VMWARE VSPHERE WITH CUMULUS LINUX Driving Towards Operational Efficiencies Figure 1. VMware vsphere and Cumulus Linux A deterministic and predictable data center is paramount, and Cumulus Linux is an excellent solution in conjunction with vsphere to deploy servers and networks together. With a disaggregated hardware and software model, both VMware and Cumulus Networks have created a simple approach for managing both servers and networks. In the VMware server environment, the software-defined data center (SDDC) enables an architect to model all the services and rules deployed in a traditional physical data center. Once this model is built, an entire virtual data center can be cloned, moved, and expanded without any changes to the physical environment. In addition, with the advent of multi-tenancy, data centers can be created for a single business unit or application to ensure micro-segmentation and strict access control. Likewise, Cumulus Linux simplifies network management through hardware agnosticism and using standard Linux automation and orchestration tools. Cumulus Linux is a full-featured network operating system and presents a standard interface, regardless of the underlying hardware or features enabled. Automation tools can process templates that are written once and reused across the entire environment for provisioning and configuration management, substantially decreasing the number of upper level management systems and operating expenses as well as increasing the speed at which new networks are deployed. You can change key variables as needed in a template and power on an entirely new network in minutes without interacting with a CLI. You can leverage open source protocols and tools such as DHCP, Open Network Install Environment (ONIE), zero touch provisioning, and automation and orchestration tools of DevOps choice such as Ansible, Chef, Puppet, Salt, and CFEngine. These same tools are already used by many organizations to simplify server deployments, and modifying them to provision entire racks of vsphere ESXi hosts and network switches becomes a simple task of converged administration. www.cumulusnetworks.com 5

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Intended Audience for Network Design and Build The intended audience for this guide is a data center cloud architect or administrator experienced with VMware vsphere and familiar with Layer 2 networking, including interfaces, link aggregation (LAG) or bonds, and VLANs. The network architecture and build steps provided in this document can be used as a reference for planning and implementing vsphere with Cumulus Linux in your environment. A basic understanding of Linux commands is assumed, such as accessing a Linux installation, navigating the file system, and editing files. If you are using this guide to help you set up your vsphere and Cumulus Linux environment, we assume you have Cumulus Linux installed and licensed on switches from the Cumulus Linux Hardware Compatibility List (HCL) at cumulusnetworks.com/hcl. Additional information on Cumulus Linux software, licensing, and supported hardware may be found on cumulusnetworks.com or by contacting sales@cumulusnetworks.com. VMware vsphere in a Traditional Layer 2 Enterprise Data Center Network Architecture and Design Considerations The network design of a typical enterprise data center pod running VMware vsphere is shown in Figure 2. The pod consists of a pair of aggregation/spine switches connected with one or more pairs of access/leaf switches, which in turn provide dual-homed connectivity to ESXi hosts and storage elements. Figure 2. Traditional Layer 2 Hierarchical Enterprise Data Center Network Pod 6

VMWARE VSPHERE IN A TRADITIONAL LAYER 2 ENTERPRISE DATA CENTER This document describes the network architecture for a vsphere data center that follows the traditional hierarchical aggregation and access, also called spine and leaf, structure where Layer 2 networking is used to connect all elements. For optimal network performance, hosts are connected via dual 10G links to the access/leaf switch layer, which in turn is connected via 40G links to the aggregation/spine layer. Representative spine and leaf switches running Cumulus Linux are show below in Figure 3, although the details of your specific models may vary. Spine switches typically have thirty-two 40G switch ports, whereas leaf switches have forty-eight 10G access ports and up to six 40G uplinks. In Cumulus Linux, switch ports are labeled swp1, swp2, swp3, and so forth. Figure 3. Switch Port Numbering for Aggregation/Spine and Access/Leaf Switches The network design employs multi-chassis link aggregation for network path redundancy and link aggregation for network traffic optimization. Multi-Chassis Link Aggregation, or CLAG, is the MLAG implementation in Cumulus Linux. The use of CLAG allows for active/active ports across physical switches and avoids the underutilized network situation of an environment running just Spanning Tree Protocol (STP) with blocked ports to prevent network loops. CLAG is deployed using the following steps: Select a pair of physical switches, either a pair of leaf switches or a pair of spine switches. There can only be two switches in a single CLAG logical switch configuration, although you can have multiple CLAG pairs in a topology. Establish a peer link between pair members. It is recommended to set up the peer link as an LACP bond for increased reliability and bandwidth. A clagd daemon runs on each switch in the CLAG pair and communicates over a subinterface of the peer link. Figure 4. Switch-to-Switch CLAG The peer link between the switches in a CLAG pair should be sized accordingly to ensure there is sufficient bandwidth to handle additional traffic during uplink failure scenarios. Typical deployments utilize a bond for the peer link for both bandwidth and redundancy reasons. www.cumulusnetworks.com 7

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE All ESXi hosts allow for load balancing via Route based on the originating virtual port ID. These deployments leverage Cumulus Linux as independent bridges;, contact your Cumulus Networks Customer Solutions Engineer or file a support case for help in with these deployments. This design guide assumes the typical enterprise use case of vsphere deployments leveraging the vsphere Distributed Switch (vds) to provide LACP bonds between the host and the leaf switches, and provide load balancing based on IP hash, as shown in Figure 5. Figure 5. Host Link Redundancy and Aggregation, LACP Refer to the KB article, Enabling or disabling LACP on an Uplink Port Group using the vsphere Web Client (2034277), kb.vmware.com/kb/2034277/ for further information. Using LACP with a vds ensures that a logical pair of switches configured for CLAG appears as a single logical switch from both the host and switch sides. In a typical vsphere environment, VLANs are used to segment traffic by type such as VMs, VMkernel (enabled for vsphere services such as vmotion traffic, fault tolerance logging, management traffic, virtual SAN traffic), or VMkernel for storage (iscsi). To allow room for growth, several hundred VLANs are allocated, even though not all VLANs may be used initially. Figure 6. Host Network Segmentation 8

VMWARE VSPHERE IN A TRADITIONAL LAYER 2 ENTERPRISE DATA CENTER Management and Out-of-Band Networking Considerations An important supplement to the high capacity production data network is the management network used to administer infrastructure elements, such as network switches, physical servers, and storage systems. The architecture of these networks vary considerably based on their intended use, the elements themselves, and access isolation requirements. This solution guide assumes that a single Layer 2 domain is used to administer the network switches, storage elements, and vmkernel interfaces on the ESXi hosts. These operations include imaging the elements, configuring them, and monitoring the running system. Some installations will also use this network for IPMI, also known as DRAC or ilo, access to the ESXi host. This network is expected to host both DHCP and HTTP servers, such as isc-dhcp and apache2, as well as provide DNS reverse and forward resolution. In general, these networks provide some means to connect to the corporate network, typically a connection through a router or jump host. Figure 7 below shows the logical and, where possible, the physical connections of each element as well as the services required to realize this deployment. Storage Considerations Figure 7. Out-of-Band Management Data, image, and metadata storage are fundamental components of any VMware vsphere deployment. Any form of network-based storage imparts specific considerations on the design and deployment. The most common forms of network storage used with vsphere are: NFS (file) Hyper-converged storage iscsi SAN (block) Refer to the vsphere Storage Guide in the VMware vsphere Documentation Center for a comprehensive look at the various storage solutions. Below, we detail the topologies and architectural considerations for each of these storage options and their impact to networking. This deployment guide targets NFS and Hyper-Converged storage solutions. iscsi deployments are very dependent on your target and goals, and a Cumulus Networks Customer Solutions Engineer or Consultant can provide guidance specific to your situation. www.cumulusnetworks.com 9

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE NFS NAS Configuration Considerations For general guidance on configuring vsphere with NFS storage, refer to VMware s technical paper, Best Practices for Running vsphere on NFS Storage. From a networking perspective, most NFS installations leverage LACP bonds to connect both ESXi hosts and networkattached storage (NAS, or NFS filers), as shown below in Figure 8. Figure 8. Connecting NFS Storage Typical NFS deployments leverage a common set of configurations: Jumbo frames (9K IP MTU) to increase per-packet performance of data transfers 10G links and LACP bonds with IP hashing for ESXi hosts and NFS filers for performance and redundancy One or more isolated VLANs allocated to VMkernel NFS interfaces and NFS filers Multiple NFS datastores to improve load balancing Hyper-Converged Storage Hyper-converged storage solutions leverage the flash and spindle-based storage on ESXi hosts to create a clustered storage solution. Example solutions include VMware Virtual SAN and Nutanix Virtual Computing Platform. Since each disk write will usually need to be replicated to at least one other host for failover purposes, hyper-converged storage platforms typically have significant east-west traffic requirements, ideally supported by a high capacity spine/leaf network topology. Each hyper-converged storage solution comes with its own set of networking best practices. For example, VMware Virtual SAN requires that all systems be connected as part of the same subnet (Layer 2 domain) and leverage IGMP for system operation. Therefore, Virtual SAN traffic is typically isolated to its own VLAN. Host VMkernel interfaces are added to this VLAN and configured to pass Virtual SAN traffic. Refer to the Virtual SAN Networking Requirements and Best Practices chapter in the vsphere Storage Guide in the vsphere Documentation Center for more information. On the switches running Cumulus Linux that pass Virtual SAN traffic, it is necessary to enable IGMP snooping on the Virtual SAN VLAN, and configure an IGMP querier within the fabric for optimal performance. Refer to the IGMP and MLD Snooping section of the Cumulus Linux Documentation for more information. Nutanix recommends hypervisor hosts and the Nutanix OS (NOS) controller VMs be placed on the same subnet (Layer 2 domain) for simplicity. In addition, IPv6 link-local address should be enabled to support the Web-based configuration tool. Nutanix replication traffic is fully routable and is sent unicast between Nutanix Controller VM (CVM) primary IP addresses. ESXi storage traffic to a Nutanix cluster is sent over a local connection direct to the native CVM on each ESXi host, except in CVM failover situations. In a failover scenario, storage is accessed via a single cluster IP address, presented by an elected CVM in the cluster. The cluster IP address is typically used for administering the cluster as well. For this reason, Nutanix 10

VMWARE VSPHERE IN A TRADITIONAL LAYER 2 ENTERPRISE DATA CENTER clusters in general do not span Layer 2 boundaries. Refer to the Nutanix Virtual Computing Platform Setup Guide for more information. Hyper-converged storage solutions are connected through ESXi host VMkernel ports, as shown in Figure 9. Figure 9. Connecting Hyper-Converged Storage Typical converged storage deployments leverage a common set of configurations: Jumbo frames (9K IP MTU) to increase per-packet performance of data transfers 10G links and LACP bonds with IP hashing for ESXi hosts for both performance and redundancy iscsi SAN Configuration Considerations For general guidance on configuring vsphere with iscsi SAN storage, refer to VMware s technical papers, Best Practices for Running vsphere on iscsi and Multipathing Configuration for Software iscsi Using Port Binding. You should also look for guidance specific to your iscsi target provider. iscsi storage network designs typically use dedicated physical NICs, VMkernel interfaces, and switch ports for storage traffic. In some cases, dedicated switches for the iscsi SAN are deployed. Alternatively, some deployments use NICs and network fabric that share bandwidth for storage, management, and VM traffic. Typical iscsi storage deployments leverage a common set of configurations: Jumbo frames (9K IP MTU) to increase per-packet performance of data transfers 10G links to both ESXi hosts and iscsi arrays www.cumulusnetworks.com 11

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Scaling Out Scaling out the vsphere architecture involves adding more hosts to the access switch pairs, and then adding more access switches in pairs as needed, as shown in Figure 10. Figure 10. Adding Additional Switches Once the limit for the spine switch pair approaches, an additional network pod of spine/leaf switches may be added, as shown in Figure 11. The only constraint is that in a Layer 2-only environment, additional spine switches should be added in pairs. Figure 11. Adding Network Pods/vSphere Clusters 12

EXTERNAL CONNECTIVITY External Connectivity The spine switches can connect to core switches in one of two ways, depending on where they sit relative to the Layer 2 and Layer 3 boundary: External connectivity at Layer 2, with routing and gateway services provided outside the cluster or by NSX for vsphere External connectivity at Layer 3, with routing services provided by the spine switches External Connectivity at Layer 2 In this scenario, the spine switches connect to core switches that handle gateway services as shown in Figure 12. Figure 12. External Connectivity at Layer 2 When connecting to core switches at Layer 2, the core switches need to support a vendor-specific form of multi-chassis link aggregation (e.g. vpc, MC-LAG, MLAG) with which the spine switches can pair. If the core switches are not capable of MLAG, then you may need to consider that with the VLAN-aware bridge introduced in Cumulus Linux 2.5, a single instance of spanning tree runs on each switch. By default, BPDUs are sent on the native VLAN 1 when spanning tree is enabled on the VLAN-aware bridge. The native VLAN should match on the core switches to ensure spanning tree compatibility. To adjust the untagged or native VLAN for an interface, refer to the Configure VLANs step in the Build Steps later in this document. In addition, all VLANs that are to be trunked to the core should be allowed on all the uplink trunk interfaces to avoid potentially blocked VLANs. www.cumulusnetworks.com 13

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE External Connectivity at Layer 3 In this scenario, the spine switches connect at Layer 3 as shown in Figure 13. Alternatively, the spine switches can be dual connected to each core switch at Layer 3 (not shown in Figure 13). Figure 13. External Connectivity at Layer 3 In this design, the spine switches route traffic. To connect to the core switches, you will need to determine whether the routing is static or dynamic, and the protocol OSPF or BGP if dynamic. In addition, you will need to provide a gateway and routing between all Layer 2 subnets on the spine switches. Additional network design considerations for this scenario include setting up Virtual Router Redundancy (VRR) between the spine switch pair to support active/active forwarding. For more information about VRR, read the Cumulus Linux Documentation. 14

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX Building a Data Center Network with VMware vsphere and Cumulus Linux Network Architecture for Hierarchical Layer 2 Leaf, Spine, and Core The instructions that follow detail the steps needed for building a representative network for VMware vsphere using switches running Cumulus Linux. Actual configurations reference the following network topology. Figure 14. Network Topology The details for the switches, hosts, and logical interfaces are as follows: leaf01 connected to Logical Interface Description Physical Interfaces leaf02 peerlink peer bond utilized for CLAG traffic swp47, swp48 leaf02 peerlink.4094 subinterface used for clagd communication N/A (swp45, swp46 reserved for expansion) spine01, spine02 uplink1 for CLAG between spine01 and spine02 swp49, swp50 future use uplink2 reserved for additional connections to spines swp51, swp52 multiple hosts access ports connect to hosts and storage swp1 through swp44 www.cumulusnetworks.com 15

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE leaf01 connected to Logical Interface Description Physical Interfaces esx-host-01 esx-host-01 bond to esx-host-01 for host-to-switch CLAG swp1 leaf02 connected to Logical Interface Description Physical Interfaces leaf01 peerlink peer bond utilized for CLAG traffic swp47, swp48 leaf01 peerlink.4094 subinterface used for clagd communication N/A (swp45, swp46 reserved for expansion) spine01, spine02 uplink for CLAG between spine01 and spine02 swp49, swp50 future use uplink2 reserved for additional connections to spines swp51, swp52 multiple hosts access ports connect to hosts and storage swp1 through swp44 esx-host-01 esx-host-01 bond to esx-host-01 for host-to-switch CLAG swp1 leaf0n connected to Logical Interface Description Physical Interfaces Repeat above configurations for each additional pair of leafs. spine01 connected to Logical Interface Description Physical Interfaces spine02 peerlink peer bond utilized for CLAG traffic swp25, swp26 spine02 peerlink.4093 subinterface used for clagd communication N/A leaf01, leaf02 leaf-pair-01 bond to another peer-link group swp1, swp2 core1, core2 core-uplink bond to core switches swp31, swp32 spine02 connected to Logical Interface Description Physical Interfaces spine01 peerlink peer bond utilized for CLAG traffic swp25, swp26 16

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX spine02 connected to Logical Interface Description Physical Interfaces spine01 peerlink.4093 subinterface used for clagd communication N/A leaf01, leaf02 downlink1 bond to another peer-link group swp1, swp2 core1, core2 core-uplink bond to core switches swp31, swp32 www.cumulusnetworks.com 17

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Build Steps The steps for building out a vsphere network environment with switches running Cumulus Linux are as follows. Step Tasks Physical Network 1. Set up physical network and basic configuration of all switches. Rack and cable all network switches. Install Cumulus Linux. Verify connectivity. Configure out-of-band management. Set hostname. Configure DNS. Network Topology 2. Configure leaf switches. Configure each switch in pair individually. Create peer bond between pair. Enable CLAG peering between leaf switches. 3. Configure spine switches. Configure each switch in pair individually. Create peer bond between pair. Enable CLAG peering between spine switches. 4. Set up spine/leaf network fabric. Create a switch-to-switch uplink bond on each leaf switch to the spine pair and verify CLAG. 5. Configure VLANs Set up VLANs for traffic. Create a switch-to-switch bond on the spine pair to each leaf switch pair and verify CLAG. 6. Connect vsphere ESXi hosts. Provision vsphere ESXi hosts (if not done already). Connect hosts to leaf switches. Configure vsphere Distributed Switch (vds) with LACP. 7. Configure virtual networking. Create VM and VMkernel networks using configured virtual switches. Use virtual networks. 8. Connect spine switches to core. Configure depending on Layer 2 or Layer 3 connectivity: For Layer 2: Create a CLAG port channel to the core for each spine switch. On each core, create an MLAG or vendor-equivalent Multi-Chassis Link Aggregation to the logical spine switch pair. For Layer 3: Configure Layer 3 switch virtual interface (SVI) gateways 18

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX In a green field environment, the order of configuring spine or leaf switches does not matter, so steps 3 and 4 above may be done in reverse order. In a brown field environment, start with leaf switches as shown above for minimal network service disruptions. The build order is detailed out in the following steps. Reference the Network Design and Setup Checklist in the Appendix of this guide for use in building out your network. 1. Set Up Physical Network and Basic Configuration of All Switches Rack and cable all network switches Install Cumulus Linux Configure out-of-band management Verify connectivity Set hostname Configure DNS After racking and cabling all switches, install the Cumulus Linux OS and license on each switch. Refer to the Quick Start Guide of the Cumulus Linux Documentation for more information. Next, configure the out-of-band management as needed. To verify all cables are connected and functional, check the link state. To check the link state of a switch port, run the ip link show command, which displays the physical link and administrative state of the interface and basic Layer 2 information. To show the Layer 3 information as well, such as IPv4 or IPv6 address, use the ip addr show command. Look for the UP states in the output. This command checks link state but does not detect traffic flow. Refer to the Link and Administrative State section of the Understanding Network Interfaces chapter of the Cumulus Linux Documentation for more information. cumulus@leaf01$ sudo ip link set up swp47 cumulus@leaf01$ ip link show swp47 49: swp47: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master peerlink state UP mode DEFAULT qlen 500 link/ether 70:72:cf:9d:4e:64 brd ff:ff:ff:ff:ff:ff To help verify that cables are properly connected according to your network topology diagram, check what neighbors are observed from each switch port. For example, to see what port is connected to swp47 on leaf01, run the following command and observe the LLDP neighbor s output to verify that swp47 on leaf02 is connected. cumulus@leaf01$ sudo lldpctl swp47 ------------------------------------------------------------------------------- LLDP neighbors: ------------------------------------------------------------------------------- Interface: swp47, via: LLDP, RID: 7, Time: 14 days, 20:06:51 Chassis: ChassisID: mac c4:54:44:bc:ff:f0 SysName: leaf02 SysDescr: Cumulus Linux MgmtIP: 192.168.0.91 Capability: Bridge, on Capability: Router, on Port: PortID: ifname swp47 PortDescr: swp47 ------------------------------------------------------------------------------- www.cumulusnetworks.com 19

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Note: To configure files in Cumulus Linux, use your choice of text editor. A standard Cumulus Linux installation includes nano, vi, and zile. Additionally, you can install additional editors such as vim from the Cumulus Linux repository, repo.cumulusnetworks.com. The examples in this guide use vi as the text editor. If you are not familiar with using vi, substitute the command with nano or zile, which may have more familiar user interfaces. The default configuration for eth0, the management interface, is DHCP. To reconfigure eth0 to use a static IP address, edit the /etc/network/interfaces file by adding an IP address/mask and an optional gateway. Refer to the Quick Start Guide, Wired Ethernet Management section of the Cumulus Linux Documentation for more information. For example: cumulus@leaf01$ sudo vi /etc/network/interfaces auto eth0 iface eth0 address 192.168.0.90/24 gateway 192.168.0.254 The default for the hostname of a switch running Cumulus Linux is cumulus. Change this to the appropriate name based on your network architecture diagram, e.g. leaf01 or spine01, by modifying /etc/hostname and /etc/hosts. Refer to the Quick Start Guide, Setting Unique Host Names section of the Cumulus Linux Documentation for more information. For example: cumulus@leaf01$ sudo vi /etc/hostname leaf01 cumulus@leaf01$ sudo vi /etc/hosts 127.0.0.1 leaf01 localhost Modify your DNS settings if needed. Add your domain, search domain, and nameserver entries to the /etc/resolv.conf file. cumulus@leaf01$ sudo vi /etc/resolv.conf domain example.com search example.com nameserver x.x.x.x 2. Configure Leaf Switches Configure each switch in the CLAG pair individually Create peer link bond between pair Enable CLAG peering between leaf switches Configure Each Switch By default, a switch with Cumulus Linux freshly installed has no switch port interfaces defined. Define the basic characteristics of swp1 through swpn by creating stanza entries for each switch port (swp) in the /etc/network/interfaces file. Required statements include the following: auto <switch port name> iface <switch port name> 20

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX An MTU setting of 9000 is recommended to avoid packet fragmentation. For example: cumulus@leaf01$ sudo vi /etc/network/interfaces # physical interface configuration auto swp1 iface swp1 mtu 9000 auto swp2 iface swp2 mtu 9000 auto swp3 iface swp3 mtu 9000... auto swp52 iface swp52 mtu 9000 Additional attributes such as speed and duplex can be set. Refer to the Persistent Configuration section of the Understanding Network Interfaces chapter of the Cumulus Linux Documentation for more information. Configure all leaf switches identically. Instead of manually configuring each interface definition, you can programmatically define them by using shorthand syntax that leverages Python Mako templates. Refer to the Automation Considerations chapter found later in this guide. Once all configurations have been defined in the /etc/network/interfaces file, run the ifquery command to ensure that all syntax is proper and the interfaces are created as expected: cumulus@leaf01$ ifquery -a auto lo iface lo inet loopback auto eth0 iface eth0 address 192.168.0.90/24 gateway 192.168.0.254 auto swp1 iface swp1 mtu 9000... www.cumulusnetworks.com 21

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Once all configurations have been defined in the /etc/network/interfaces file, apply the configurations to ensure they are loaded into the kernel. There are several methods for applying configuration changes depending on when and what changes you want to apply. Command sudo ifreload -a sudo service networking restart sudo ifup <swp> Action Parse interfaces labelled with auto that have been added to or modified in the configuration file, and apply changes accordingly. Apply changes to existing interfaces labelled with auto in the configuration file that have been modified. Note: This command is disruptive to traffic only on interfaces that have been modified. Restart all interfaces labelled with auto as defined in the configuration file, regardless of what has or has not been recently modified. Note: This command is disruptive to all traffic on the switch including the eth0 management network. Parse an individual interface labelled with auto as defined in the configuration file and apply changes accordingly. Note: This command is disruptive to traffic only on interface swpx. For example, on leaf01: or individually: cumulus@leaf01:~$ sudo ifreload -a cumulus@leaf01:~$ sudo ifup swp1 cumulus@leaf01:~$ sudo ifup swp2... cumulus@leaf01:~$ sudo ifup swp52 22

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX Create Peer Link Bond between Switches Next, create a peer link bond on both switches by editing /etc/network/interfaces and placing the bond configuration after the swpn interfaces. Configure the peer link bond identically on both switches in the CLAG pair. For example, add the following peerlink stanza on leaf01 with bond members swp47 and swp48. The configuration will be identical on leaf02. For more information on bond settings, refer to the CLAG chapter in the Cumulus Linux Documentation. cumulus@leaf01$ sudo vi /etc/network/interfaces # peerlink bond for clag auto peerlink iface peerlink bond-slaves swp47 swp48 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 Enable CLAG Peering between Switches An instance of the clagd daemon runs on each CLAG switch member to keep track of various networking information, including MAC addresses, needed to maintain the peer relationship. clagd communicates with its peer on the other switch across a Layer 3 interface between the two switches. This Layer 3 network should not be advertised by routing protocols, nor should the VLAN be trunked anywhere else in the network. This interface is designed to be a keep-alive reachability test and for synchronizing the switch state across the directly attached peer bond. Create the VLAN subinterface for clagd communication and assign an IP address for this subinterface. A unique.1q tag is recommended to avoid mixing data traffic with the clagd control traffic. To enable CLAG peering between switches, configure clagd on each switch by creating a peerlink subinterface in /etc/network/interfaces with a unique.1q tag. Set values for the following parameters under the peerlink subinterface: address. The local IP address/netmask of this peer switch. o Recommended to use a link local address for example 169.254.1.X/30 clagd-enable. Set to yes (default) clagd-peer-ip. Set to the IP address assigned to the peer interface on the peer switch. clagd-sys-mac. Set to a unique MAC address you assign to both peer switches. o Recommended within the Cumulus Networks reserved range of 44:38:39:FF:00:00 through 44:38:39:FF:FF:FF. www.cumulusnetworks.com 23

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE For example, configure leaf01 and leaf02 as follows: cumulus@leaf01$ sudo vi /etc/network/interfaces # VLAN for clagd communication auto peerlink.4094 iface peerlink.4094 address 169.254.1.1/30 clagd-enable yes clagd-peer-ip 169.254.1.2 clagd-sys-mac 44:38:39:ff:40:94 cumulus@leaf02$ sudo vi /etc/network/interfaces # VLAN for clagd communication auto peerlink.4094 iface peerlink.4094 address 169.254.1.2/30 clagd-enable yes clagd-peer-ip 169.254.1.1 clagd-sys-mac 44:38:39:ff:40:94 Note: CLAG can use any valid IP address pair for communication; however, we suggest using values from the IPv4 link local range defined by 169.254.0.0/16. These addresses are not exported by routing protocols and, since the peer communication VLAN is local to this peer link, the same IP address pairs can be used on all CLAG switch pairs. Because MTU configurations on a subinterface are inherited from the parent interface and the parent interface (peerlink) was previously defined with the MTU setting, there is no need to set MTU in the subinterface stanza. Reload the network configuration for the CLAG changes to be applied and start clagd: On leaf01: On leaf02: cumulus@leaf01:~$ sudo ifreload -a cumulus@leaf02:~$ sudo ifreload -a or individually restart just the peerlink and subinterface to apply the CLAG changes and start clagd: On leaf01: On leaf02: cumulus@leaf01:~$ sudo ifup peerlink cumulus@leaf01:~$ sudo ifup peerlink.4094 cumulus@leaf02:~$ sudo ifup peerlink cumulus@leaf02:~$ sudo ifup peerlink.4094 24

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX Once clagd is configured under the peerlink subinterface, it will automatically start when the system is booted. Once the interfaces have been started, verify the interfaces are up and have the proper IP addresses assigned. cumulus@leaf01:~$ ip addr show peerlink.4094 115: peerlink.4094@peerlink: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP link/ether c4:54:44:9f:49:1e brd ff:ff:ff:ff:ff:ff inet 169.254.1.1/30 scope global peerlink.4094 inet6 fe80::7272:cfff:fe9d:4e64/64 scope link valid_lft forever preferred_lft forever cumulus@leaf02$ ip addr show peerlink.4094 105: peerlink.4094@peerlink: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP link/ether c4:54:44:bd:00:20 brd ff:ff:ff:ff:ff:ff inet 169.254.1.2/30 scope global peerlink.4094 inet6 fe80::c654:44ff:fe9f:491e/64 scope link valid_lft forever preferred_lft forever Next, verify connectivity between the switches by issuing a ping across the peer bond from each switch to its peer switch. For example, from leaf01 you should be able to ping the IP address of the clagd subinterface on leaf02, 169.254.1.2. cumulus@leaf01:~$ ping 169.254.1.2 PING 169.254.1.2 (169.254.1.2) 56(84) bytes of data. 64 bytes from 169.254.1.2: icmp_req=1 ttl=64 time=0.798 ms 64 bytes from 169.254.1.2: icmp_req=2 ttl=64 time=0.554 ms --- 169.254.1.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.554/0.676/0.798/0.122 ms Likewise, from leaf02 you should be able to ping the IP address of the clagd subinterface on leaf01, 169.254.1.1. Verify CLAG port channel operation and the peer roles have been established. In the example below, switch leaf01 is operating as the secondary peer switch while leaf02 is the primary peer switch. By default, priority values for both switches are equal and set to 32768. The switch with the lower MAC address is assigned the primary role in the instance of a priority tie. cumulus@leaf01$ clagctl The peer is alive Our Priority, ID, and Role: 32768 c4:54:44:9f:49:1e primary Peer Priority, ID, and Role: 32768 c4:54:44:bd:00:20 secondary Peer Interface and IP: peerlink.4094 169.254.1.2 System MAC: 44:38:39:ff:40:94 When a CLAG-enabled switch is in the secondary role, it does not send BPDUs on dual-connected links; it only sends BPDUs on single-connected links. Also, in case the peer switch is determined to be not alive, the switch in the secondary role will roll back the link MAC address to be the bond interface MAC address instead of the clagd-sys-mac. By contrast, the switch in the primary role always uses the clagd-sys-mac and sends BPDUs on all single- and dual-connected links. The role of primary vs. secondary peer switch becomes important to consider when restarting switches. If a secondary peer switch is restarted, the LACP system ID remains the same. However, if a primary peer switch is restarted, the LACP system ID will change, which can be disruptive. www.cumulusnetworks.com 25

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Changing the priority does not cause a traffic interruption but will take a few seconds to switch over while the switch waits for the next peer update. To change the priority for leaf02 so that it becomes the primary and leaf01 becomes secondary, use the clagctl command with the priority parameter: cumulus@leaf01$ sudo clagctl priority 4096 cumulus@leaf02$ clagctl The peer is alive Our Priority, ID, and Role: 4096 c4:54:44:bd:00:20 primary Peer Priority, ID, and Role: 32768 c4:54:44:9f:49:1e secondary Peer Interface and IP: peerlink.4094 169.254.1.1 System MAC: 44:38:39:ff:40:94 3. Configure Spine Switches Configure each switch in pair individually Create peer link bond between pair Enable CLAG peering between spine switches Configure Each Switch As in the case of the leaf switches, define all the switch ports on your spine switches that will be in use. A typical spine switch that is fully populated has swp1 through 32 defined. Define a port by creating stanza entries for each switch port (swp) in the /etc/network/interfaces file. For example: cumulus@spine01$ sudo vi /etc/network/interfaces # physical interface configuration auto swp1 iface swp1 mtu 9000 auto swp2 iface swp2 mtu 9000 auto swp3 iface swp3 mtu 9000... auto swp32 iface swp32 mtu 9000 Configure both spine switches identically. Instead of manually configuring each interface definition, you can programmatically define them by using shorthand syntax that leverages Python Mako templates. Refer to the Automation Considerations chapter found later in this guide. 26

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX As you did previously with the leaf switches, Use sudo ifquery -a to verify all interfaces were properly defined in the configuration file. Bring up the interfaces using sudo ifreload -a or individually using sudo ifup swpn. Create Peer Link Bond between Switches Next, create a peer link bond on both spine switches in the same manner as previously done for the leaf switch pairs. Do this by editing the /etc/network/interfaces file and placing the bond configuration after the swpn interfaces. Configure the peer link bond identically on both switches in the CLAG pair. Add the following peerlink stanza on spine01, with swp31 and swp32 for bond members. The configuration will be identical on spine02. cumulus@spine01$ sudo vi /etc/network/interfaces # peerlink bond for clag auto peerlink iface peerlink bond-slaves swp31 swp32 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 Enable CLAG Peering between Switches To enable CLAG peering between switches, you configure the clagd daemon by adding the CLAG configuration parameters under the subinterface similarly as previously configured in the leaf switches. cumulus@spine01$ sudo vi /etc/network/interfaces # VLAN for clagd communication auto peerlink.4093 iface peerlink.4093 address 169.254.1.1/30 clagd-enable yes clagd-peer-ip 169.254.1.2 clagd-sys-mac 44:38:39:ff:40:93 cumulus@spine02$ sudo vi /etc/network/interfaces # VLAN for clagd communication auto peerlink.4093 iface peerlink.4093 address 169.254.1.2/30 clagd-enable yes clagd-peer-ip 169.254.1.1 clagd-sys-mac 44:38:39:ff:40:93 www.cumulusnetworks.com 27

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE We are using.4093 for the peerlink communication between spine01 and spine02 in contrast to.4094 between leaf01 and leaf02. Using different VLAN IDs for different peerlink communication links avoids the potential for creating an undesired loop. Next, reload the network configuration for the CLAG changes to be applied and start clagd: On spine01: On spine02: cumulus@spine01:~$ sudo ifreload -a cumulus@spine02:~$ sudo ifreload -a or individually restart just the peerlink and subinterface to apply the CLAG changes and start clagd: On spine01: On spine02: cumulus@spine01:~$ sudo ifup peerlink cumulus@spine01:~$ sudo ifup peerlink.4093 cumulus@spine02:~$ sudo ifup peerlink cumulus@spine02:~$ sudo ifup peerlink.4093 Once the interfaces have been started, verify the interfaces are up and have the proper IP addresses assigned. For example, on spine01: cumulus@spine01$ ip addr show peerlink.4093 36: peerlink.4093@peerlink: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP link/ether c4:54:44:72:cf:ad brd ff:ff:ff:ff:ff:ff inet 169.254.1.1/30 scope global peerlink.4093 inet6 fe80::c654:44ff:fe72:cfad/64 scope link valid_lft forever preferred_lft forever Next, verify connectivity between the switches by issuing a ping across the peer bond from each switch to its peer switch. For example, from spine01 you should be able to ping the IP address of the clagd subinterface on spine02, 169.254.1.2. cumulus@spine01:~$ ping 169.254.1.2 PING 169.254.1.2 (169.254.1.2) 56(84) bytes of data. 64 bytes from 169.254.1.2: icmp_req=1 ttl=64 time=0.725 ms 64 bytes from 169.254.1.2: icmp_req=2 ttl=64 time=0.916 ms --- 169.254.1.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.725/0.820/0.916/0.099 ms Likewise, from spine02 you should be able to ping the IP address of the clagd subinterface on spine01, 169.254.1.1. Verify CLAG port channel operation and the peer roles have been established. In the following example, spine01 is operating as the primary peer switch and spine02 is the secondary peer switch: cumulus@spine01$ clagctl The peer is alive Our Priority, ID, and Role: 32768 c4:54:44:72:cf:ad primary Peer Priority, ID, and Role: 32768 c4:54:44:72:dd:c9 secondary Peer Interface and IP: peerlink.4093 169.254.1.2 System MAC: 44:38:39:ff:40:93 28

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX 4. Set Up Spine/Leaf Network Fabric Create a switch-to-switch uplink bond on each leaf switch to the spine pair and verify CLAG Create a switch-to-switch bond on the spine pair to each leaf switch pair and verify CLAG Creating Switch-to-Switch Spine Bond on Leaf Switches Now that the peer relationship has been established on the leaf and spine switches, create the switch-to-switch bonds on the leaf switches by editing the /etc/network/interfaces file and placing the bond configuration after the swpn interfaces. You must specify a unique clag-id for every dual-connected bond on each peer switch; the value must be between 1 and 65535 and must be the same on both peer switches in order for the bond to be considered dual-connected. For the uplink1 bond the clag-id is set to 1000 and the host bond clag-ids start at 1. cumulus@leaf01$ sudo vi /etc/network/interfaces # uplink bond to spine auto uplink1 iface uplink1 bond-slaves swp49 swp50 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 1000 Configure leaf02 identically as leaf01. Once these interfaces have been created, apply the configuration by using the ifreload command or individually bringing up the new interfaces on both switches: On leaf01: On leaf02: or individually: On leaf01: On leaf02: cumulus@leaf01:~$ sudo ifreload -a cumulus@leaf02:~$ sudo ifreload -a cumulus@leaf01:~$ sudo ifup uplink1 cumulus@leaf02:~$ sudo ifup uplink1 www.cumulusnetworks.com 29

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Verify switch-to-switch CLAG port channel operation on both leaf switches using the clagctl command. On leaf01: cumulus@leaf01$ clagctl The peer is alive Peer Priority, ID, and Role: 4096 c4:54:44:bd:00:20 primary Our Priority, ID, and Role: 32768 c4:54:44:9f:49:1e secondary Peer Interface and IP: peerlink.4094 169.254.1.2 System MAC: 44:38:39:ff:40:94 Creating Switch-to-Switch CLAG Bond on Spine Pair to Each Leaf Switch Pair Create the switch-to-switch CLAG bonds on the spine switches by editing the /etc/network/interfaces file and placing the bond configuration after the swpn interfaces. For example, on spine01, create the downlink1 to aggregate traffic between the leaf01 and leaf02 pair and the leaf03 and leaf04 pair. The clag-id on both switches for the downlink1 bond is set to 1 and for the downlink2 bond set to 2. cumulus@spine01$ sudo vi /etc/network/interfaces # leaf01-leaf02 downlink auto downlink1 iface downlink1 bond-slaves swp1 swp2 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 1 # leaf03-leaf04 downlink auto downlink2 iface downlink2 bond-slaves swp3 swp4 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 2 Configure spine02 identically to spine01. Once these interfaces have been created, apply the configuration by using the ifreload command or individually bringing up the new interfaces on both switches: On spine01: cumulus@spine01:~$ sudo ifreload a 30

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX On spine02: or individually: On spine01: On spine02: cumulus@spine02:~$ sudo ifreload -a cumulus@spine01:~$ sudo ifup downlink1 cumulus@spine02:~$ sudo ifup downlink1 Verify switch-to-switch CLAG port channel operation on both spine switches using the clagctl command. On spine01: cumulus@spine01$ clagctl The peer is alive Our Priority, ID, and Role: 32768 c4:54:44:72:cf:ad primary Peer Priority, ID, and Role: 32768 c4:54:44:72:dd:c9 secondary Peer Interface and IP: peerlink.4093 169.254.1.2 System MAC: 44:38:39:ff:40:93 Dual Attached Ports Our Interface Peer Interface CLAG Id ---------------- ---------------- ------- downlink1 downlink1 1 Verify the CLAG connection by running clagctl on the leaf switches. cumulus@leaf01$ clagctl The peer is alive Peer Priority, ID, and Role: 4096 c4:54:44:bd:00:20 primary Our Priority, ID, and Role: 32768 c4:54:44:9f:49:1e secondary Peer Interface and IP: peerlink.4094 169.254.1.2 System MAC: 44:38:39:ff:40:94 Our Interface Peer Interface CLAG Id ---------------- ---------------- ------- uplink1 uplink1 1000 5. Configure VLANs Create the VLANs expected for your traffic. For example: VLAN 10: in-band VMkernel for vmotion VLAN 15: in-band VMkernel for Virtual SAN storage VLAN 20-23: VM traffic for WWW services VLAN 30: VM traffic for application 1 data VLAN 40: VM traffic for application 2 data To support VLANs in Cumulus Linux 2.5, a single bridge must be created in /etc/network/interfaces. www.cumulusnetworks.com 31

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Create the VLAN-aware bridge on all spine and leaf switches. For example, on leaf01: cumulus@leaf01$ sudo vi /etc/network/interfaces auto bridge iface bridge bridge-vlan-aware yes bridge-ports peerlink uplink1 bridge-vids 10,15,20,21,22,23,30,40,1000-2000 bridge-pvid 1 bridge-stp on This stanza defines a VLAN-aware bridge for high VLAN scale and assigns the infrastructure ports used for Layer 2 links to the bridge, and assign VLANs to the network infrastructure. This list of VLAN IDs is inherited by all Layer 2 interfaces in the bridge, unless different values are specified under an interface. In this configuration, all VLAN IDs are trunked to all Layer 2 interfaces and bonds. The untagged or native VLAN for the infrastructure ports is defined by bridge-pvid; if one is not specified, the default value is VLAN ID 1. Setting the ID to 1 is a best practice for spanning tree switch interoperability. Finally, the stanza enables spanning tree on the bridge. To verify spanning tree operation on the bridge, use the mstpctl command. The following example shows the spanning tree port information for the uplink interface cumulus@leaf01$ mstpctl showportdetail bridge uplink1 bridge:uplink CIST info enabled yes role Root port id 8.006 state forwarding external port cost 305 admin external cost 0 internal port cost 305 admin internal cost 0 designated root 1.000.44:38:39:FF:00:00 dsgn external cost 305 dsgn regional root 1.000.44:38:39:FF:77:00 dsgn internal cost 0 designated bridge 1.000.44:38:39:FF:77:00 designated port 8.001 admin edge port no auto edge port yes oper edge port no topology change ack no point-to-point yes admin point-to-point auto restricted role no restricted TCN no port hello time 2 disputed yes bpdu guard port no bpdu guard error no network port no BA inconsistent no Num TX BPDU 31237 Num TX TCN 11 Num RX BPDU 38123 Num RX TCN 119 Num Transition FWD 4 Num Transition BLK 4 bpdufilter port no clag ISL no clag ISL Oper UP no clag role primary clag dual conn mac 44:38:39:ff:77:0 clag remote portid F.FFF clag system mac 44:38:39:ff:40:94 32

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX Finally, to verify the VLAN assignment, use the bridge vlan show command for example on spine01: cumulus@spine01$ bridge vlan show port vlan ids peerlink 1 PVID Egress Untagged 10 15 20-23 30 40 1000-2000 downlink1 1 PVID Egress Untagged 10 15 20-23 30 40 1000-2000 downlink2 1 PVID Egress Untagged 10 15 20-23 30 40 1000-2000 To verify the interfaces with which a specific VLAN is associated, for example VLAN 10, use the bridge vlan show vlan command: cumulus@spine01$ bridge vlan show vlan 10 VLAN 10: peerlink downlink1 downlink2 6. Connect vsphere ESXi Hosts Provision vsphere ESXi hosts Connect hosts to leaf switches using virtual switches Provision the vsphere ESXi hosts, if you have not already. Refer to the vsphere Installation and Setup Guide for detailed steps. This design guide assumes that imaging and basic configuration of the ESXi hosts occurs over an out-of-band management network. Since the data center networking fabric has already been set up, it s time to connect the vsphere ESXi hosts to the leaf switches. To improve network reliability and optimization, use host link redundancy and aggregation. Set up LACP on your vsphere Distributed Switch (vds). Refer to Enable Link Layer Discovery Protocol on a vsphere Distributed Switch in the VMware vsphere Documentation Center. See also the VMware vsphere Distributed Switch Best Practices white paper. On each leaf switch, create the interface and assign the ESXi host-facing ports to that VLAN. www.cumulusnetworks.com 33

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE To enable LACP, first create a new host bond interface. This is a single interface in a bond interface configured on both switches in a CLAG-pair of leaf switches. For example, create the following bond on leaf01. Make the identical configuration on leaf02. cumulus@leaf01$ sudo vi /etc/network/interfaces auto esx-host-01 iface esx-host-01 bond-slaves swp1 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 1 Once the host bond is created, add the bond to the VLAN-aware bridge to enable VLAN trunking to the host. cumulus@leaf01$ sudo vi /etc/network/interfaces auto bridge iface bridge bridge-ports peerlink uplink1 esx-host-01 By default, the list of VLANs is inherited on the esx-host-01 interface. To override this (to prune VLANs 1000-2000 for example), configure the allowed VLANs on the esx-host-01 interface without the 1000-2000 from the global configuration as follows: cumulus@leaf01$ sudo vi /etc/network/interfaces auto esx-host-01 iface esx-host-01 bridge-vids 10,15,20,21,22,23,30,40 # optional bridge-pvid 1 mstpctl-bpduguard yes Optional: Set the untagged or native VLAN for the host bond if changing from the default VLAN ID of 1 or changing from the global value set under the VLAN-aware bridge. Optional: It can be a best practice to enable BPDU guard on all the host-facing ports and bonds. Doing so prevents the accidental connection of a switch into the network on a host port; in the event a BPDU is unintentionally received, BPDU guard disables that port. To verify that BPDU guard has been enabled on a port, use the mstpctl command and look for bpdu information: cumulus@leaf01$ mstpctl showportdetail bridge esx-host-01 grep bpdu bpdu guard port yes bpdu guard error no bpdufilter port no 34

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX Optional: By default, the list of VLANs is inherited on the esx-host-01 interface. If the host only connects to a single VLAN, for example VLAN 10, instead of a trunk set the port to an access port as follows: cumulus@leaf01$ sudo vi /etc/network/interfaces auto esx-host-01 iface esx-host-01 bridge-access 10 7. Configure Virtual Networking Create VM and VMkernel networks using configured virtual switches Use virtual networks As a final step, create virtual networks as you would for VM, storage, and vsphere services traffic. Refer to the vsphere Installation and Setup Guide for detailed steps. For example, go back to vcenter and add Networking, and choose either VMkernel Network Adapter for VMkernel traffic or Virtual Machine Port Group for a Standard vswitch for a network for your VMs to connect to. 8. Connect Spine Switches to Core Method 1. External Connectivity at Layer 2 The following steps assume the spine switches connect to core switches at Layer 2. Create a CLAG port channel to the core for each spine switch On each core device, create a vendor-equivalent Multi-Chassis Link Aggregation bond to the spine switch pair Configure the core Depending on your core devices support for an MLAG-type configuration, you may need to configure two separate bonds if your core switches do not support a multi-chassis LACP solution. If your core devices support MLAG or the equivalent, create a single switch-to-switch CLAG bond on the spine switches by editing the /etc/network/interfaces file and placing the bond configuration after the swpn interfaces. For example, on spine01, create the core-uplink interface to aggregate traffic between spine01 and spine02 to the core. The clag-id for the core-uplink bond is set to 1000 on both spine switches. cumulus@spine01$ sudo vi /etc/network/interfaces # bond to core auto core-uplink iface core-uplink bond-slaves swp31 swp32 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 1000 Configure spine02 identically to spine01 to aggregate traffic between spine01 and spine02 into a single CLAG bond. www.cumulusnetworks.com 35

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Once these interfaces have been created, apply the configuration by using the ifreload command or individually bringing up the new interfaces on both switches: On spine01: On spine02: or individually: On spine01: On spine02: cumulus@spine01:~$ sudo ifreload -a cumulus@spine02:~$ sudo ifreload -a cumulus@spine01:~$ sudo ifup core-uplink cumulus@spine02:~$ sudo ifup core-uplink Verify switch-to-switch CLAG port channel operation on both spine switches using the clagctl command. On spine01: cumulus@spine01$ clagctl The peer is alive Our Priority, ID, and Role: 32768 c4:54:44:72:cf:ad primary Peer Priority, ID, and Role: 32768 c4:54:44:72:dd:c9 secondary Peer Interface and IP: peerlink.4093 169.254.1.2 System MAC: 44:38:39:ff:40:93 Dual Attached Ports Our Interface Peer Interface CLAG Id ---------------- ---------------- ------- downlink1 downlink1 1 downlink2 downlink2 2 core-uplink core-uplink 1000 Method 2. External Connectivity at Layer 3 To provide a Layer 3 gateway for a VLAN, use the first hop redundancy protocol, Virtual Router Redundancy (VRR), provided in Cumulus Linux. VRR provides Layer 3 redundancy by using the same virtual IP and MAC addresses on each switch, allowing traffic across a CLAG to be forwarded, regardless of which switch the traffic arrived on. In this configuration the switch pair work in an active/active capacity. VRR also works in a non-clag environment where a host is in an active/active or active/standby role. For more information, refer to the Virtual Router Redundancy (VRR) chapter in the Cumulus Linux Documentation. The following steps assume the spine switches connect to core switches at Layer 3. Configure Layer 3 switch virtual interface (SVI) gateways and connect the spine switches to core switches. Configure core-facing interfaces for IP transfer Configure dynamic routing protocol To enable the gateway, first create the Layer 3 virtual interface for that VLAN on the VLAN-aware bridge. 36

BUILDING A DATA CENTER NETWORK WITH VMWARE VSPHERE AND CUMULUS LINUX For example, to configure this on VLAN 10, add the following stanza to the network configuration: cumulus@spine01$ sudo vi /etc/network/interfaces auto bridge.10 iface bridge.10 address 10.1.10.2/24 address-virtual 00:00:5E:00:01:01 10.1.10.1 cumulus@spine02$ sudo vi /etc/network/interfaces auto bridge.10 iface bridge.10 address 10.1.10.3/24 address-virtual 00:00:5E:00:01:01 10.1.10.1 This stanza defines a routable interface to VLAN 10 and assigns an IP address to the interface. If, for example, your gateway is the first IP address in the subnet, such as 10.1.10.1, you should assign the actual interface IP address to 10.1.10.2. On spine02, make sure that the base IP address is unique, such as 10.1.10.3. In this example configuration, address-virtual creates a virtual interface with the address of 10.1.10.1, assigned to the bridge VLAN ID 10, with virtual MAC 00:00:5E:00:01:01. These virtual IP and MAC addresses will be used between the pair of switches for load balancing and failover. The MAC address is in the VRRP MAC address range of 00:00:5E:00:01:XX and does not overlap with other MAC addresses in the network. For each desired gateway VLAN, replicate the above configuration, changing the IP addressing to match the subnet and the bridge.n where N is the VLAN ID. To verify the virtual address has been created, first check the bridge.10 interface: cumulus@spine01$ ip addr show bridge.10 53: bridge.10@bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP link/ether c4:54:44:72:cf:35 brd ff:ff:ff:ff:ff:ff inet 10.1.10.2/24 scope global bridge.10 inet6 fe80::c654:44ff:fe72:cf35/64 scope link valid_lft forever preferred_lft forever When the address-virtual keyword is put under a Layer3 bridge ID it automatically creates a virtual interface with the syntax: (bridge name)-(vlan ID)-v(virtual instance). In the above example, the = bridge name is bridge, the VLAN ID is 10, and the virtual instance is 0. Thus, the interface, bridge-10-v0, will have the virtual MAC address and virtual IP address assigned under the bridge.10 interface. If additional virtual addresses are added to the interface, each will have its own instance. To see that the virtual interface is operational, use the ip addr show command. cumulus@spine01$ ip addr show bridge-10-v0 77: bridge-10-v0@bridge.10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff inet 10.1.10.1/32 scope global bridge-10-v0 inet6 fe80::200:5eff:fe00:101/64 scope link valid_lft forever preferred_lft forever www.cumulusnetworks.com 37

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Automation Considerations Automation and Converged Administration Because Cumulus Linux is Linux, you can draw from a rich ecosystem of solutions to automate the provisioning and management of your switches. Automation tools can range from Linux-based scripts to applications that run natively in the Linux OS. In order to have distinct organizational roles and allow both server and network teams to focus on their areas of expertise and responsibility, yet have a ubiquitous form of centralized management, you can leverage tools such as Puppet, Chef, Ansible, and others that are widely used for compute management. Additionally, a demo of the VMware vsphere Validated Design in the Cumulus Workbench is available as a Knowledge Base article at support.cumulusnetworks.com/hc/en- us/articles/203567318. The following sections provide automation and template examples using zero touch provisioning, Mako, Ansible, and Puppet. Additional examples are available in the form of demos in the Cumulus Networks Knowledge Base under Demos and Training at support.cumulusnetworks.com/hc/en-us/sections/200398866; the source code for the demos can be found in the Cumulus Networks GitHub repository at: github.com/cumulusnetworks/. Automated Switch Provisioning with Zero Touch Provisioning By default, a switch with Cumulus Linux freshly installed has the option to look for and invoke an automation script. This process is called zero touch provisioning and is triggered by the following conditions: Management port (eth0) configured to DHCP Management port is restarted, or switch is powered on, using the one of the following commands: o service networking restart o reboot o ifdown and ifup the switch port DHCP server is configured with option cumulus-provision-url code 239 DHCP server is configured with URL of automation script to execute Alternatively, zero touch provisioning can be run manually by running /usr/lib/cumulus/autoprovision. For example: cumulus@switch$ sudo /usr/lib/cumulus/autoprovision -u http://10.99.0.1/script.sh More details on the autoprovision command may be obtained by running the command with the -h option. Zero touch provisioning will run a script using the specified URL provided by DCHP or manually. Supported languages include Bash, Ruby, Python, and Perl. As a failsafe mechanism, zero touch provisioning will look for a CumulusLinux- AutoProvision flag in the HTTP header when retrieving the script prior to executing it. The script can automate many provisioning functions, such as: Install the Cumulus Linux license Change the hostname Add to CMDB Run apt-get update Install an automation tool such as Puppet or Chef Create users or integrate with authentication Configure sudoers 38

AUTOMATION CONSIDERATIONS For example, a script that installs the Cumulus Linux license and SSH keys from a central server 10.99.0.1 is as follows: #!/bin/bash # CUMULUS-AUTOPROVISIONING #install license wget -q -O /root/license.txt http://10.99.0.1/license.txt /usr/cumulus/bin/cl-license -i /root/license.txt #install ssh keys /usr/bin/wget -O /root/.ssh/authorized_keys http://10.99.0.1/authorized_keys exit 0 For more information, refer to the Zero Touch Provisioning chapter of the Cumulus Linux Documentation. Network Configuration Templates Using Mako In the prior section Network Architecture Build Steps, we showed interface configurations that are manually entered in the /etc/network/interfaces file. Instead of manually configuring each interface definition, you can programmatically define them by using shorthand syntax that leverages Python Mako templates. You can use the following Mako template to represent what would take much longer to manually configure. For example, the following syntax can programmatically define the interface ports for the vsphere ESXi hosts. This Mako template: cumulus@leaf01$ sudo vi /etc/network/interfaces <% %> ESXi_host_ports = range(1,45) is equivalent to: % for i in: ESXi_host_ports auto swp${i} iface swp${i} mtu 9000 % endfor cumulus@leaf01$ sudo vi /etc/network/interfaces auto swp1 iface swp1 mtu 9000 auto swp2 iface swp2 mtu 9000 auto swp3 iface swp3 mtu 9000. www.cumulusnetworks.com 39

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE.. auto swp43 iface swp43 mtu 9000 auto swp44 iface swp44 mtu 9000 For more information and an example, see the KB article, Configuring /etc/network/interfaces with Mako, at: support.cumulusnetworks.com/hc/en-us/articles/202868023. Automated Network Configuration Using Ansible Ansible is an open source, lightweight configuration management tool that can be used to automate many configuration tasks. Ansible does not require an agent be run on a switch; instead, Ansible manages nodes over SSH. Using Ansible, you can run automation tasks across many end points, whereas you use Mako within the context of a single switch. A particular script that runs a variety of tasks is referred to as a playbook in Ansible. The following example changes the MTU for a group of switch ports using the example previously shown with Mako. On the controller, run the tree command to show where the playbook and related files reside: root@ubuntu# tree. ansible.cfg ansible.hosts roles vsphere tasks main.yml templates interfaces.j2 vars main.yml vsphere.yml The following files show how the playbook is run: root@ubuntu# cat ansible.cfg [defaults] host_key_checking=false hostfile = ansible.hosts The ansible.hosts file specifies switch1 and switch2 as the DNS names of two bare metal switches running Cumulus Linux: root@ubuntu# cat ansible.hosts [switches] switch1 40

AUTOMATION CONSIDERATIONS switch2 The vsphere.yml file is what is processed to run the playbook. It points to the roles that should be run. root@ubuntu# cat vsphere.yml --- - hosts: switches user: root roles: - vsphere The tasks/main.yml includes all the tasks being run; for example, to overwrite the /etc/network/interfaces file with the template file, roles/vsphere/templates/interfaces.j2. root@ubuntu# cat roles/vsphere/tasks/main.yml - name: configure interfaces template: src=interfaces.j2 dest=/etc/network/interfaces root@ubuntu# cat roles/vsphere/templates/interfaces.j2 auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp {% if switches[inventory_hostname].interfaces is defined -%} {% for item in switches[inventory_hostname].interfaces -%} auto {{ item }} iface {{ item }} mtu 9000 {% endfor -%} {% endif -%} {% if switches[inventory_hostname].start_port is defined -%} {% for item in range(switches[inventory_hostname].start_port int(),switches[inventory_hostname].stop_p ort int()) -%} auto swp{{ item }} iface swp{{ item }} mtu 9000 {% endfor -%} {% endif -%} www.cumulusnetworks.com 41

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE The vars/main.yml is the template from where values are retrieved: root@ubuntu# cat roles/vsphere/vars/main.yml switches: switch1: start_port: "1" stop_port: "44" switch2: interfaces: ["swp1", "swp2", "swp3", "swp4", "swp17", "swp18", "swp19", "swp20"] For switch1, a range similar to the Mako example is used, where switch ports 1 through 44 are set. switch2 has a different configuration where swp1 through swp4 are set, and then swp17 through swp20. To run the playbook, use the ansible-playbook command. The -k flag allows you to use a plaintext password rather than SSH keys. For example: root@ubuntu# ansible-playbook vsphere.yml -k SSH password: PLAY [switches] *************************************************************** GATHERING FACTS *************************************************************** ok: [switch2] ok: [switch1] TASK: [vsphere configure interfaces] **************************************** changed: [switch2] changed: [switch1] PLAY RECAP ******************************************************************** switch1 : ok=2 changed=1 unreachable=0 failed=0 switch2 : ok=2 changed=1 unreachable=0 failed=0 42

AUTOMATION CONSIDERATIONS Automated Network Configuration Using Puppet Puppet is an open source tool that can automate configuration management through the use of a controller that syncs with agents installed on each end point. While similar in functionality to Ansible, Puppet relies on agents installed on each switch being managed. Puppet utilizes TCP port 61613 for syncing between agents and a controller. In Puppet, a particular script that runs a variety of tasks is referred to as a manifest, similar to the idea of an Ansible playbook.the following example written in Puppet repeats the previous examples shown with Mako and Ansible. On the controller, run the tree command to show where the manifest and related directories and files reside: root@ubuntu# tree. auth.conf autosign.conf fileserver.conf manifests site.pp modules base manifests interfaces.pp role switch.pp templates interfaces.erb puppet.conf templates The following files show how the manifest is run. The site.pp file is the main manifest. It contains site-wide and node-specific statements or definitions, which are blocks of Puppet code that will only be included in a node s catalog information that is specific to that node. For example, this is where you specify which interfaces you want to change MTU to 9000: root@ubuntu# cat manifests/site.pp node 'switch2' { $int_enabled = true $int_mtu = { swp1 => {}, swp2 => {}, swp3 => {}, swp4 => {}, } include base::role::switch } www.cumulusnetworks.com 43

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE In this example, the module is simply called base and contains a single manifest, interfaces.pp, containing the class base::interfaces. Classes generally configure all the packages, configuration files, and services needed to run an application. root@ubuntu# cat modules/base/manifests/interfaces.pp class base::interfaces { if $int_enabled == undef { $int_enabled = false } } if ($int_enabled == true) { file { '/etc/network/interfaces': owner => root, group => root, mode => '0644', content => template('base/interfaces.erb') } service { 'networking': ensure => running, subscribe => File['/etc/network/interfaces'], hasrestart => true, restart => '/sbin/ifreload -a', enable => true, hasstatus => false, } } The interfaces.erb file is a template that fetches the variables from site.pp. The template keeps eth0 as DHCP, checks to see if int_mtu is defined in site.pp, then loops through each interface provided and sets MTU to 9000. root@ubuntu# cat modules/base/templates/interfaces.erb auto eth0 iface eth0 inet dhcp <% if @int_mtu %> # interfaces <% int_mtu.each_pair do key, value_hash %> auto <%= key %> iface <%= key %> mtu 9000 <% end %> <% else %> # no interfaces <% end %> 44

AUTOMATION CONSIDERATIONS The role is included in the main manifest. It ties all the manifests together that are utilized for this particular node. root@ubuntu# cat modules/base/manifests/role/switch.pp class base::role::switch { include base::interfaces } The puppet.conf file is the main configuration file on both the Puppet master and the Puppet client. On the Puppet master side, change the dns_alt_names value from the default, puppet: root@ubuntu# cat puppet.conf [main] logdir=/var/log/puppet vardir=/var/lib/puppet ssldir=/var/lib/puppet/ssl rundir=/var/run/puppet factpath=$vardir/lib/facter #modulepath=/etc/puppet/modules #templatedir=$confdir/templates dns_alt_names = puppet,cumulus-vm [master] # These are needed when the puppetmaster is run by passenger # and can safely be removed if webrick is used. ssl_client_header = SSL_CLIENT_S_DN ssl_client_verify_header = SSL_CLIENT_VERIFY On the Puppet client side, change the server value from the default, puppet: cumulus@switch2:~$ cat /etc/puppet/puppet.conf [main] logdir=/var/log/puppet vardir=/var/lib/puppet ssldir=/var/lib/puppet/ssl rundir=/var/run/puppet factpath=$vardir/lib/facter #templatedir=$confdir/templates server=cumulus [master] # These are needed when the puppetmaster is run by passenger # and can safely be removed if webrick is used. ssl_client_header = SSL_CLIENT_S_DN ssl_client_verify_header = SSL_CLIENT_VERIFY www.cumulusnetworks.com 45

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Puppet agents run every 30 minutes by default. You can manually force a Puppet agent to run using the --test option. For example: cumulus@switch2:~# sudo puppet agent --test Info: Retrieving pluginfacts Info: Retrieving plugin Info: Caching catalog for switch2 Info: Applying configuration version '1415654245' Notice: /Stage[main]/Base::Interfaces/File[/etc/network/interfaces]/content: --- /etc/network/interfaces 2014-11-10 22:39:55.000000000 +0000 +++ /tmp/puppet-file20141110-3198-b8i1bh 2014-11-10 22:40:20.714158235 +0000 @@ -3,4 +3,27 @@ +# interfaces + +auto swp1 +iface swp1 + mtu 9000 + +auto swp2 +iface swp2 + mtu 9000 + +auto swp3 +iface swp3 + mtu 9000 + +auto swp4 +iface swp4 + mtu 9000 + + Info: /Stage[main]/Base::Interfaces/File[/etc/network/interfaces]: Filebucketed /etc/network/interfaces to puppet with sum 6f8a42d7ebd62f41c19324868384e095 Notice: /Stage[main]/Base::Interfaces/File[/etc/network/interfaces]/content: content changed '{md5}6f8a42d7ebd62f41c19324868384e095' to '{md5}ebf607a6ab09b595e81d1ff63e4b1196' Info: /Stage[main]/Base::Interfaces/File[/etc/network/interfaces]: Scheduling refresh of Service[networking] Notice: /Stage[main]/Base::Interfaces/Service[networking]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Base::Interfaces/Service[networking]: Unscheduling refresh on Service[networking] Notice: Finished catalog run in 6.99 seconds 46

OPERATIONAL AND MANAGEMENT CONSIDERATIONS Operational and Management Considerations Authentication and Authorization vsphere ESXi hosts have local root and user accounts. Remote SSH login as root is disabled by default. This is for standalone purposes. In a typical data center, authentication is managed through vcenter Server. vcenter Server has a single-sign on (SSO) feature that coordinates access to multiple VMware products. SSO can integrate with OpenLDAP. For more information, refer to the KB article, VMware vcenter Single Sign-on Server 5.5 FAQs (2057799), kb.vmware.com/kb/2057799. Cumulus Linux switches can be configured to use OpenLDAP v2.4 or later. Roles are used to segment privileges No Access, Read Only, Administrator, Custom. In Cumulus Linux, users belonging to sudo group are equivalent to Administrator. Users created that are not part of the sudo group can be created and used for read-only access. Refer to the Authentication, Authorization, and Accounting section of the System Management and Diagnostics chapter of the Cumulus Linux Documentation for more information. Security Hardening From a security hardening perspective, the following ports are used by vsphere and Cumulus Linux. Ports needed for vsphere host management: TCP 443 and 902: vsphere Client, vcenter Agent UDP 53: DNS client TCP and UDP 427: CIM Service Location Protocol (SLP) TCP 8000: vmotion TCP 22: SSH Ports needed for Cumulus Linux switch management: TCP 22: SSH TCP 23: Telnet Ports needed for CLAG communication between Cumulus Linux switches: TCP 5342 (default port) Refer to the vsphere Security Hardening guides at www.vmware.com/security/hardening-guides for further information. www.cumulusnetworks.com 47

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Accounting and Monitoring ESXi host logging uses syslog. vsphere includes syslog collectors. Refer to the KB article, Configuring syslog on ESXi 5.x (2003322), kb.vmware.com/kb/2003322/ for further information. Cumulus Linux log files are written to the /var/log directory. Key log files include: Cumulus Linux Log File syslog daemon.log quagga/zebra.log quagga/{protocol}.log switchd.log clagd.log Description This log file shows all issues from the kernel, Cumulus Linux HAL process (switchd), and issues from almost all other application processes, such as DHCP, smond (look for facility and INFO entries), and so forth. Details when processes start and stop in the system. Details issues from the Quagga zebra daemon. Details issues from Layer 3 routing protocols, like OSPF or BGP. Logs activity on the switch monitored by the switchd process. Logs activity of the clagd daemon for CLAG. Quality of Service (QoS) Considerations The use of industry-standard switches allows for unprecedented lower hardware costs and the ability to add additional bandwidth and hot spares. Thus, QoS becomes less important, and you do not have to think about QoS in the traditional sense. Instead, make sure to provision sufficient bandwidth to minimize oversubscription. When applications begin choking due to bandwidth restrictions, it is much more productive to increase total bandwidth rather than tweak QoS to starve certain traffic. Since Cumulus Linux provides hardware choice, which has reduced pricing of switches significantly, the problem of expensive bandwidth is no longer a problem in today s data center. Cumulus Networks strongly recommends not changing the default buffer and queue management values, as they are optimized for typical environments. However, Cumulus Networks realizes that many situations are unique and require further tuning based on the requirements of a particular network. To configure buffer and queue Management, Cumulus Linux divides inbound packets into four different priority groups. Cumulus Linux by default divides traffic into priority groups based on their Class of Service (CoS) values, rather than Differentiated Services Code Point (DSCP). Priority Group Description Default CoS Values Control Highest priority 7 Service Second highest priority traffic 2 Lossless Traffic protected by priority flow control none Bulk All remaining traffic 0, 1, 3, 4, 5, 6 To change the default behavior, edit the /etc/cumulus/datapath/traffic.conf file: cumulus@switch$ sudo vi /etc/cumulus/datapath/traffic.conf To specify to which queue a CoS value maps, edit the priority_group.<name>.cos_list values. For example: 48

OPERATIONAL AND MANAGEMENT CONSIDERATIONS priority_group.control.cos_list = [1,3] To change from CoS to DSCP, change the value of traffic.packet_priority_source, keeping in mind DSCP values must be mapped to each of the priority groups. traffic.packet_priority_source = dscp Changes to traffic.conf require you to run service switchd restart before they take effect. Note: this command is disruptive to all switch port interfaces. cumulus@switch$ sudo service switchd restart Refer to the Configuring Buffer and Queue Management chapter of the Cumulus Linux Documentation for further information. Link Pause The pause frame is a flow control mechanism that halts transmission for a specified period of time. For example, a server or other network node within the data center may be receiving traffic faster than it can handle and thus may benefit from using the pause frame. In Cumulus Linux, individual ports can be configured to execute link pause by: Transmitting pause frames when its ingress buffers become congested (TX pause enable) and/or Responding to received pause frames (RX pause enable) You can configure link pause by editing /etc/cumulus/datapath/traffic.conf. For example, to enable both types of link pause for swp2 and swp3: # To configure pause on a group of ports: # uncomment the link pause port group list # add or replace a port group name to the list # populate the port set, e.g. # swp1-swp4,swp8,swp50s0-swp50s3 # enable pause frame transmit and/or pause frame receive # link pause link_pause.port_group_list = [port_group_0] link_pause.port_group_0.port_set = swp2-swp3 link_pause.port_group_0.rx_enable = true link_pause.port_group_0.tx_enable = true A port group refers to one or more sequences of contiguous ports. Multiple port groups can be defined by: Adding a comma-separated list of port group names to the port_group_list. Adding the port_set, rx_enable, and tx_enable configuration lines for each port group. You can specify the set of ports in a port group in comma-separated sequences of contiguous ports. The syntax supports: A single switch port, like swp1s0 or swp5 A range of regular switch ports, like swp2-swp5 A sequence within a breakout switch port, like swp6s0-swp6s3 It does not accept a sequence that includes multiple split ports; for example, swp6s0-swp7s3 is not supported. Restart switchd to allow your link pause configuration changes to take into effect. Note: this command is disruptive to all switch port interfaces. cumulus@switch$ sudo service switchd restart www.cumulusnetworks.com 49

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Conclusion Summary The fundamental abstraction of hardware from software and providing customers a choice through a hardware agnostic approach is core to the philosophy of Cumulus Networks and fits very well within the software-defined data center approach of VMware through a shared vision. Just as VMware customers have choice in server compute and storage, VMware customers can tap the power of Open Networking and select from a broad range of switch providers running Cumulus Linux. Choice and CapEx savings are only the beginning. OpEx savings come from agility through automation. Just as VMware vsphere powers the software-defined data center by enabling the automated provisioning of hosts, networks, and VMs through the use of templates and profiles, Cumulus Linux enables network and data center architects to leverage automated provisioning tools and templates to define and provision networks. References Article/Document vsphere Documentation vsphere Installation and Setup Guide vsphere Networking Guide vsphere Storage Guide VMware KB Articles 2034277: Enabling LACP 2057799: vcenter Single Sign-on 2003322: Configuring syslog on ESXi vsphere Hardening Guides VMware Technical Papers Best Practices for Running vsphere on NFS Storage Best Practices for Running vsphere on iscsi Multipathing Configuration for Software iscsi Using Port Binding Nutanix Virtual Computing Platform Setup Guide URL https://www.vmware.com/support/pubs/vsphere-esxi-vcenterserver-pubs.html http://kb.vmware.com/kb/2034277/ http://kb.vmware.com/kb/2057799/ http://kb.vmware.com/kb/2003322/ http://www.vmware.com/security/hardening-guides http://www.vmware.com/resources/techresources/10096 http://www.vmware.com/resources/techresources/1006 http://www.vmware.com/resources/techresources/10275 https://portal.nutanix.com/#/page/docs 50

CONCLUSION Article/Document Cumulus Linux Documentation Quick Start Guide Understanding Network Interfaces CLAG IGMP and MLD Snooping LACP Bypass Virtual Router Redundancy (VRR) Authentication, Authorization, and Accounting Configuring Buffer and Queue Management Cumulus Linux KB Articles Configuring /etc/network/interfaces with Mako Configuring a Management Namespace Demos and Training, specifically Implementing the VMware vsphere Design Guide in the Cumulus Workbench Cumulus Linux Product Information Software Pricing Hardware Compatibility List (HCL) Cumulus Linux Downloads Cumulus Linux Repository Cumulus Networks GitHub Repository URL http://docs.cumulusnetworks.com https://support.cumulusnetworks.com/hc/enus/articles/202868023 https://support.cumulusnetworks.com/hc/enus/articles/202325278 https://support.cumulusnetworks.com/hc/enus/sections/200398866 https://support.cumulusnetworks.com/hc/enus/articles/203567318 http://cumulusnetworks.com/product/pricing/ http://cumulusnetworks.com/hcl/ http://cumulusnetworks.com/downloads/ http://repo.cumulusnetworks.com https://github.com/cumulusnetworks/ www.cumulusnetworks.com 51

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Appendix A: Example /etc/network/interfaces Configurations leaf01 cumulus@leaf01$ cat /etc/network/interfaces auto eth0 iface eth0 address 192.168.0.90/24 gateway 192.168.0.254 # physical interface configuration auto swp1 iface swp1 mtu 9000 auto swp2 iface swp2 mtu 9000 auto swp3 iface swp3 mtu 9000... auto swp52 iface swp52 mtu 9000 # peerlink bond for clag auto peerlink iface peerlink bond-slaves swp47 swp48 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 # VLAN for clagd communication 52

APPENDIX A: EXAMPLE /ETC/NETWORK/INTERFACES CONFIGURATIONS auto peerlink.4094 iface peerlink.4094 address 169.254.1.1/30 clagd-enable yes clagd-peer-ip 169.254.1.2 clagd-sys-mac 44:38:39:ff:40:94 # uplink bond to spine auto uplink1 iface uplink1 bond-slaves swp49 swp50 bond-mode 802.3ad bond-lacp-rate 1 bond-min-links 1 bond-miimon 100 bond-use-carrier 1 bond-xmit-hash-policy layer3+4 clag-id 1000 auto esx-host-01 iface esx-host-01 bond-slaves swp27 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 bridge-vids 10,15,20,21,22,23,30,40 mstpctl-bpduguard yes clag-id 1 auto bridge iface bridge bridge-vlan-aware yes bridge-ports peerlink uplink1 esx-host-01 bridge-vids 10,15,20,21,22,23,30,40,1000-2000 bridge-pvid 1 bridge-stp on www.cumulusnetworks.com 53

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE leaf02 cumulus@leaf02$ sudo vi /etc/network/interfaces auto eth0 iface eth0 address 192.168.0.91/24 gateway 192.168.0.254 # physical interface configuration auto swp1 iface swp1 mtu 9000 auto swp2 iface swp2 mtu 9000 auto swp3 iface swp3 mtu 9000... auto swp52 iface swp52 mtu 9000 # peerlink bond for clag auto peerlink iface peerlink bond-slaves swp47 swp48 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 # VLAN for clagd communication auto peerlink.4094 iface peerlink.4094 address 169.254.1.2/30 clagd-enable yes clagd-peer-ip 169.254.1.1 clagd-sys-mac 44:38:39:ff:40:94 # uplink bond to spine 54

APPENDIX A: EXAMPLE /ETC/NETWORK/INTERFACES CONFIGURATIONS auto uplink1 iface uplink1 bond-slaves swp49 swp50 bond-mode 802.3ad bond-lacp-rate 1 bond-min-links 1 bond-miimon 100 bond-use-carrier 1 bond-xmit-hash-policy layer3+4 clag-id 1000 auto esx-host-01 iface esx-host-01 bond-slaves swp27 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 bridge-vids 10,15,20,21,22,23,30,40 mstpctl-bpduguard yes clag-id 1 auto bridge iface bridge bridge-vlan-aware yes bridge-ports peerlink uplink1 esx-host-01 bridge-vids 10,15,20,21,22,23,30,40,1000-2000 bridge-pvid 1 bridge-stp on www.cumulusnetworks.com 55

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE spine01 cumulus@spine01$ sudo vi /etc/network/interfaces auto eth0 iface eth0 address 192.168.0.94/24 gateway 192.168.0.254 # physical interface configuration auto swp1 iface swp1 mtu 9000 auto swp2 iface swp2 mtu 9000 auto swp3 iface swp3 mtu 9000... auto swp32 iface swp32 mtu 9000 # peerlink bond for clag auto peerlink iface peerlink bond-slaves swp31 swp32 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 # VLAN for clagd communication auto peerlink.4093 iface peerlink.4093 address 169.254.1.1/30 clagd-enable yes clagd-peer-ip 169.254.1.2 clagd-sys-mac 44:38:39:ff:40:93 # leaf01-leaf02 downlink 56

APPENDIX A: EXAMPLE /ETC/NETWORK/INTERFACES CONFIGURATIONS auto downlink1 iface downlink1 bond-slaves swp1 swp2 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 1 auto downlink2 iface downlink2 bond-slaves swp3 swp4 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 2 # Need connection to core auto bridge iface bridge bridge-vlan-aware yes bridge-ports peerlink downlink1 downlink2 bridge-vids 10,15,20,21,22,23,30,40,1000-2000 bridge-pvid 1 bridge-stp on mstpctl-treeprio 4096 www.cumulusnetworks.com 57

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE spine02 cumulus@spine02$ sudo vi /etc/network/interfaces auto eth0 iface eth0 address 192.168.0.95/24 gateway 192.168.0.254 # physical interface configuration auto swp1 iface swp1 mtu 9000 auto swp2 iface swp2 mtu 9000 auto swp3 iface swp3 mtu 9000... auto swp32 iface swp32 mtu 9000 # peerlink bond for clag auto peerlink iface peerlink bond-slaves swp31 swp32 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 # VLAN for clagd communication auto peerlink.4093 iface peerlink.4093 address 169.254.1.2/30 clagd-enable yes clagd-peer-ip 169.254.1.1 clagd-sys-mac 44:38:39:ff:40:93 # Need connection to core 58

APPENDIX A: EXAMPLE /ETC/NETWORK/INTERFACES CONFIGURATIONS # leaf01-leaf02 downlink auto downlink1 iface downlink1 bond-slaves swp1 swp2 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 1 auto downlink2 iface downlink2 bond-slaves swp3 swp4 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 2 # Need connection to core auto bridge iface bridge bridge-vlan-aware yes bridge-ports peerlink downlink1 downlink2 bridge-vids 10,15,20,21,22,23,30,40,1000-2000 bridge-pvid 1 bridge-stp on mstpctl-treeprio 4096 www.cumulusnetworks.com 59

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE oob-mgmt When utilizing a management networking switch running Cumulus Linux, the switch can be configured by editing /etc/network/interfaces file with the following configuration: cumulus@oob-mgmt$ sudo vi /etc/network/interfaces auto br0 iface br0 bridge-ageing 300 bridge-ports regex (swp[0-9]*[s]*[0-9]) bridge-stp on and reloading all networking: cumulus@oob-mgmt$ sudo ifreload -a 60

APPENDIX B: CONFIGURING A VDS Appendix B: Configuring a vds Refer to the Setting Up vsphere Networking with vsphere Distributed Switch in the vsphere Networking Guide in the VMware vsphere Documentation Center for detailed information on setting up a vds uplink group with multiple teamed vmnics. For example, using a distributed vswitch, create a new vswitch1 to provide uplinks to the leaf switches for VM or VMkernel traffic. This is separate from a dedicated vswitch0 for the out-of-band management network. Creating the new vswitch1 requires four steps: 1. Add networking. Create a new distributed switch. 2. Select the number of uplinks. In this example network there are 2 physical uplink interfaces from each host. 3. Add and manage hosts. Add the vsphere ESXi hosts to the distributed switch. 4. Edit switch settings. Change the MTU and discovery type. First, create a new distributed switch from vcenter. www.cumulusnetworks.com 61

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Next, specify the number of uplink ports. Add hosts. 62

APPENDIX B: CONFIGURING A VDS Configure the adapters on the hosts. www.cumulusnetworks.com 63

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Finally, set the MTU and discovery settings for LACP. 64

APPENDIX C: NETWORK DESIGN AND SETUP CHECKLIST Appendix C: Network Design and Setup Checklist Tasks Considerations 1. Set up physical network and basic configuration of all switches. Select network switches Plan cabling Install Cumulus Linux Refer to the HCL and hardware guides at http://cumulusnetworks.com/support/hcl. Out-of-band management: Leaf switches: Spine switches: Assume minimal traffic requirements, used for initial image loading and then management and monitoring, with no day-to-day data traffic. 48 port 1G switch is sufficient. Choose between at least a 48 port 10G switch or 32 port 40G switch with breakout cables. Consider price and future-proofing. Breakout cables provide more 10G ports on a 40G switch than a single 10G switch. Choose at least a 10G switch or a 40G switch; 40G for more traffic aggregation. Consider price and future-proofing. Use identical switches in pairs to facilitate easier management; more for hot spares. Refer to KB article, Suggested Transceivers and Cables: https://support.cumulusnetworks.com/hc/en-us/articles/202983783. Generally, higher number ports on a switch are reserved for uplink ports, so: Assign downlinks or host ports to the lower end, like swp1, swp2 Reserve higher number ports for network Reserve highest ports for CLAG peer links Connect all console ports. See the Quick Start Guide in the Cumulus Linux Documentation. Obtain the latest version of Cumulus Linux. Obtain license key, which is separate from Cumulus Linux OS distribution. To minimize variables and aid in troubleshooting, use identical versions across switches same version X.Y.Z, packages, and patch levels. At a minimum, ensure switches in CLAG pairs have identical versions. See the Quick Start Guide in the Cumulus Linux Documentation. www.cumulusnetworks.com 65

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE Tasks Reserve management space Determine IP addressing Edit configuration files Considerations Reserve pool of IP addresses. Define hostnames and DNS. RFC 1918 should be used where possible. Use DCHP to avoid manually configuring on each switch: gateway, IP address, DNS information, hostname information, zero touch provisioning URL, installation URL for ONIE. Or use static IP addresses for explicit control, and avoiding managing MAC address to IP address table. Apply standards and conventions to promote similar configurations. For example, place stanzas in the same order in configuration files across switches and specify the child interfaces before the parent interfaces (so a bond member appears earlier in the file than the bond itself, for example). This allows for standardization and easier maintenance and troubleshooting, and ease of automation and the use of templates. Consider naming conventions for consistency, readability, and manageability. Doing so helps facilitate automation. For example, call your leaf switches leaf01 and leaf02 rather than leaf1 and leaf2. Use all lowercase for names Avoid characters that are not DNS-compatible. Define child interfaces before using them in parent interfaces. For example, create the member interfaces of a bond before defining the bond interface itself. Refer to the Configuring Network Interfaces with ifupdown chapter of the Cumulus Linux Documentation for more information. 2. Configure leaf switches. Define switch ports (swp) in /etc/network/interfaces on a switch Set MTU Set speed and duplex Create peer link bond between pair of switches Enable CLAG Assign clagd-sys-mac Assign priority Instantiate swp interfaces for using the ifup and ifdown commands. By default, MTU is set to 1500. Set to a high value, like 9000, to avoid packet fragmentation. These settings are dependent on your network. Assign IP address for clagd peerlink. Consider using a link local address (RFC 3927, 169.254/16) to avoid advertising, or an RFC 1918 private address. Use a very high number VLAN if possible to separate the peer communication traffic from typical VLANs handling data traffic. Valid VLAN tags end at 4096. Set up CLAG in switch pairs. There s no particular order necessary for connecting pairs. Assign a unique clagd-sys-mac value per pair. This value is used for spanning tree calculation, so assigning unique values will prevent overlapping MAC addresses. Use the range reserved for Cumulus Networks: 44:38:39:FF:00:00 through 44:38:39:FF:FF:FF. Define primary and secondary switches in a CLAG, if desired. Otherwise, by default the switches will elect a primary switch on their own. Set priority if you want to explicitly control which switches are designated primary switches. 66

APPENDIX C: NETWORK DESIGN AND SETUP CHECKLIST Tasks Automate setup Considerations Use automation for setting up many switches. Set up a pair of switches manually and verify all connectivity first before attempting to programmatically recreate. Try to configure switches as similarly as possible. The main differences between two switches would be the loopback address, management IP address, CLAG Peer, and VRR. 3. Configure spine switches. Repeat steps for configuring leaf switches Steps for configuring leaf switches are similar to configuring spine switches. Consider using a different VLAN number for spine peer bonds than leaf peer bonds for distinction to avoid accidentally trunking across the same VLAN. 4. Set up spine/leaf network fabric. Create a switch-to-switch uplink bond on each leaf switch to the spine pair and verify CLAG. Create a switch-to-switch bond on the spine pair to each leaf switch pair and verify CLAG. Use different clag-ids for uplinks versus host bonds. 5. Configure VLANs. Define VLANS Determine list of VLANs needed. What VLANs on what interfaces. Prune VLANs where possible. Set native VLAN for trunk ports. 6. Connect vsphere ESXi hosts. Set up high availability to hosts Configure vds. Set LACP to fast mode (default). Enable BDPU guard for port-facing hosts to prevent hooking up a switch on the host port. 7. Configure virtual networking. Create VM and VMkernel networks Use virtual networks Follow VMware networking best practices. www.cumulusnetworks.com 67

VMWARE VSPHERE AND CUMULUS LINUX: VALIDATED DESIGN GUIDE 8. Connect spine switches to core. Connect to core switch at Layer 2, if applicable Connect to core switch at Layer 3, if applicable Check MTU setting for the connection to the core. This depends on what the core needs. Check MLAG-type capability. Determine what VLANs need to be trunked to the core instead of pruned. Ensure the native VLAN matches the core native VLAN. Check MTU setting for the connection to the core. This depends on what the core needs. Determine how to handle the default route: originate or learn? Specify the IP address subnet information for Layer 3 VLANs. Decide what IP address to use for the gateway. Typically this is either.1 or.254. Assign IP addresses for VRR. Typically, they are adjacent to the gateway, so.2 or.253. Assign virtual MAC addresses to use for VRR. For better manageability, use the same MAC address on both peer switches, from the reserved range for VRRP: 00:00:5E:00:01:XX. Determine if routing will use static or dynamic routing. If dynamic: OSPF: BGP: Specify router-id and advertised networks. Consider IPv4 and/or IPv6. Determine protocol, OSPF or BGP. Define area. Verify MTU setting. OSPF in particular can run into problems with improper MTU settings. Define reference bandwidth. Set timers. Define network type, such as point-to-point. Choose between OSFP numbered or OSPF unnumbered interfaces. Define autonomous system number (ASN). Set timers. Choose between ibgp or ebgp. 68