White Paper Cisco SPAN for Cisco Application Centric Infrastructure: A Modern Port Analyzer for the Next-Generation Data Center What You Will Learn Cisco Switched Port Analyzer (SPAN) on Cisco Application Centric Infrastructure (Cisco ACI ) offers new techniques for modern, multitenant data centers with transient (virtual) workloads. This document discusses the challenges presented by today s data centers and how Cisco ACI addresses through four types of SPAN: Tenant SPAN Fabric SPAN Access SPAN Virtual SPAN For each SPAN type, you will learn the available source and destination options and how to configure them. You will also learn the main benefits that each SPAN type can provide in your business and any restrictions that you need to consider. What Is Cisco SPAN? Cisco Switch Port Analyzer, or SPAN, is a Cisco standard widely adopted by the networking industry and available across a wide range of products that is used to copy traffic from one or more ports, port channels, or virtual port channels to a destination. The destination can be a local port or a remote device. This copied traffic can then be run through a variety of analysis tools to reach conclusions about its nature. SPAN is commonly used, for example, to monitor traffic to check for suspicious activity, copy traffic to meet regulatory compliance requirements, and inspect traffic for connectivity problems. Current Challenges for SPAN Traditional switches require the administrator to connect to a terminal and configure a SPAN session through the command-line interface (CLI). This requirement can be a problem in large and complex networks because the administrator will need to manually initiate a SPAN session on every required switch in the potential traffic path. This traffic path may not be known ahead of time, especially in modern, multitenant, transient data centers, in which applications can exist within containers that move between physical hardware outside the control of network engineers. A common example of this scenario is an application virtual machine that is automatically migrated to a new hypervisor when the current hypervisor reaches a predefined resource limit. With traditional switches, the network administrator would need to be informed of this move, end the original SPAN session, and create a new SPAN session on the appropriate switch. If this move is not communicated, problems occur both from a business perspective and from a technical perspective. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 21
From a business perspective, required network traffic may not be appropriately mirrored. At best, the business may fail to learn operational or security information. At worst, traffic for which security is critical may not be inspected, or regulatory requirements may not be met. From a technical perspective, unnecessary network traffic will continue to be replicated on each switch in the original path until the SPAN sessions are ended manually, unnecessarily wasting network bandwidth. The network operator will need to be extremely diligent to remove any unnecessary SPAN sessions because they are a hardware-limited resource. The network operator also will need to find the location to which the virtual machine has been migrated and begin a whole new set of SPAN sessions to adequately capture the traffic, wasting valuable staff time. Organizations need a new way to capture and analyze traffic that solves this problem by adapting to modern data center trends. Introducing a New SPAN Concept with Cisco ACI Cisco ACI introduces a new layer of policy abstraction on top of the switch hardware. This layer includes the logical networking construct of endpoint groups (EPGs, see For More Information at the end of this document). EPGs consume switch hardware resources only when relevant endpoints are present. As workloads move around the data center, the EPG expands and contracts to meet resource needs. A SPAN session based on static hardware ports cannot address this scenario. Cisco ACI thus has introduced the new concept of Tenant SPAN. Tenant SPAN aggregates SPAN sessions across multiple leaf switches transparently and on demand. The administrator is free to describe semantically how traffic should be replicated, and the Cisco Application Policy Infrastructure Controller (APIC) will command the appropriate hardware resources to initiate SPAN sessions on demand to capture relevant traffic. Limitations of Solutions from Other Vendors In current software-only software-defined networking (SDN) solutions, the controller has no integration with the underlying switches, so it cannot initiate or control hardware SPAN sessions. Furthermore, unlike Cisco ACI, in which the copy operation is performed in optimized application-specific circuits (ASICs) and has no impact on the CPU, software-only SDN solutions must rely on software to copy traffic from virtual network ports. This approach limits SPAN to virtual machine only traffic. It also consumes precious CPU cycles on the hypervisor: an extremely valuable resource in any data center. Continued Support for SPAN, RSPAN, and ERSPAN Although Tenant SPAN is excellent for dynamic workloads in a multitenant Cisco ACI fabric, Cisco ACI is used in many different scenarios. Cisco ACI thus continues to make available, provide support for, and build on the tried and tested Remote SPAN (RSPAN) and Encapsulated RSPAN (ERSPAN) features. In addition, if virtual workloads need to be spanned directly within a virtual switch (vswitch), Cisco ACI can be paired with Cisco Application Virtual Switch (AVS) and used to create and manage Virtual SPAN (vspan) sessions, thus providing a full end-to-end SPAN feature set. Easy Deployment The process of configuring SPAN in Cisco ACI is straightforward, especially after you become familiar with the terminology and know the use case that is relevant to your requirements. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 21
Use of Filters to Capture Only the Traffic Needed Cisco ACI introduces a concept of SPAN filters. Different SPAN sources have different filtering options, which will be discussed in each use case presented in this document. In general, filters restrict traffic to specific EPGs, bridge domains, or Virtual Routing and Forwarding (VRF) instances, allowing the network operator to easily remove unnecessary traffic from a SPAN session. Identifying the ERSPAN Traffic Source In Cisco ACI, SPAN sessions can be dynamically initiated by the APIC on demand, so you need a way at the destination to identify the switch that is the traffic source, especially if multiple leaf switches are sourcing SPAN traffic in the same session. You can use the source IP prefix to provide this identification. The configured source IP prefix is combined with the sending switch s node ID to produce a unique IP address in the destination EPG. For example, Table 1 shows sample results for source IP prefix 192.168.0.0/24. Table 1. Identifying the ERSPAN Traffic Source Leaf Switch Node ID ERSPAN Source IP Address Leaf 101 101 192.168.0.101/24 Leaf 102 102 192.168.0.102/24 The SPAN source does not need a network path to the destination EPG. In fact, the source can be in an entirely different tenant, VRF instance, or bridge domain than the destination. Support for Local and Remote Destinations When traffic is replicated, it needs to be delivered to a destination. Originally SPAN traffic could be mirrored only locally on the switch. Extensions such as RSPAN and ERSPAN allowed traffic to be encapsulated and sent to a remote switch or device. Cisco ACI supports local and remote (ERSPAN) destinations in the various types of SPAN. Not all combinations are supported, however, as discussed later in this document. How ERSPAN Reaches the Destination When ERSPAN is used, the destination EPG must belong to a bridge domain that has unicast routing enabled and at least one subnet configured. The ERSPAN packet is injected into the destination EPG on the source leaf switch with the outer source address set to the generated IP address (See Identifying the ERSPAN Traffic Source earlier in this document) and the outer destination IP address set to the destination IP address. The packet then follows the same forwarding path as normal traffic in this EPG. Therefore, the destination must be reachable from this EPG. ERSPAN Types I and II When mirroring traffic to a remote destination, you need to consider the type of ERSPAN traffic that is generated. As previously mentioned, with Cisco ACI the copying and encapsulation of SPAN traffic is offloaded to the switch ASICs. This approach is beneficial because it eliminates the need for any CPU work and has no negative effect on control plane traffic that is dependent on CPU time. Because Cisco ACI uses a merchant+ methodology (in which Broadcom and Cisco chips are combined in one chassis), you must be aware of the way that the different chips implement ERSPAN. When generating remote (ERSPAN) traffic, you need to know which chip encapsulated the packet so that you can validate your remote device to decode the packet correctly. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 21
Tenant and Access SPAN use Type I (Broadcom chips) and Fabric SPAN uses Type II (Cisco chips). Note, though, that if you are using ERSPAN Type I and Wireshark, by default, Wireshark will not decode the packets. To decode them, you need to choose Preferences > Protocols > ERSPAN and then select Force to decode fake ERSPAN frame (Figure 1). Figure 1. Decoding ERSPAN Type I in Wireshark Use Cases This document discusses four use cases: Mirror all traffic to and from an EPG to a remote destination (Tenant SPAN) Mirror all traffic to and from my spine switches to a remote destination (Fabric SPAN) Mirror all traffic to and from leaf host ports locally or to a remote destination (Access SPAN) Mirror a virtual interface on a virtual machine to a remote destination (Virtual SPAN) Use Case Topology All the use cases use the same topology (Figure 2). The topology has: Two spine switches Two leaf switches Two local SPAN destinations Two remote SPAN destinations Two hypervisors Two tenants Three EPGs 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 21
Figure 2. Network Topology for Use Case Examples Use Case: Mirror All Traffic to and from an EPG to a Remote Destination Tenant SPAN Main Facts The source can be only an EPG. The destination can be only ERSPAN. ERSPAN encapsulation Type I The direction can be: Inbound Outbound Both No filtering is possible. In this use case, you want to mirror traffic when you do not know where the physical source interfaces are ahead of time, but you know that you want to capture all traffic in and out of any physical port that belongs to this EPG (Figure 3). 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 21
Figure 3. Tenant SPAN: Possible Sources and Destinations Configuring the Destination Choose Tenants > your tenant > Troubleshoot Policies > SPAN > SPAN Destination Groups. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 21
2. Click the + icon to add a destination. 2. Choose the destination EPG. a. This EPG must have connectivity to the destination IP address. (See How ERSPAN Reaches the Destination ) 3. Specify the EPG source IP prefix. a. Refer to the discussion earlier in this document for details about how the source IP address is generated (See Identifying the ERSPAN Traffic Source ). Configuring the Source Choose Tenants > your tenant > Troubleshoot Policies > SPAN > SPAN Source Groups. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 21
2. Select the destination group. 3. Click the + icon to add a source. 1. Specify the span source name. 2. Choose the direction. 3. Select the source EPG. You will need to configure a remote collector or analysis tool to capture the ERSPAN traffic and decode it to view the original packet. For example, a common remote collector, Wireshark, can capture the ERSPAN traffic, decode the outer ERSPAN encapsulation, and display the original packet header and payload, including the original source and destination IP and MAC addresses. Use Case: Mirror All Traffic to and from My Spine Switches to a Remote Destination Fabric SPAN Main Facts The source must be a fabric (uplink) port on a leaf or spine switch. 1/49 to 1/60 on Cisco Nexus 9396 (leaf switch) 1/49 to 1/54 on Cisco Nexus 9372 (leaf switch) 1/1 to 1/36 on Cisco Nexus 9336 (spine switch) The destination can be only ERSPAN. ERSPAN encapsulation Type II 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 21
The direction can be: Inbound Outbound Both The filter options are: Private network Bridge domain Multiple source paths are supported. You can have multiple switches (leaf or spine) with the same SPAN policy. In this use case, you want to mirror traffic that is traversing the spine switches within the fabric (Figure 4). You can choose one or more fabric ports (on leaf or spine) and then replicate the traffic to a remote location. Figure 4. Fabric SPAN: Possible Sources and Destinations 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 21
Configuring the Destination Choose Fabric > Fabric Policies > Troubleshoot Policies > SPAN > SPAN Destination Groups. 2. Click the + icon to add a destination. 2. Choose the destination EPG. a. This EPG must have connectivity to the destination IP address (See How ERSPAN Reaches the Destination ). 3. Specify the EPG source IP prefix. a. Refer to the discussion earlier in this document for details about how the source IP address is generated (See Identifying the ERSPAN Traffic Source ). 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 21
Configuring the Source Choose Fabric > Fabric Policies > Troubleshoot Policies > SPAN > SPAN Source Groups. 2. Select the destination group. 3. Click the + icon to add a source. 2. Choose the direction. 3. (Optional) Select a filter. 4. Add one or more paths. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 21
Use Case: Mirror All Traffic to and from a Switch Port Locally or to a Remote Destination Access SPAN Main Facts The source port can be any access port. The destination can be another access port (not a port channel or virtual port channel [vpc]) or ERSPAN. ERSPAN encapsulation Type I The direction can be: Inbound Outbound Both The filter options are: Tenant Application profile Endpoint group Multiple source paths are supported. In this use case, you want to mirror traffic that is flowing to and from any host-facing ports on a leaf switch (Figure 5). You can locally mirror the traffic to a switch port, or you can send it to a remote destination. A local destination is useful when you want to help ensure that the mirrored traffic does not leave this switch: an important decision to make when planning network capacity. Figure 5. Access SPAN: Possible Sources and Destinations 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 21
Configuring the Destination Choose Fabric > Access Policies > Troubleshoot Policies > SPAN > SPAN Destination Groups. 2. Click the + icon to add a destination. Adding an EPG Destination 2. Choose the destination EPG. a. This EPG must have connectivity to the destination IP address (See How ERSPAN Reaches the Destination ). 3. Specify the EPG source IP prefix. a. Refer to the discussion earlier in this document for details about how the source IP address is generated (See Identifying the ERSPAN Traffic Source ). 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 21
Adding a Switch Interface Destination 2. Choose the destination path Configuring the Source Choose Fabric > Access Policies > Troubleshoot Policies > SPAN > SPAN Source Groups. 2. Select the destination group. 3. Click the + icon to add a source. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 21
2. Choose the direction. 3. (Optional) Select a filter. 4. Select the source. Use Case: Mirror a Virtual Interface on a Virtual Machine to a Remote Destination Virtual SPAN Main Facts vspan requires Cisco Application Virtual Switch. The source can be an EPG or a virtual interface. The destination can be ERSPAN or a virtual interface. No filtering is possible. The direction can be: Inbound Outbound Both In this use case, you want to take advantage of the Application Virtual Switch to mirror traffic from a virtual switch (Figure 6). This approach is useful when traffic is being switched locally within the hypervisor and therefore cannot be captured by the physical leaf switch. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 21
Figure 6. Virtual SPAN: Possible Sources and Destinations Configuring the Destination Choose Fabric > Access Policies > Troubleshoot Policies > VSPAN > VSPAN Destination Groups. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 21
2. Click the + icon to create a destination group. 2. Select the destination type. a. ERSPAN (remote destination) Note: vspan ERSPAN traffic is sourced differently. Traffic is sourced from the tunnel endpoint (TEP) address of the Application Virtual Switch in the infrastructure EPG. Verify that the remote IP address is reachable from this context. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 21
b. LSPAN (virtual interface) i. Choose a tenant, application, and EPG. ii. Select the virtual machine interface of the endpoint to which traffic should be sent. Note: Service graph enabled virtual machine interfaces are not available. Configuring the Source Choose Fabric > Access Policies > Troubleshoot Policies > VSPAN > VSPAN Sessions. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 21
2. Select the destination group. 3. Click the + icon to add a source. 2. Choose the capture direction. 3. Select the source type. a. EPG i. Select the desired tenant, application profile, and EPG. ii. Choose a source path (port channel, vpc, or port). b. CEP (virtual endpoint) i. Select the desired tenant, application profile, and EPG. ii. iii. Select the source client endpoint (CEP). Choose a source path (port channel, vpc, or port). Using SPAN to Troubleshoot Two Endpoints Quickly You now know the four SPAN types, their usefulness in a modern data center, and how to configure them. However, sometimes, in a troubleshooting session, you may need to quickly configure a SPAN session to capture traffic between two endpoints. To do so, you can use the Troubleshooting SPAN Wizard. The Troubleshooting SPAN Wizard is especially useful for network operations teams. It does not use a different SPAN method, but relies on Access SPAN. It is a feature of the Cisco ACI Visibility and Troubleshooting Tool (See ACI Visibility and Troubleshooting Tool in the For More Information section at the end of this document), which can be viewed as a one-stop shop for network operations teams. Given two endpoints, the troubleshooting tool will dynamically build a temporary Access SPAN session to mirror the necessary traffic to capture the flow. After the capture is complete, the SPAN session is taken down. 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 21
Two distinct destinations are introduced here: the APIC and the host through the APIC. Both require in-band management to be configured for the fabric (See Configuring In-Band Management Access in the For More Information section at the end of this document). For the APIC destination type, the APIC acts as a capture device from which the mirrored traffic can be downloaded or inspected. The host through the APIC destination type causes the APIC to act as a proxy, forwarding mirrored traffic to an external analyzer. The Visibility and Troubleshooting Tool is available from the Operations tab (Figure 7). Figure 7. Troubleshooting SPAN Wizard SPAN Type Comparison Table Table 2 provides a summary that shows the differences among the SPAN types. Table 2. SPAN Type Comparison SPAN Type Source Filter Destination Fabric SPAN Fabric port Bridge domain Private network Access SPAN Access port Tenant Application profile Endpoint group Remote (ERSPAN Type II) Remote (ERSPAN Type I) Local Tenant SPAN Endpoint group Remote (ERSPAN Type I) Virtual SPAN Virtual machine interface Remote (ERSPAN Type I) LSPAN (virtual machine interface) Scalability As with all network devices, you must plan capacity appropriately when you use SPAN with Cisco ACI. For each leaf, you can have: Four Tenant or Access SPAN sessions Four Fabric SPAN sessions For each SPAN session, you may have: Up to all leaf access ports as the source (Access SPAN) Up to all fabric ports as the source (Fabric SPAN) 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 21
Up to 280 EPGs or bridge domains as the source (Tenant SPAN) Note that after SPAN traffic has been captured, it will compete with normal traffic on the fabric to be delivered. Be sure to plan for SPAN traffic accordingly to avoid link oversubscription. For more information, please see the current verified scalability guide (Version 1.2(1i) at the time of writing). Conclusion In a modern, multitenant datacenter with transient (virtual) workloads, you need a network that can shift at the speed of your business while still delivering all the capabilities available to the switch hardware. Cisco ACI with Cisco SPAN is the only SDN solution that offers all these features while continuing to innovate with new ideas such as Tenant SPAN. The Cisco solution offers robust capabilities well known to network engineers and tried and tested throughout the world in thousands of data centers. If you need SPAN, you need Cisco ACI. For More Information For additional information, see the following: Cisco APIC Troubleshooting Guide Cisco ACI SPAN Guidelines and Restrictions Video: Cisco APIC Configuring a SPAN Session Cisco Application Centric Infrastructure (ACI) - Endpoint Groups (EPG) Usage and Design Cisco ACI Visibility and Troubleshooting Tool Configuring In-Band Management Access Printed in USA C11-736974-00 04/16 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 21