Cisco Application Virtual Switch Troubleshooting Guide, Release 5.2(1)SV3(1.x) First Published: February 22, 2015 Last Modified: February 24, 2016 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental. This product includes cryptographic software written by Eric Young (eay@cryptsoft.com). This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/) This product includes software written by Tim Hudson (tjh@cryptsoft.com). Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http:// www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R) 2014-2015 Cisco Systems, Inc. All rights reserved.
CONTENTS Preface Preface v Audience v Related Documentation: Cisco AVS v Related Documentation: Cisco ACI and Cisco APIC v Documentation Feedback vi Obtaining Documentation and Submitting a Service Request vi CHAPTER 1 Overview of Troubleshooting 1 About the Troubleshooting Process 1 About Best Practices 1 Common Troubleshooting Tasks 2 Troubleshooting Basics 2 Troubleshooting Preliminary Steps 2 Verifying Ports 3 Verifying Layer 2 Connectivity 3 Contacting Cisco Customer Support 3 Collecting and Exporting Cisco AVS Log Files for Cisco Customer Support 4 Manually Generating Log Files for Cisco Customer Support 4 Collecting and Exporting Cisco AVS Log Files Using the Advanced GUI 5 Collecting and Exporting Cisco AVS Log Files Using the Basic GUI 5 Collecting and Exporting Cisco AVS Log Files Using the NX-OS Style CLI 6 CHAPTER 2 Installation and Configuration 9 Verifying Your VMware License 9 Recovering from a Cisco AVS Creation Failure 10 Recovering the Network Administrator Password 10 iii
Contents CHAPTER 3 Adding Hosts 11 Adding Hosts 11 Ensuring that a VTEP vmknic is Added for Valid Host 11 CHAPTER 4 OpFlex 13 Enabling OpFlex 13 OpFlex Connection Sequence 15 Recovering the Connection with APIC after Deleting an OpFlex vmknic 17 Recovering from an OpFlex Failure Due to a Certificate Issue 18 CHAPTER 5 Ports, Endpoint Groups, and Layer 2 21 About Data Paths 21 Diagnosing Port Activity 22 Troubleshooting Unavailable Ports 23 Checking Port Synchronization Using Port Counters 24 Troubleshooting Endpoint Groups 25 Recovering from Endpoint Group Creation Failure 26 Troubleshooting Layer 2 Switching 26 CHAPTER 6 Port Channels 29 Port Channel Overview 29 Verifying Port Channels 29 Troubleshooting Port-Channel Creation 33 CHAPTER 7 Switched Port Analyzer 35 About the Switched Port Analyzer 35 Viewing the Switched Port Analyzer Configuration 36 Troubleshooting the Switched Port Analyzer 37 CHAPTER 8 System Troubleshooting 41 VEM Commands 41 iv
Preface This preface contains the following sections: Audience, page v Related Documentation: Cisco AVS, page v Related Documentation: Cisco ACI and Cisco APIC, page v Documentation Feedback, page vi Obtaining Documentation and Submitting a Service Request, page vi Audience This guide is intended for network and systems administrators who install and deploy the Cisco Application Virtual Switch (AVS) on the Cisco Application Policy Infrastructure Controller (APIC). Related Documentation: Cisco AVS This section lists the documents used with the Cisco AVS and available at the following URL: http://www.cisco.com/c/en/us/support/switches/application-virtual-switch/tsd-products-support-series-home.html Cisco Application Virtual Switch Installation Guide Cisco Application Virtual Switch Configuration Guide Cisco Application Virtual Switch Release Notes Related Documentation: Cisco ACI and Cisco APIC Documentation for the Cisco Application Centric Infrastructure (ACI) and the Cisco APIC can be found here: http://www.cisco.com/c/en/us/support/cloud-systems-management/ application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html v
Documentation Feedback Preface Web-Based Documentation Cisco APIC Management Information Model Reference Cisco APIC Online Help Reference Cisco ACI MIB Support List Downloadable Documentation Cisco Application Centric Infrastructure Release Notes Cisco Application Centric Infrastructure Fundamentals Guide Cisco APIC Getting Started Guide Cisco APIC REST API User Guide Cisco APIC Command Line Interface User Guide Cisco APIC Faults, Events, and System Message Guide Cisco APIC Layer 4 to Layer 7 Device Package Development Guide Cisco APIC Layer 4 to Layer 7 Services Deployment Guide Cisco ACI Firmware Management Guide Cisco ACI Troubleshooting Guide Cisco ACI NX-OS Syslog Reference Guide Cisco ACI Switch Command Reference, NX-OS Release 11.0 Cisco ACI MIB Quick Reference Cisco Nexus CLI to Cisco APIC Mapping Guide Application Centric Infrastructure Fabric Hardware Installation Guide Documentation Feedback To provide technical feedback on this document, or to report an error or omission, please send your comments to avs-docfeedback@cisco.com. We appreciate your feedback. Obtaining Documentation and Submitting a Service Request For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation at: http:// www.cisco.com/c/en/us/td/docs/general/whatsnew/whatsnew.html Subscribe to What s New in Cisco Product Documentation, which lists all new and revised Cisco technical documentation as an RSS feed and delivers content directly to your desktop using a reader application. The RSS feeds are a free service. vi
CHAPTER 1 Overview of Troubleshooting This chapter contains the following sections: About the Troubleshooting Process, page 1 About Best Practices, page 1 Common Troubleshooting Tasks, page 2 Troubleshooting Basics, page 2 Contacting Cisco Customer Support, page 3 About the Troubleshooting Process To troubleshoot the Cisco Application Virtual Switch (AVS), follow these general steps: 1 Gather information that defines the specific symptoms. 2 Identify all potential problems that could be causing the symptoms. 3 Systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear. About Best Practices Best practices are the recommended steps you should take to ensure the proper operation of your network. We recommend that you follow these best practices: Maintain the same Cisco Application Virtual Switch (AVS) release across all network devices. Refer to the release notes for your Cisco Application Virtual Switch release for the latest features, limitations, and caveats. Verify and troubleshoot any new configuration changes after implementing the change. 1
Common Troubleshooting Tasks Overview of Troubleshooting Common Troubleshooting Tasks Using a given a set of symptoms on a network, you should be able to diagnose and correct software configuration issues and defective hardware with minimal disruption to the network. With help from this guide, you can perform the following common troubleshooting tasks: Identify key Cisco Application Virtual Switch troubleshooting tools. Obtain and analyze protocol traces using Switched Port Analyzer (SPAN) or Ethanalyzer on the CLI. Identify or rule out physical port issues. Identify or rule out switch module issues. Diagnose and correct Layer 2 issues. Troubleshooting Basics Obtain diagnostic data for use by the Cisco Technical Assistance Center (TAC). This section introduces questions to ask yourself when you are troubleshooting a problem with the Cisco Application Virtual Switch (AVS) or connected devices. Use the answers to these questions to identify the scope of the problem and to plan a course of action. This section includes the following topics: Troubleshooting Preliminary Steps, on page 2 Verifying Ports, on page 3 Verifying Layer 2 Connectivity, on page 3 Troubleshooting Preliminary Steps To discover a network problem, use the following general network troubleshooting steps: Before You Begin By answering the questions in this section and the following subsections, you can determine the paths you need to follow and the components that you should investigate further. Answer the following questions to determine the status of your installation: Is this a newly installed system or an existing installation? (For example, is it a new host, switch, or VLAN?) Has the host ever been able to see the network? Are you trying to solve an existing application problem? (For example, is the device too slow, is the latency too high, the response time excessively long, or did the problem occur only recently?) 2
Overview of Troubleshooting Verifying Ports What changed in the configuration or in the overall infrastructure immediately before the applications started to have problems? Step 1 Gather information about the problems in your system. See individual sections for instructions on how gather information. Step 2 Verify Layer 2 connectivity. See Verifying Layer 2 Connectivity, on page 3. Step 3 Verify the configuration for your end devices (storage subsystems and servers). Verifying Ports In addition to gathering software-configured port information, answer the following questions to verify physical port integrity: Are you using the correct media (such as copper, optical, or fiber)? Are the media broken or damaged? Are you checking a physical Ethernet port? If so, are you looking at the server, or are you looking at an upstream switch? Verifying Layer 2 Connectivity Answer the following questions to quickly eliminate common problems with Layer 2 connectivity: Are the necessary interfaces in the same VLANs? Are all ports in a port channel configured the same for speed, duplex, and trunk mode? Contacting Cisco Customer Support If you are unable to solve a problem after using the troubleshooting suggestions in this guide, contact a Cisco customer service representative for assistance and further instructions. Before you call, have the following information ready to help your representative assist you as quickly as possible: Version of the Cisco Application Virtual Switch (AVS) software that you are running. Version of the VMware vsphere (ESXi) and vcenter Server software that you are running. Contact phone number. Brief description of the problem. Brief explanation of the steps that you have already taken to isolate and resolve the problem. 3
Collecting and Exporting Cisco AVS Log Files for Cisco Customer Support Overview of Troubleshooting Collecting and Exporting Cisco AVS Log Files for Cisco Customer Support Cisco Customer Support might ask you to provide log files from Cisco AVS. In releases preceding Cisco AVS Release 5.2(1)SV3(1.10), you issue a vem command to generate Cisco AVS log files into a tar file that you can provide to Cisco Customer Support when asked. In Cisco AVS Release 5.2(1)SV3(1.10) and later releases, you can use the APIC GUI or NX-OS style CLI to collect and export Cisco AVS log files to a designated remote server. However, be aware of the following: In Release 5.2(1)SV3(1.10), you can use the APIC GUI or NX-OS style CLI for collecting and exporting log files if you are using IPV4 addresses, provided that the destination server supports IPV4 addresses. However, if you are using IPV6 addresses, you must issue a vem command to generate a.tar file. In Release 5.2(1)SV3(1.15) and later releases, you can use the APIC GUI or NX-OS style CLI for exporting log files if you are using IPV4 addresses or IPV6 addresses, provided that the destination server supports IPV4 or IPV6 addresses. For instructions, see the following sections in this guide: Manually Generating Log Files for Cisco Customer Support Collecting and Exporting Cisco AVS Log Files Using the Advanced GUI Collecting and Exporting Cisco AVS Log Files Using the Basic GUI Collecting and Exporting Cisco AVS Log Files Using the NX-OS Style CLI Caution Cisco recommends that you do not mix Cisco APIC GUI configuration modes (Advanced or Basic). When you make a configuration in either mode and change the configuration using the other mode, unintended changes can occur. For example, if you apply an interface policy to two ports using Advanced mode and then change the settings of one port using Basic mode, your changes might be applied to both ports. Manually Generating Log Files for Cisco Customer Support You can use a vem command to generate Cisco AVS log files in a.tar file that you can send to Cisco Customer Support when asked. Step 1 Step 2 Log in to the Cisco AVS host. Perform the following command:vem-support all The resulting log files will be generated in a tar file with name in this format: cisco-vem-<year>-<monthday>-<time>-<hostname>.tgz. /tmp # vem-support -h /bin/vem-support [options] all Options: -i Interactive; hit a key to continue each step -v Verbose; prints each command before executing 4
Overview of Troubleshooting Collecting and Exporting Cisco AVS Log Files for Cisco Customer Support -z Dry run; just print what would be done -t <dir> Specify a directory where the data will be gathered. Default: /tmp /tmp # vem-support all Generated /tmp/cisco-vem-2016-0201-1657-localhost.cisco.com-module1.tgz /tmp # ls -l total 24588 -rw-r--r-- 1 root root 15993309 Feb 1 16:58 cisco-vem-2016-0201-1657-localhost.cisco.com-module1.tgz Step 3 Copy the.tar file to wherever you need in order to send it to Cisco Customer Support. Collecting and Exporting Cisco AVS Log Files Using the Advanced GUI Before You Begin You must make sure that the destination server supports IPV4 or IPV6. The OpFlex communication channel between Cisco APIC and the Cisco AVS host must be in active state. Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Log in to Cisco APIC, choosing Advanced mode. Go to Admin > Import/Export. In the Import/Export navigation pane, expand the Export Policies folder. Right-click the AVS TechSupport folder and choose Create AVS TechSupport. In the Create AVS TechSupport dialog box, in the Name field, enter a name. In the Export Destination area, choose a destination from the drop-down list or create one if you have not already done so. In the AVS area, click the + icon. From the drop-down list, choose the AVS hosts from which you want to collect log files. Click UPDATE and then click SUBMIT. The AVS TechSupport work pane displays the new policy. Right-click the new policy, and choose Collect TechSupports. Click the Operational tab to view the status of the collection. What to Do Next You can retrieve the exported log files from the remote destination. Collecting and Exporting Cisco AVS Log Files Using the Basic GUI Before You Begin You must make sure that the destination server supports IPV4 or IPV6. 5
Collecting and Exporting Cisco AVS Log Files for Cisco Customer Support Overview of Troubleshooting The OpFlex communication channel between Cisco APIC and the Cisco AVS host must be in active state. Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Log in to Cisco APIC, choosing Basic mode. Go to Admin > TechSupport/Core Export. In the TechSupport navigation pane, right-click the AVS TechSupport folder, and choose Create AVS TechSupport. In the Create AVS TechSupport dialog box, in the Name field, enter a name. In the Export Destination area, choose a destination from the drop-down list or create one if you have not already done so. In the AVS area, click the + icon. From the drop-down list, choose the AVS hosts from which you want to collect log files. Click UPDATE and then click SUBMIT. The AVS TechSupport work pane displays the new policy. Right-click the new policy, and choose Collect TechSupports. Click the Operational tab to view the status of the collection. What to Do Next You can retrieve the exported log files from the remote destination. Collecting and Exporting Cisco AVS Log Files Using the NX-OS Style CLI This section provides instructions for using the NX-OS style CLI to collect and export Cisco AVS log files. The example commands shown use an IPV4 address; if you are using Cisco AVS Release 5.2(1)SV3(1.15), you can use IPV4 or IPV6 addresses provided that the destination server supports them. Note For information about accessing and using the NX-OS style CLI, see Cisco APIC Getting Started Guide. Before You Begin You must make sure that the destination server supports IPV4 or IPV6. Step 1 Define the destination on a remote server to receive exported Cisco AVS log files, providing the IP address, port, protocol, username, and filepath for the destination. Note Secure Copy Protocol (SCP) is the only protocol supported for exporting Cisco AVS log files. Step 2 apic1# configure apic1(config)# apic(config)# techsupport remote dest1 10.0.0.1 8000 scp username /tmp/ <CR> Press Enter, enter a password, and then exit configuration mode. 6
Overview of Troubleshooting Collecting and Exporting Cisco AVS Log Files for Cisco Customer Support Step 3 apic(config)# techsupport remote dest1 10.0.0.1 8000 scp username /tmp/ Destination password: ********* apic1#(config)# exit Export the Cisco AVS log files, providing the host ID number and remote destination. Step 4 apic1# trigger techsupport host 167819358 remotename dest1 View the progress of the export. apic1# show techsupport host 167819358 status Nodeid : 167819358 Collection Time : 2015-10-16T19:03:43.528+00:00 Status : preinit Detailed status : Waiting to be scheduled. Note It can take several minutes for the export to complete. You might see status from previous times you entered the command. What to Do Next You can retrieve the exported log files from the remote destination. 7
Collecting and Exporting Cisco AVS Log Files for Cisco Customer Support Overview of Troubleshooting 8
CHAPTER 2 Installation and Configuration This chapter contains the following sections: Verifying Your VMware License, page 9 Recovering from a Cisco AVS Creation Failure, page 10 Recovering the Network Administrator Password, page 10 Verifying Your VMware License You can verify that your ESXi server uses the VMware Enterprise Plus license. This license includes the distributed virtual switch (DVS) feature, which allows you visibility into the Cisco Application Virtual Switch (AVS). Before You Begin Ensure that you are logged into vsphere Web Client on the VMware vsphere (ESXi) server. Ensure that you are logged into the Cisco AVS. Step 1 Step 2 From vsphere Web Client, choose the host whose Enterprise Plus license you want to check. Examine the Enterprise Plus licensed features: a) Click the Configuration tab. b) Choose Licensed Features. Step 3 Verify that the following features are included in the licensed features: An Enterprise Plus license. The DVS feature. Step 4 If your ESXi server does not have an Enterprise Plus license, upgrade your VMware license to an Enterprise Plus license so that you can have visibility into the Cisco AVS. 9
Recovering from a Cisco AVS Creation Failure Installation and Configuration Recovering from a Cisco AVS Creation Failure After you create a Virtual Machine Manager (VMM) domain on the Application Policy Infrastructure Controller (APIC) GUI, configuration mistakes can prevent a distributed virtual switch (DVS) from being created. If you do not see DVS creation on the vcenter GUI, perform the following steps in the APIC GUI: Step 1 Step 2 Step 3 Step 4 Step 5 Ensure that the correct attachable entity profile is associated with the VMM domain that you used to create Cisco Application Virtual Switch (AVS) in vcenter. Ensure that the correct vcenter credentials are associated with the VMM domain that you used to create the Cisco AVS in vcenter. Ensure that the correct vcenter credential profile is associated with the vcenter controller profile for the VMM domain that you used to create the Cisco AVS in vcenter. Ensure that the correct data center name is entered in the vcenter controller profile associated with the VMM domain that you used to create the Cisco AVS in vcenter. Ensure that the correct vcenter IP address is entered in the vcenter controller profile associated with the VMM domain that used to create the Cisco AVS in vcenter. Recovering the Network Administrator Password The network administrator password is recoverable if lost. For instructions on recovering the network administrator password, see the Cisco Nexus 1000V Password Recovery Procedure. 10
CHAPTER 3 Adding Hosts This chapter contains the following section: Adding Hosts, page 11 Ensuring that a VTEP vmknic is Added for Valid Host, page 11 Adding Hosts This chapter contains the following section: Ensuring that a VTEP vmknic is Added for Valid Host When you add valid and invalid hosts to the Cisco Application Virtual Switch (AVS), the Virtual Extensible LAN (VXLAN) tunnel endpoint (VTEP) virtual kernel NIC (vmknic) might not be added for the valid hosts. (A valid host has vsphere Installation Bundle (VIB) installed; an invalid host does not.) When you try to install valid and invalid hosts, vcenter returns a Simple Object Access Protocol (SOAP) error. 11
Ensuring that a VTEP vmknic is Added for Valid Host Adding Hosts Complete the following steps to ensure that VTEP vmknics are added the valid hosts. Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Log into vcenter. Choose Home and then Hosts and Clusters. Choose the host, click the Configuration tab, and then click vsphere Distributed Switch. Click Manage Virtual Adapters. In the Manage Virtual Adapters dialog box, click Add. In the Add Virtual Adapters dialog box, make sure that the New virtual adapter radio button is selected and then click Next. In the vsphere Distributed Switch area, click the Select port group radio button, choose vtep from the drop-down menu, and then click Next. Click the Obtain IP settings automatically radio button and then click Next. Click Finish. Wait for about a minute, choose the VTEP vmknic in the Manage Virtual Adapters dialog box, and verify that the VTEP vmknic received a DHCP address in the IP Address field. What to Do Next Verify that OpFlex comes online by entering the following command on the valid host's console and examining the output:vemcmd show openflex The status should be 12 (Active). 12
CHAPTER 4 OpFlex This chapter contains the following sections: Enabling OpFlex, page 13 OpFlex Connection Sequence, page 15 Recovering the Connection with APIC after Deleting an OpFlex vmknic, page 17 Recovering from an OpFlex Failure Due to a Certificate Issue, page 18 Enabling OpFlex OpFlex is the control protocol between the vleaf and the Application Policy Infrastructure Controller (APIC). It is automatically enabled when the Cisco Application Virtual Switch (AVS) is added to the distributed virtual switch (DVS) in vsphere Web Client. If OpFlex is not enabled, virtual machine (VM) ports fail to come up. If this situation occurs, use the following procedure. Step 1 On an ESXi hypervisor console, verify that OpFlex is online by completing the following steps: a) Enter the following command: vemcmd show openflex The system shows the runtime status of OpFlex. avs-instance# vemcmd show openflex Status: 12 (Active) Dvs name: comp/prov-vmware/ctrlr-[mininet]-o3-vcenter/sw-dvs-2923 Remote IP: 10.0.0.30 Port: 8000 Infra vlan: 2 FTEP IP: 10.0.0.32 Switching Mode: LS A status of 12 (Active) indicates that OpFlex is online. Any other status indicates a problem. See OpFlex Connection Sequence, on page 15 for more information about possible OpFlex status values. The system provides other useful information, including the remote and fabric tunnel endpoint (FTEP) (leaf) IP addresses, the infra VLAN, and the switching mode. Note 13
Enabling OpFlex OpFlex Step 2 b) If OpFlex is not running, restart the Cisco AVS by removing and readding the host to the DVS. c) If restarting does not fix the problem, call the Cisco Technical Assistance Center (TAC). Verify that the Virtual Ethernet Module (VEM) agent is running. a) Enter the following command: vem status The system shows the status of the VEM agent and which VEM modules are running, as shown in the following example: avs-instance# vem status VEM modules are loaded Switch Name Num Ports Used Ports Configured Ports MTU Uplinks vswitch0 128 9 128 1500 vmnic0 DVS Name Num Ports Used Ports Configured Ports MTU Uplinks mininet 1024 13 1024 1500 vmnic2 Step 3 VEM Agent (vemdpa) is running b) If the VEM is not up, call Cisco TAC. Verify that the Virtual Extensible LAN (VXLAN) tunnel endpoint (VTEP) acquired a valid DHCP IP address. a) Enter the following command: esxcfg-vmknic -l Step 4 avs-instance# esxcfg-vmknic -l Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type vmk0 Management Network IPv4 10.30.13.55 255.255.254.0 10.30.13.255 f8:72:ea:a4:97:0a 1500 65535 true STATIC vmk0 Management Network IPv6 fe80::fa72:eaff:fea4:970a 64 f8:72:ea:a4:97:0a 1500 65535 true STATIC, PREFERRED vmk1 11 IPv4 10.0.28.93 255.255.0.0 10.0.255.255 00:50:56:61:ef:11 1500 65535 true DHCP vmk1 11 IPv6 fe80::250:56ff:fe61:ef11 64 00:50:56:61:ef:11 1500 65535 true STATIC, PREFERRED Note Alternatively, you can check the VTEP port group in the vcenter GUI. b) If the VTEP is not configured properly, remove the host from the DVS and add it back in as described in Step 1a. c) If removing and readding the host does not correct the problem, call Cisco TAC. Verify the switch opaque data. a) Display the switch opaque data by entering the following command: vemcmd show sod avs-instance# vemcmd show sod Switch Opaque Data data-version 2.0 control-protocol open-flex open-flex port 8000 open-flex ipaddr 10.0.0.30 ftep ipaddr 10.0.0.32 dvs-name comp/prov-vmware/ctrlr-[mininet]-o3-vcenter/sw-dvs-2923 profile dvportgroup-2925 encap vlan 2 profile dvportgroup-2925 capability open-flex profile dvportgroup-2924 port-channel active 14
OpFlex OpFlex Connection Sequence profile dvportgroup-2924 mtu 9000 server-data type crt length 952 Switch opaque data is bootstrapping information for the host that is given through vcenter by the APIC. The system shows the port channel status as one of the following: active passive mac-pinning static The status should be same as configured on the APIC as shown in the example. Step 5 b) If the switch opaque data is incorrect (for example, the port channel is not in the state specified in the configuration), call Cisco TAC. Verify the uplinks by completing the following steps: The uplinks and the VTEP should be in the forwarding state. The port channel local target logic (PC-LTL) number of the uplink port should be nonzero. The corresponding port channel port also should be in the forwarding state. a) Enter the following command and examine its output: vemcmd show port avs-instance# vemcmd show port LTL VSM Port Admin Link State Cause PC-LTL SGID Vem Port Type 19 Eth1/3 UP UP FWD - 561 0 vmnic2 49 UP UP FWD - 0 0 vmk1 561 Po1 UP UP FWD - 0 The example output shows that the uplink port (here named vmnic2) and the VTEP (vmk1) are in the forwarding state. This is indicated by Admin UP, Link UP and State FWD. The example also shows a port channel in the FWD state and that the PC-LTL number of the uplink port is 561. The system also shows the ports used by each connection. b) If the port data is incorrect (for example, a port is not in forwarding state), call Cisco TAC. OpFlex Connection Sequence You can use the vemcmd show openflex command to view the state of the OpFlex connection. Normal OpFlex Connection Sequence The following table describes the normal sequence for a successful OpFlex connection. The messages appear quickly one after the other and end with the oplex_channel_active (12) message. Table 1: Normal OpFlex Connection Sequence OpFlex State opflex_channel_discovering (0) Description This state is the initial Opflex state. 15
OpFlex Connection Sequence OpFlex OpFlex State opflex_channel_send_discover (1) opflex_channel_recv_discover (2)/opflex_CHANNEL_DISCOVERED (3) opflex_channel_connecting (6) opflex_channel_send_hello (7) opflex_channel_send_id (9)/opflex_CHANNEL_SEND_FUNCTION (10) opflex_channel_connected (11) opflex_channel_active (12) Description The Opflex client sends a DISCOVER message to the anycast IP address. (The anycast IP address can be found by entering vemcmd show sod; it is the open-flex ipaddr.) The DISCOVER message (reply) is received from the leaf. The client sends a CONNECT message (similar to the SEND_DISCOVER message but this is for the actual OpFlex connection) after the initial discovery phase. The client sends a HELLO message to the IFM on the leaf. This is the first message after the SSL session is up for IFM. The ID and the FUNC data are sent to the leaf in these states. Once the ACKs are received, the state moves to ACTIVE. OpFlex is UP. Solutions to OpFlex Channel State Issues OpFlex can get stuck during discovery and connection. The following table lists the abnormal states that can be viewed in the output of the vemcmd show openflex command, their causes, and solutions. OpFlex State opflex_channel_send_discover (1) opflex_channel_recv_discover (2) opflex_channel_discovered (3) opflex_channel_version_mismatch (4) Cause A network issue probably has occurred. The client code has a problem. SSL keys might be missing. An IFM version mismatch due to software incompatibility has occurred. Solution Check the network. If there are no network issues, reboot the leaf. Call Cisco Technical Assistance Center (TAC) support. Call Cisco TAC support. Call Cisco TAC support. 16
OpFlex Recovering the Connection with APIC after Deleting an OpFlex vmknic OpFlex State Cause Solution opflex_channel_disconnected (5)/opflex_CHANNEL_CONNECTION_ATTEMPT (14) Network connectivity is down between the vleaf and the leaf switch. Check that the vmknic is up and has a valid IP address and that the anycast IP address is reachable by a ping. opflex_channel_send_id (9)/opflex_CHANNEL_SEND_FUNCTION (10) opflex_channel_inactive (13) opflex_channel_invalid_dvs (15) The images are incompatible. The contents of the ACK are inconsistent. There is a difference in the distributed virtual switch (DVS) ID between the leaf switch and the vleaf. Call Cisco TAC support. Call Cisco TAC support. Call Cisco TAC support. Recovering the Connection with APIC after Deleting an OpFlex vmknic An OpFlex vmknic provides the connection between the Cisco Application Policy Infrastructure Controller (APIC) and Cisco Application Virtual Switch (AVS) in addition to balancing data traffic. If you accidentally delete the OpFlex vmknic, you can loose the connection between the Cisco APIC and the Cisco AVS. If you have more than one vmknic, the second vmknic takes over as the OpFlex vmknic. The connection between the Cisco APIC and the Cisco AVS might not be reestablished immediately. However, if you use a MAC pinning policy on the Cisco AVS and have more than one vmknic, you can recover the connection quickly by resetting DHCP on the takeover vmknic. Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Log in to the Cisco AVS and enter the following command: vemcmd show opflex. The output shows the status as 12 (Active) if the second vmknic took over as the OpFlex vmknic and the connection between the Cisco AVS and Cisco APIC was reestablished. However, if the status is 4 (Disconnected), the connection is not yet reestablished even if the second vmknic took over. On the Cisco AVS console, enter the following command vemcmd show avs macpinning and note its output. The output displays the IP address of the vmknics and indicates which one took over as the OpFlex vmknic. Log in to vcenter. Click Home and then Hosts and Clusters. Choose the host, click the Configuration tab, click Networking, and then click vsphere Distributed Switch. Click Manage Virtual Adapters. Give the vmknic that took over as the OpFlex vmknic a static IP address by completing the following steps: a) In the Manage Virtual Adapters dialog box, choose the vmknic that took over as the OpFlex vmknic. b) Click Edit. 17
Recovering from an OpFlex Failure Due to a Certificate Issue OpFlex c) In the Edit Virtual Adapter dialog box, click the IP Settings tab. d) Click the Use the following IP settings radio button, and then enter a new IP Address field and a new subnet in the Subnet Mask field. e) Click OK. Step 8 Step 9 Step 10 Enable the vmknic that took over as the OpFlex vmknic to obtain IP settings automatically by completing the following steps: a) In the Manage Virtual Adapters dialog box, choose the vmknic that took over as the OpFlex vmknic. b) Click Edit. c) In the Edit Virtual Adapter dialog box, click the IP Settings tab. d) Click the Obtain IP settings automatically radio button. e) Click OK. Wait for about 30 seconds for the settings to change and for OpFlex to recover. Verify that the OpFlex vmknic is active by entering the following command: vemcmd show avs macpinning The output displays the IP address of the vmknics and which is the OpFlex vmknic. Recovering from an OpFlex Failure Due to a Certificate Issue When you upgrade previous releases of Cisco ACI to 1.2(2g), OpFlex might fail to come up on your ESXi host on Cisco AVS because of a certification problem. You can recover your setup by following the procedure in this section. The recovery procedure requires that you perform tasks on VMware vcenter and the AVS host, Cisco APIC, and leaf and spine switches in the Cisco ACI fabric. Before You Begin Make sure that the certificates on Cisco APIC, Cisco AVS, and the leaf and spine switches really are mismatching. Log in as root to Cisco APIC, each of the leaf and spine switches, and Cisco AVS and enter the following commands: Cisco APIC: root@apic1:~# cksum /securedata/vssl/server.* Leaf and spine switches: leaf1# cksum /securedata/vssl/server.* Cisco AVS hosts: [root@localhost:~] cksum /tmp/server.* 18
OpFlex Recovering from an OpFlex Failure Due to a Certificate Issue Export the configuration in Cisco APIC with Advanced Encryption Standard (AES) encryption. See the Cisco ACI Basic Configuration Guide and the KB article Importing and Exporting Configuration Files for information about using configuration file encryption. Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Log in to vcenter using the VMware vsphere Client. Using Virtual Machine Properties, release the VM from the port group created for the Cisco AVS host. Choose Inventory > Hosts and Clusters > Host*1 > Configuration > Networking > vsphere Distributed Switch > Manage Virtual Adapters > vmk1 > Remove. This step removes the vmk NIC. Note In Step 3 though Step 6, *1 indicates the host name of IP address configured for the AVS host, and *2 indicates the distributed virtual switch where the Cisco AVS host is attached. Choose Inventory > Networking > DVS*2 folder > DVS*2 > Hosts, right-click Host*1 and then choose Remove vsphere Distributed Switch. This step removes the host from the DVS. Choose Inventory > Networking > DVS*2 folder and then right-click DVS*2 and choose Remove. Choose Inventory > Networking, right-click the DVS*2 folder and choose Remove. Step 5 and Step 6 remove the DVS in vcenter. Log in to the Cisco AVS host as root. Remove certificate files if they exist: Step 9 Step 10 [root@localhost:~] rm /tmp/server.* Log in to the Cisco APIC as root. Remove the ODev key file. Step 11 root@apic1:~# rm /data/odev_keys_created Clean the configuration data. Step 12 root@apic1:~#acidiag touch clean Reload Cisco APIC. Step 13 Step 14 root@apic1:~#reboot On one of the leaf or spine switches in the fabric, log in as root. Remove the ODev key file if it exists. leaf1# rm /data/odev_keys_created Step 15 Clean the configuration data. 19
Recovering from an OpFlex Failure Due to a Certificate Issue OpFlex Step 16 leaf1# setup-clean-config.sh Reload the switch. Step 17 Step 18 Step 19 Step 20 Step 21 leaf1# vsh -c 'reload' Repeat Step 13 through Step 16 on each of the other leaf and spine switches in the fabric. Log in to Cisco APIC in a web browser. Complete the fabric membership. See the sections for registering unregistered switches in the Cisco APIC Getting Started Guide. Create an import policy with AES encryption. Import the configuration into Cisco APIC. What to Do Next You now need to verify the certificates for Cisco APIC, the leaf and spine switches, and Cisco AVS hosts. Complete the following steps: 1 Log in to Cisco APIC and each fabric switch as root. 2 Verify the certificates in Cisco APIC and on the fabric switches, as shown in the following examples: root@apic1:~# cksum /securedata/vssl/server.* leaf1# cksum /securedata/vssl/server.* Note The keys in Cisco APIC and on the switches must be the same. They will appear as follows: xxxxxxxx 812 /securedata/vssl/server.crt xxxxxxxx 637 /securedata/vssl/server.csr xxxxxxxx 889 /securedata/vssl/server.key 3 Create or import configurations for the interface policy, switch profile, and VMM domain. 4 In vcenter, add the Cisco AVS host to the DVS. 5 Verify the certificates on the Cisco AVS host, as shown in the following example: [root@localhost:~] cksum /tmp/server.* 6 Repeat Step 5 on the other Cisco AVS hosts. Note The keys in Cisco APIC and the Cisco AVS hosts must be the same. 7 Verify that OpFlex is active. 20
CHAPTER 5 Ports, Endpoint Groups, and Layer 2 This chapter contains the following sections: About Data Paths, page 21 Diagnosing Port Activity, page 22 Troubleshooting Unavailable Ports, page 23 Checking Port Synchronization Using Port Counters, page 24 Troubleshooting Endpoint Groups, page 25 Recovering from Endpoint Group Creation Failure, page 26 Troubleshooting Layer 2 Switching, page 26 About Data Paths Before a switch can use interfaces to relay frames from one data link to another, you must define the characteristics of the sending and receiving interfaces. The configured interfaces can be Ethernet (physical) interfaces and virtual Ethernet (veth) interfaces. On the virtual side of the switch, three layers of ports are mapped together: Virtual NICs VMware has two types of virtual NICs. The virtual NIC (vnic) is part of the virtual machine (VM) and represents a VM virtual port that is plugged into the virtual switch. The virtual kernel NIC (vmknic) is used by the hypervisor for management, VMotion, iscsi, Network File System (NFS), and other network access needed by the kernel. This interface carries the IP address of the hypervisor itself, and is also bound to a veth port. veth Ports A veth port is a port on the Cisco Application Virtual Switch (AVS). VEth ports are allocated to hosts that run VMs. VEth ports are assigned to port groups. 21
Diagnosing Port Activity Ports, Endpoint Groups, and Layer 2 Local Virtual Ethernet ports (lveth) Each host has a number of lveth ports. These ports are dynamically allocated as needed. Three types of ports are on the physical side of the switch. From bottom to top, they are: Virtual machine NICs Each physical NIC in VMware is represented by an interface called a vmnic. The vmnic number is allocated during VMware installation (or when a new physical NIC is installed) and remains the same for the life of the host. Uplink ports Uplink ports associate port configuration with vmnics. Each uplink port on the host represents a physical interface. Because physical ports do not move between hosts, there is a 1:1 mapping between uplink ports and vmnics. Uplink ports are managed entirely by VMware. Physical ports Each physical port added to a Cisco AVS appears as a physical Ethernet port, just as it would on a hardware-based switch. Each interface, regardless of type, has the following characteristics: Administrative configuration You can set configuration attributes. The administrative configuration does not change unless you modify it using the Application Policy Infrastructure Controller (APIC). Operational state The operational state includes such attributes as the interface speed. These values are read-only; they cannot be changed. Some values might not be valid when the interface is down (for example, the operation speed). Diagnosing Port Activity You can diagnose port activity by examining the following: Administrative state Speed Trunk VLAN status Number of frames sent and received 22
Ports, Endpoint Groups, and Layer 2 Troubleshooting Unavailable Ports Transmission errors, including discards, errors, cyclic redundancy checks (CRCs), and invalid frames Step 1 Step 2 Step 3 Verify that the host is connected to the fabric entering the vemcmd show openflex command and examining its output as described in Enabling OpFlex, on page 13. In vsphere Web Client connected to the vcenter Server, verify that the correct port profiles are assigned to the physical and virtual NICs. On an ESXi hypervisor console, verify that the ports have been created by entering the following command: vemcmd show port avs-instance# vemcmd show port LTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port 20 Eth1/4 UP UP FWD - 1039 3 0 0 vmnic3 49 UP UP FWD - 0 2 0 0 vmk1 50 UP UP FWD - 0 2 0 0 orion3-vm2.eth1 51 UP UP FWD - 0 3 0 0 orion3-vm1.eth1 1039 Po1 UP UP FWD - 0 0 0 The output of the command should look like the example, with Admin, Link, and State equal to UP, UP, and FWD for each port. Step 4 If the port states are incorrect, see Troubleshooting Unavailable Ports, on page 23. Troubleshooting Unavailable Ports If ports are not coming up, enter the vemcmd show port command to diagnose the cause. The following table lists the possible causes and solutions for troubleshooting unavailable ports. Output from vemcmd show port WAIT ACK or WAIT EPG WAIT ACK or WAIT EPG (same as above) Zero Mac Possible Cause The port is coming up and is waiting for the endpoint group (EPG) to be downloaded. The OpFlex control channel is not online. The interface on the virtual machine (VM) is down. Solution Wait for the port to come up (approximately 10 minutes). Troubleshoot the OpFlex control channel as described in Enabling OpFlex, on page 13. Bring up the interface manually on the VM, and then edit the appropriate configuration files to make sure the interface comes up during bootup (or after a reboot). 23
Checking Port Synchronization Using Port Counters Ports, Endpoint Groups, and Layer 2 Output from vemcmd show port BPDU Viol Possible Cause Bridge Protocol Data Unit (BPDU) Guard feature is enabled, and BPDUs are received from the VM. Solution The port will be down in an error disabled state for 30 seconds but should come up after that. To avoid this error, make sure that no BPDUs are sent from the VM or disable BPDU Guard. In all cases listed above, if the solution does not enable the port to come up, detach the VM from the Cisco Application Virtual Switch (AVS) and then reattach it. If the issue persists, contact Cisco TAC. Checking Port Synchronization Using Port Counters Counters can show synchronization problems by revealing a large disparity between received and transmitted frames. Step 1 Step 2 Create a baseline by clearing the counters by entering the following command: vemcmd clear stats Note The values stored in the counters are meaningless for a port that has been active for an extended period. Clearing the counters provides a profile of the actual link behavior over a known period of time. Display the total number of packets sent and received by entering the following command: vemcmd show stats Step 3 avs-instance# vemcmd show stats LTL Received Bytes Sent Bytes Txflood Rxdrop Txdrop Name 8 3 202 0 0 0 0 0 9 0 0 3 202 3 0 0 10 10 772 7 420 7 0 0 12 7 420 10 772 10 0 0 16 5 582 0 0 0 0 0 ar 19 935 187513 456 48497 11 7 0 vmnic2 20 830 170397 355 37063 21 7 0 vmnic3 49 743 81212 714 137646 0 0 0 vmk1 50 60 4816 44 3856 0 0 0 orion3-vm2.eth1 51 45 3688 46 3748 0 0 0 orion3-vm1.eth1 1039 1004 201869 457 48557 32 13 0 Display a breakdown of packets into unicast, broadcast, multicast, and flood by entering the following command: vemcmd show packets. avs-instance# vemcmd show packets LTL RxUcast TxUcast RxMcast TxMcast RxBcast TxBcast Txflood Rxdrop Txdrop RxJumbo TxJumbo Name 19 1033 596 835 3 104 445 18 94 0 0 0 vmnic2 20 0 0 835 0 104 11 9 0 0 0 0 vmnic3 49 588 1027 5 0 430 11 0 0 0 1 0 vmk1 50 8 6 0 0 8 107 107 0 0 0 1 orion3-vm4.eth1 51 0 0 0 0 8 107 107 0 0 0 0 orion3-vm3.eth1 1039 2066 1192 3337 6 416 912 54 188 0 0 0 24
Ports, Endpoint Groups, and Layer 2 Troubleshooting Endpoint Groups Troubleshooting Endpoint Groups In the Cisco Application Virtual Switch (AVS), an endpoint group (EPG) is an entity that is assigned multiple interfaces, giving them all the same configuration. Changes to an EPG configuration are propagated automatically to all interfaces that are assigned to it. In vcenter Server, an EPG is represented as a port group. The virtual Ethernet (veth) interfaces are assigned in vcenter Server to an EPG in order to do the following: Define the port configuration by the policy. Apply a single policy across a large number of ports. EPGs that are configured as uplinks can be assigned by the server administrator to physical ports (which can be vmnics or PNICs). EPGs that are not configured as uplinks can be assigned to a virtual machine (VM) virtual port. For more information about assigning EPGs, see your VMware documentation. Step 1 Step 2 Enter the following command: echo dump profile_cfg > /tmp/dpafifo Enter the following command and check the output of the file: vi /var/log/vemdpa.log Profile: alias: dvportgroup-3228 pp_id 3228 flags 0 mode: Trunk admin_state: no shut mtu 9000 allowed_vlans: 1-4095 EPP Switching mode: NS EPP Encap : VLAN 1 EPP seg id 0 EPP reused 0 EPP seg arp flood 0 chan: mode on sg_type mac-pinning Ports: cnt 3 19 20 561 Profile: alias: dvportgroup-3229 pp_id 3229 flags 20 mode: Access admin_state: no shut access_vlan 2 EPP Switching mode: NS EPP Encap : VLAN 4093 EPP seg id 4093 EPP reused 0 EPP seg arp flood 0 Ports: cnt 1 49 Profile alias: dvportgroup-3230 pp_id 3230 flags 0 mode: Access admin_state: no shut access_vlan 3 EPP Switching mode: LS 25
Recovering from Endpoint Group Creation Failure Ports, Endpoint Groups, and Layer 2 EPP Encap : VLAN 39 EPP seg id 15433636 EPP reused 1 EPP seg arp flood 1 Ports: cnt 2 50 51 All the ports associated with a port group should belong to the same profile. In the first example above, dvportgroup-3228 is the port group, and 19, 20, and 561 are the ports. Recovering from Endpoint Group Creation Failure After you create a virtual machine manager (VMM) domain on the Application Policy Infrastructure Controller (APIC) GUI, configuration mistakes can prevent endpoint groups (EPGs) from being created. If the EPGs that you create on the APIC GUI do not appear under the Cisco Application Virtual Switch (AVS), use the following procedure: Step 1 Step 2 On the APIC GUI, ensure that the VMM domain that is created in vcenter is associated with the correct EPG group. Ensure that a large enough address pool exists to support all the EPGs you have defined. a) In VLAN mode, ensure that the VLAN pool contains a number of VLANs equal to or greater than the number of EPGs that you defined. b) In Virtual Extensible LAN (VXLAN) mode, ensure that the multicast IP pool contains a number of multicast IP addresses equal to or greater than the number of EPGs that you defined. Troubleshooting Layer 2 Switching You can troubleshoot connections between two Layer 2 endpoints. Step 1 Verify that the switch mode is correct by completing the following steps: a) Verify that OpFlex is online by entering the following command: vemcmd show openflex A status of 12 (Active) indicates that OpFlex is online. b) Move profile information into an output file by entering the following command: echo dump profile_cfg > /tmp/dpafifo c) Display the resulting output file by entering the following command and examine its output: vi /var/log/vemdpa.log The output should show valid virtual extensible LAN (VXLAN) network IDs (VNIDs) and endpoint group (EPG) multicast addresses. The EPG multicast addresses should be from the pool to which they were assigned earlier in the Application Policy Infrastructure Controller (APIC). 26
Ports, Endpoint Groups, and Layer 2 Troubleshooting Layer 2 Switching Step 2 Verify the configuration and status of the ports by completing the following steps: a) Display port information by entering the following command: vemcmd show port avs-instance# vemcmd show port LTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port 20 Eth1/4 UP UP FWD - 1039 3 0 0 vmnic3 49 UP UP FWD - 0 2 0 0 vmk1 50 UP UP FWD - 0 2 0 0 orion3-vm2.eth1 51 UP UP FWD - 0 3 0 0 orion3-vm1.eth1 1039 Po1 UP UP FWD - 0 0 0 The output of the command should look like the example, with Admin, Link, and State equal to UP, UP, and FWD, respectively for each port. b) On the Cisco Application Virtual Switch (AVS) instances that are attached to the two endpoints, enter the following command: vemcmd show port vlans The VLAN or VXLAN tags at either endpoint should match each other. Step 3 Step 4 Step 5 For inter-epg traffic, use the APIC GUI to ensure that contracts are correctly configured between the EPGs. See the Cisco APIC Getting Started Guide and the Cisco APIC Layer 4 to Layer 7 Services Deployment Guide for information about configuring contracts. For traffic issues involving VXLAN encapsulation, ensure that the IGMP querier and IGMP snooping policy are configured in the APIC GUI under tenant infra and infra bridge domain. See the Cisco Application Virtual Switch Configuration Guide for information about configuring IGMP querier and IGMP snooping policy. For VXLAN mode, verify that IGMP joins were sent out correctly from the Cisco AVS for the EPGs attached to the endpoints: a) Enter the following command: vemcmd show epp multicast # vemcmd show epp multicast Number of Group Additions 1 Number of Group Deletions 0 Multicast Address EPP Ref Count 225.2.1.92 1 b) If no group is present, ensure that the multicast pool for the virtual machine manager (VMM) domain contains at least as many multicast addresses as there are EPGs. The multicast-to-epg association is one-to-one; therefore, a multicast pool that is too small prevents some EPGs from being created. If a new multicast pool is created and associated with the VMM domain, then all the EPGs need to be disassociated from that VMM domain and associated back. Step 6 Verify that the port channel is up and that it matches the upstream port channel configuration by completing the following steps: a) Enter the following command: vemcmd show pc 27
Troubleshooting Layer 2 Switching Ports, Endpoint Groups, and Layer 2 # vemcmd show pc pce_ind chan pc_ltl pce_in_pc LACP SG_ID NumVethsPinned mbrs ------- ---- ------ --------- ---- ----- -------------- ---- 0 1 561 0 N 2 1 19, 3* 2 20, * denotes a designated sub-group b) If the port channel type is static, make sure that all the interfaces that belong to the leaf switches that are associated with the interface policy group are added to the Cisco AVS. c) If the trouble still exists or if there are upstream issues, consult the Cisco ACI Troubleshooting Guide. 28
CHAPTER 6 Port Channels This chapter contains the following sections: Port Channel Overview, page 29 Verifying Port Channels, page 29 Troubleshooting Port-Channel Creation, page 33 Port Channel Overview Port channels aggregate multiple physical interfaces into one logical interface to provide higher bandwidth, load balancing, and link redundancy. Cisco AVS supports LACP, MAC pinning, and static port channels in standalone and virtual port channel (VPC) mode. A port channel performs the following functions: Increases the aggregate bandwidth on a link by distributing traffic among all links in the channel. Maintains optimal bandwidth usage by load balancing across multiple links. Provides high availability. If one link fails, its traffic is switched to the remaining links. Higher-level protocols are unaware of the failed link, although bandwidth is diminished. The MAC address tables are not affected by link failure. Verifying Port Channels Step 1 Display the ports on the Cisco Application Virtual Switch (AVS) by entering the following command: vemcmd show port # vemcmd show port LTL VSM Port Admin Link State Cause PC-LTL SGID ORG svcpath Type Vem Port 29
Verifying Port Channels Port Channels Step 2 Step 3 21 Eth1/5 UP UP FWD - 1039 0 0 0 vmnic4 22 Eth1/6 UP UP FWD - 1039 0 0 0 vmnic5 23 Eth1/7 UP UP FWD - 1039 0 0 0 vmnic6 24 Eth1/8 UP UP FWD - 1039 0 0 0 vmnic7 49 UP UP FWD - 0 0 0 0 vmk1 50 UP UP FWD - 0 0 0 vmk2 1039 Po1 UP UP FWD - 0 0 0 Verify that the uplinks are in the FWD state rather than the BLK state (or, in the case of Link Aggregation Control Protocol (LACP), that uplinks are not in the Suspended [s] state or the Individual [I] state). Gather information about the remote physical ports by completing the following steps: a) (For directly connected hosts) Display information about each local target logic (LTL) number with physical ports (21 to 24 in the previous example) by entering the following command: vemcmd show lldp ltl # vemcmd show lldp 21 Chassis Id = 7c:69:f6:df:e4:f2 Port Id = Eth1/2 Extras: 3topology/pod-1/protpaths-101-102/pathep-[esx56-vpc] leaf2 topology/pod-1 b) For leaf switches connected to a Layer 2 cloud (either Cisco Nexus 5000 or fabric interconnect), enter the following command on the Nexus 5000 or fabric interconnect console: show lldp neighbors interface <interface_id> detail Note For Cisco UCS Manager fabric interconnect (FI) nodes, you must connect to Cisco NX-OS. Step 4 AVS-N5K# show lldp neighbors interface ethernet 1/29 detail Capability codes: (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other Device ID Local Intf Hold-time Capability Port ID Chassis id: 5087.89d3.ce3f Port id: Eth1/5 Local Port id: Eth1/29 Port Description: topology/pod-1/protpaths-105-106/pathep-[n5k-vpc1] System Name: scale-leaf4 System Description: topology/pod-1/node-106 Time remaining: 100 seconds System Capabilities: B, R Enabled Capabilities: B, R Management Address: 5087.89d3.ce3f Vlan ID: not advertised Total entries displayed: 1 For each output above, note the remote physical port (Port Id; for example Eth1/2 in substep a and Eth1/5 in substep b of the preceding step), the policy group (for example esx56-vpc and N5K-VPC1), and the remote switch (leaf) name (leaf2 and scale-leaf4). Note which physical port and policy group are associated with which leaf switch. Step 5 Display VPC information by entering the following command on each leaf switch identified in Step 3 a and Step 3 b: show vpc leaf1# show vpc Legend: 30
Port Channels Verifying Port Channels (*) - local vpc is down, forwarding via vpc peer-link vpc domain id : 1 Peer status : peer adjacency formed ok vpc keep-alive status : Disabled Configuration consistency status : success Per-vlan consistency status : success Type-2 inconsistency reason : Consistency Check Not Performed vpc role : primary, operational secondary Number of vpcs configured : 4 Peer Gateway : Disabled Dual-active excluded VLANs : - Graceful Consistency Check : Enabled Auto-recovery status : Enabled (timeout = 240 seconds) Operational Layer3 Peer : Disabled vpc Peer-link status --------------------------------------------------------------------- id Port Status Active vlans -- ---- ------ -------------------------------------------------- 1 up - vpc status ---------------------------------------------------------------------- id Port Status Consistency Reason Active vlans -- ---- ------ ----------- ------ ------------ 1 Po3 up success success 4090 2 Po1 up success success 4090 3 Po4 up success success 4090 Step 6 Step 7 343 Po2 down* success success - In the previous command result, on the leaf and intermittent Layer 2 switches, verify that the VPC status is success. On each leaf switch, enter the following command: show vpc role scale-leaf3# show vpc role vpc Role status ---------------------------------------------------- vpc role : secondary Dual Active Detection Status : 0 vpc system-mac : 00:23:04:ee:be:02 vpc system-priority : 32667 vpc local system-mac : 50:87:89:a2:53:59 vpc local role-priority : 106 scale-leaf3# scale-leaf4# show vpc role Step 8 Step 9 vpc Role status ---------------------------------------------------- vpc role : primary Dual Active Detection Status : 0 vpc system-mac : 00:23:04:ee:be:02 vpc system-priority : 32667 vpc local system-mac : 50:87:89:d3:ce:71 vpc local role-priority : 105 scale-leaf4# In the previous command result, verify that there is one primary and one secondary role per VPC, similar to the example in Step 7. On each leaf switch, display port channel information by entering the following command: show port-channel summary 31
Verifying Port Channels Port Channels Step 10 leaf1# show port-channel summary Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 1 Po1(SU) Eth LACP Eth1/1(P) Eth1/2(P) 2 Po2(SU) Eth LACP Eth1/19(P) Eth1/20(P) 3 Po3(SU) Eth LACP Eth1/33(P) 4 Po4(SU) Eth LACP Eth1/34(P) leaf1# In the previous command result, verify that the protocol and the status of each member port (the flag, in parentheses after the port name) are correct for the port channel type: Port Channel Type static Protocol NONE Status Flag P LACP MACPIN LACP LACP P I Step 11 Step 12 Any other protocol or status value indicates a configuration problem. Note LACP automatically negotiates both ends of the port channel configuration, so correct protocol and status indicate success. For static and MAC pinning port channels, it is possible to have a misconfiguration and still show the correct protocol and status. The SD (Down) status is the expected behavior for a port channel with mac pinning. The remaining steps describe how to identify misconfigured port channels. Compare the outputs of the vemcmd show lldp ltl (Step 4) and show port-channel summary commands (Step 9). The outputs should show the same physical ports for a given leaf. If you find misplaced physical ports in the diagnostic steps above, log onto the Cisco Application Centric Infrastructure (APIC) GUI and fix them, as follows: a) Go to Fabric > Access Policies > Interface Profiles > Policy Groups. Find the profile corresponding to the policy groups found in the vemcmd show lldp <ltl> command. b) Go to Fabric > Access Policies > Interface Profiles > Profiles. c) For each profile that corresponds to the policy group identified above, change the configuration so that the policy reflects the actual port configuration. Alternatively, you can fix the misconfiguration by changing the physical port connections to agree with the connections that are specified in the profile. 32
Port Channels Troubleshooting Port-Channel Creation Troubleshooting Port-Channel Creation If port channel creation fails, you might have configured too many port channels. If necessary, reconfigure your system to require only eight port channels. Only eight uplinks are supported on one system. 33
Troubleshooting Port-Channel Creation Port Channels 34
CHAPTER 7 Switched Port Analyzer This chapter contains the following sections: About the Switched Port Analyzer, page 35 Viewing the Switched Port Analyzer Configuration, page 36 Troubleshooting the Switched Port Analyzer, page 37 About the Switched Port Analyzer The Switched Port Analyzer (SPAN), sometimes called port mirroring or port monitoring, selects network traffic for analysis by a network analyzer. Two types of SPAN are supported: SPAN (local SPAN) that can monitor sources within a host. Encapsulated Remote SPAN (ERSPAN) that can send monitored traffic to an IP destination. For detailed information about how to configure SPAN, see the chapter on configuring SPAN in Cisco Application Virtual Switch Configuration Guide. The interfaces from which traffic can be monitored are called SPAN sources. Traffic can be monitored in the receive direction, the transmit direction, or both directions for virtual Ethernet source interfaces (endpoints) or (EPGs). Receive source (Rx) Traffic that enters the switch through this source port is copied to the SPAN destination port. Transmit source (Tx) Traffic that exits the switch through this source port is copied to the SPAN destination port. 35
Viewing the Switched Port Analyzer Configuration Switched Port Analyzer Source Ports The Cisco Application Virtual Switch (AVS) supports multiple source ports. A source port has these characteristics: Cannot be a destination port. Can be configured to monitor the direction of traffic (receive, transmit, or both). (For local SPAN only): Must be on the same host as the destination port. SPAN Destinations The Cisco AVS supports only virtual Ethernet (veth) interfaces (endpoints) as SPAN destinations. Destination Ports Each local SPAN session must have at least one destination port (also called a monitoring port) that receives a copy of traffic from the source ports. A destination port has these characteristics: Cannot be a source port. Receives copies of transmitted and received traffic for all monitored source ports. If a destination port is oversubscribed, it can become congested. This congestion can affect traffic forwarding on one or more of the source ports. (For local SPAN only): Must be on the same host as the source port. ERSPAN Destinations ERSPAN destinations refer to an IP address to which the monitored traffic is sent. The destination IP should be in overlay-1 (infra VRF) and be reachable through the configured ERSPAN-enabled vmknic (which is also the VXLAN tunnel endpoint [VTEP]) on the host. For detailed information about how to configure ERSPAN, see the Cisco Application Virtual Switch Configuration Guide. SPAN Sessions You can create up to 64 SPAN and ERSPAN sessions to define sources and destinations on the local device. Viewing the Switched Port Analyzer Configuration To display the Switched Port Analyzer (SPAN) information, enter the following command: show span The following example output shows the expected configurations for Encapsulated Remote SPAN (ERSPAN): # vemcmd show span VEM SOURCE IP: 10.0.0.16 HW SSN ID ERSPAN ID HDR VER DST LTL/IP 1 1 2 10.0.10.10 36
Switched Port Analyzer Troubleshooting the Switched Port Analyzer RX Ltl Sources :50,58,59,60,61,62, TX Ltl Sources :50,58,59,60,61,62, Troubleshooting the Switched Port Analyzer Switched Port Analyzer Requirements A running Switched Port Analyzer (SPAN) session must meet these requirements: A maximum of 64 SPAN sessions can run at one time. At least one operational source has been configured. At least one operational destination has been configured. The configured source and destination are on the same host (for local SPAN). A port cannot be configured as both a source port and a destination port. The static client end point (CEP) has been configured accordingly on the right leaf (for Encapsulated Remote SPAN [ERSPAN]). The ERSPAN destination host is reachable through the host s VXLAN tunnel endpoint (VTEP) (for ERSPAN). When a SPAN session contains multiple transmit source ports, packets that these ports receive can be replicated even though they are not transmitted on the ports. Some examples of this behavior on source ports are as follows: Traffic that results from flooding Broadcast and multicast traffic A session is stopped if any of the following events occur: All the source ports are removed. All the destination ports are removed. All the source and destination ports are separated by a VMotion live migration. After VMotion, the following might occur: A session is stopped if the source and destination ports are separated. A session resumes if the source and destination ports end up on the same host. Troubleshooting If you encounter problems with SPAN, consult the following table for symptoms and solutions. 37
Troubleshooting the Switched Port Analyzer Switched Port Analyzer Symptom The vemcmd show span command does not show the configuration. The ERSPAN session is configured but does not see packets at the destination. Possible Cause Packets are not being spanned to a local destination or a remote destination. The ERSPAN destination is not on the overlay-1 virtual routing and forwarding (VRF). Static CEP is not configured. Solution Verify that SPAN is configured properly on the Cisco Application Policy Infrastructure Controller (APIC).. Verify that the ports on which SPAN is enabled are UP and in the Forwarding state as described in Troubleshooting Unavailable Ports. Make sure that the ERSPAN destination is hosted in the overlay-1 VRF. ERSPAN is supported in this VRF only. If the ERSPAN destination is a virtual machine (VM) on the Cisco Application Virtual Switch (AVS), make sure that it is using the VTEP endpoint group (EPG). Complete the following actions: 1 Verify that the static CEP is configured with the following: The ERSPAN destination MAC address The ERSPAN IP address The overlay-1 VLAN Type equal to tunnel endpoint (tep) The interface policy group that identifies the leaf switches behind which the ERSPAN destination VM is located 2 If the static CEP is configured but still not working, try deleting and readding the static CEP. The ERSPAN destination VM has an IP address that is not in the same subnet as that of other VTEPs in the fabric. Ensure that the ERSPAN destination VM has an IP address in the same subnet as that of the other VTEPs in the fabric. operst does not show as up. (Log into visore on the leaf switch, find tunnelif, and look for the ERSPAN destination IP.) The static CEP tunnel is not up on the leaf switches. After the static CEP is configured, ping any overlay-1 IP address (such as, 10.0.0.30) to force the fabric to learn the ERSPAN destination IP address. Without these initial pings, the tunnel goes down and ERSPAN fails. 38
Switched Port Analyzer Troubleshooting the Switched Port Analyzer Symptom The ERSPAN destination VM does not get an IP address through DHCP on the VTEP port group. Possible Cause The DHCP requests do not contain Option 61. Solution Configure the VM to add Option 61 to the DHCP requests. 39
Troubleshooting the Switched Port Analyzer Switched Port Analyzer 40
CHAPTER 8 System Troubleshooting This chapter contains the following sections: VEM Commands, page 41 VEM Commands This section lists some common commands for diagnosing the Virtual Ethernet Module (VEM) and controlling VEM kernel logging. VEM Troubleshooting Commands Use the following commands to display VEM information: vem status Collects status information. vem version Collects version information. vem-support all Collects support information. vemcmd Displays configuration and status information. 41
VEM Commands System Troubleshooting dump port ltl ltl_id useg summary Displays all information about a microsegment applied on the endpoint for a specific local target logic (LTL) number. [example@esx-console ~]# vemcmd dpa dump port ltl 61 useg summary =>dpa command is: dump port ltl 61 useg summary LTL : 61 Table ID : 15584356234873321078 Port MAC : 00:50:56:8a:8f:a9 VEM Port : AVS-CISCO-CL04-VM-14.eth0 Admin : UP Link : UP State : FWD Cause : - VLAN/VNID : 9338880 Last IP : 0.0.0.0 Parent EPG : dvportgroup-19629 VM Attr EPG : VS-CISCO-CL04-VM-14-EPG] Effective EPG : CISCO-CL-04-NEW-MAC-EPG] Multicast IP : 225.1.31.2 vemcmd dpa dump port useg summary Displays summary port Microsegmentation-related information. Combines various parameters from different commands:vemcmd show port, vemcmd show port vlans, vemcmd dpa dum useg, vemcmd show portmac, and vemcmd show microsegmentation tables brief. [example@esx-console ~]# vemcmd dpa dump port useg summary =>dpa command is: dump port useg summary LTL Table_id portmac vem-port state Cause vlan/vnid -------------------------------------------------------------------------------------------- 60 15584356234873321078 00:50:56:8a:48:84 AVS-CISCO-CL04-VM-02.eth0 FWD - 8912919 61 15584356234873321078 00:50:56:8a:9e:76 AVS-CISCO-CL04-VM-04.eth0 FWD - 8912919 62 15584356234873321078 00:50:56:8a:fb:6d AVS-CISCO-CL04-VM-24.eth0 FWD - 8912919 63 15584356234873321078 00:50:56:8a:56:74 AVS-CISCO-CL04-VM-29.eth0 FWD - 8912919 -------------------------------------------------------------------------------------------- LTL Parent-EPG VM-Attr-EPG Effective-EPG ----------------------------------------------------------------------- 51 /ap-ap1/epg-epg100] None n-te2/ap-ap1/epg-epg100] 52 /ap-ap1/epg-epg200] None n-te2/ap-ap1/epg-epg200] 60 e1/ap-ap1/epg-epg3] -AVS-CISCo-VMM-DOM-NEW5] ap-ap1/epg-ip-filter-1h] 61 e1/ap-ap1/epg-epg7] -AVS-CISCo-VMM-DOM-NEW5] ap-ap1/epg-ip-filter-1h] 62 e1/ap-ap1/epg-epg9] -AVS-CISCo-VMM-DOM-NEW5] ap-ap1/epg-ip-filter-1h] 63 1/ap-AP1/epg-EPG13] -AVS-CISCo-VMM-DOM-NEW5] ap-ap1/epg-ip-filter-1h] vemcmd help vemlog Displays the type of information you can display. [example@esx-console ~]# vemcmd help show card Show the card's global info show vlan [vlan] Show the VLAN/BD table show bd [bd] Show the VLAN/BD table show l2 <bd-number> Show the L2 table for a given BD/VLAN show l2 all Show the L2 table show port [priv vsm] Show the port table show pc Show the port channel table show portmac Show the port table MAC entries show trunk [priv vsm] Show the trunk ports in the port table show stats Show port stats Displays and controls VEM kernel logs. See the following section for specific logging commands. 42
System Troubleshooting VEM Commands vemlog show info Displays information about the log buffer setup. [example@esx-console ~]# vemlog show info Enabled: Yes Total Entries: 1092 Wrapped Entries: 0 Lost Entries: 0 Skipped Entries: 0 Available Entries: 6898 Stop After Entry: Not Specified vemlog show last number-of-entries Displays the specified number of log entries. [example@esx-console ~]# vemlog show last 5 Timestamp Entry CPU Mod Lv Message Oct 13 13:15:52.615416 1095 1 1 4 Warning vssnet_port_pg_data_ Oct 13 13:15:52.620028 1096 1 1 4 Warning vssnet_port_pg_data_ Oct 13 13:15:52.630377 1097 1 1 4 Warning svs_switch_state Oct 13 13:15:52.633201 1098 1 1 8 Info vssnet new switch Oct 13 13:16:24.990236 1099 1 0 0 Suspending log VEM Logging Commands Use the following commands to control the VEM kernel log during troubleshooting: vemlog clear Clears the log. vemlog resume Starts the log but does not clear the stop value. vemlog start number-of-entries Starts the log and stops it after the specified number of entries. vemlog stop Stops the log. vemlog stop number-of-entries Stops the log after the next specified number of entries. 43
VEM Commands System Troubleshooting 44