How to Deploy a Nexus 1000v lab with a single ESX host. By Robert Burns CCIE Data Center #37856 *Slight variation works with VMware Workstation/Fusion also. *Details of third party OS & application installation is beyond the scope of this guide. Pre-requisites: -Disable Windows firewalls on your client -Java must be installed Example Host List Host IP VM Name Description 10.85.49.215 Bare Metal ESX host 10.85.49.216 RHEL62-Test-1 Redhat test VM 10.85.49.217 vcenter-5 vcenter server 10.85.49.218 ESX5-Nested-1 Nested ESX 5.1 (VM 1) 10.85.49.219 ESX5-Nested-2 Nested ESX 5.1 (VM 2) 10.85.49.220 N1000v VSM Management IP 1. Infrastructure setup a. Install ESX bare metal host. Ensure you have > 75GB available space on your VMFS if you plan on installing a vcenter VM with Update Manager. b. Configure management network interface. c. Install one WIN2K8/2012 VM (for vcenter) allocated 30GB or more for virtual disk. Next install vcenter 5.1 on Windows VM. Alternately you can use the vcenter appliance. d. On the vcenter server install VMware Update Manager (VUM). e. Install two ESX 5.1 VMs following Nested ESX Instructions here: http://www.vcritical.com/2011/07/vmware-vsphere-can-virtualize-itself/ i. Configured four vnics each using the default vswitch VM Network port group for now. ii. Boot Nested ESX VMs and configure Management network interfaces from the VM console within VI Client. f. Test all IP connectivity between vcenter and all three ESX hosts. g. Add Nested ESX hosts to vcenter. See Fig. 1
Tip: Put the Nested ESX hosts in their own Cluster for easier management. Note: The two ESX5-Nested-X VMs below correspond to 10.85.49.218 & 10.85.49.219 hosts connected to vcenter in the ESX- Nested cluster. Fig. 1 Infrastructure setup with Nested ESX VMs installed & added to vcenter. 2. Deploy the 1000v VSM a. Download the latest 1000v SW from CCO. *For ESX 5.1 and later you must use 1000v version Nexus1000v.4.2.1.SV2.1.1 or later. http://software.cisco.com/download/release.html?mdfid=282646785&flowid=3090&softwa reid=282088129&release=4.2%281%29sv2%281.1a%29&relind=available&rellifecycle=&r eltype=latest
b. Unzip the bundle and navigate to the Install_App folder & launch the application. c. Start with the VSM Complete Installation selecting Custom install. d. Review the pre-reqs & click Next. e. Enter the appropriate details for your vcenter
f. Enter the appropriate info for your VSM. Since I m hosting the VSM pair on my single bare metal ESX host, I ve used the same ESX Host IP twice. Click Next when complete. Notes: - Recommend setting your Domain ID to something other than default 1. - I m setting my Management VLAN to what my physical switches use for their native VLAN. - I chose to use L2 mode for simplicity versus L3, but either will work. - Do not migrate hosts at this time. g. Review configuration and click Next. Be patient, deployment will take up to 15mins.
h. Next step will prompt to add additional Modules. i. Select the hosts you wish to have the VEM agent installed. Click Next. Note: This method requires VUM to be previously installed. If not, you ll need to manually install the VEM agent vibs. j. Review and then click Finish to proceed. You can monitor the progress from the VI Client Recent Task log
k. The Install App hopefully completed successfully for all hosts. **If the VEM installation fails, it likely points to a problem with VMware Update Manager (VUM). l. From the VI Client go to Home -> Inventory -> Networking and you should see your two new hosts as part of the 1000v DVS. Ensure you click on the 1000v DVS in the left pane. m. SSH into the VSM and and check the modules N1000v(config)# show mod Mod Ports Module-Type Model Status --- ----- -------------------------------- ------------------ ------------ 1 0 Virtual Supervisor Module Nexus1000V active * 2 0 Virtual Supervisor Module Nexus1000V ha-standby 3 248 Virtual Ethernet Module NA ok 4 248 Virtual Ethernet Module NA ok
Mod Sw Hw -- ------------------ ------------------------------------------------ 1 4.2(1)SV2(1.1) 0.0 2 4.2(1)SV2(1.1) 0.0 3 4.2(1)SV2(1.1) VMware ESXi 5.1.0 Releasebuild-838463 (3.1) 4 4.2(1)SV2(1.1) VMware ESXi 5.1.0 Releasebuild-838463 (3.1) Mod MAC-Address(es) Serial-Num --- -------------------------------------- ---------- 1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA 2 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA 3 02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA 4 02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA Mod Server-IP Server-UUID Server-Name --- --------------- ------------------------------------ -------------------- 1 10.85.49.220 NA NA 2 10.85.49.220 NA NA 3 10.85.49.218 422954ef-1f4d-f096-2a47-4b64cd67b932 10.85.49.218 4 10.85.49.219 42299b1d-8226-47bc-f375-2432c7cbe87e 10.85.49.219 3. Migrate Test VM & Test Connectivity a. Cold migrate (powered off) your Test VM to one the Nested ESX hosts. If you get any errors, you ve likely done one of the following: - Didn t modify your Nested ESX VM to Virtual Machine Version 9 prior to install - Didn t enable the Expose NX/XD flag to guest option in the Nested ESX VM.
b. Before we power it up we re going to create a Port Profile for it on the 1000v. N1000v(config)# port-profile type vethernet rhel-pp N1000v(config-port-prof)# switchport mode access N1000v(config-port-prof)# switchport access vlan 711 N1000v(config-port-prof)# state enabled N1000v(config-port-prof)# no shut N1000v(config-port-prof)# vmware port-group c. Now change the virtual network binding of your test VM from the vswitch to the 1000v port profile. d. Power up your test VM, and verify the interface on the 1000v. N1000v(config)# show interface virtual ------------------------------------------------------------------------------- Port Adapter Owner Mod Host ------------------------------------------------------------------------------- Veth1 Net Adapter 1 RHEL62-Test-1 3 10.85.49.218 N1000v(config)# Assuming your networking & port profiles are setup correctly you should have connectivity to your Test VM.
4. Advanced Configuration (optional) a. Now that we have basic connectivity, let s add the remaining uplinks to your Nested ESX VEM hosts. Select the host Configuration Networking vsphere Distributed Switch tab Manage Physical Adapters b. Find the uplink port profile and click Add NIC. Add each of the 2 remaining NICs from each host. c. Verify the uplinks on the 1000v. Your uplink port profile should be configured for mac pinning in which case you should see two new Port channels automatically created. N1000v(config)# show int brief Port VRF Status IP Address Speed MTU mgmt0 -- up 10.85.49.220 1000 1500 Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # Eth3/2 711 eth trunk up none 1000 1 Eth3/3 711 eth trunk up none 1000 1 Eth3/4 711 eth trunk up none 1000 1 Eth4/2 711 eth trunk up none 1000 2 Eth4/3 711 eth trunk up none 1000 2 Eth4/4 711 eth trunk up none 1000 2 Port-channel VLAN Type Mode Status Reason Speed Protocol Interface Po1 711 eth trunk up none a-1000(d) none Po2 711 eth trunk up none a-1000(d) none <snip>
5. Exercise - Determine which uplink your Test VM is utilizing i. Find which module the VM is hosted by. N1000v(config)# show int virtual ------------------------------------------------------------------------------- Port Adapter Owner Mod Host ------------------------------------------------------------------------------- Veth1 Net Adapter 1 RHEL62-Test-1 3 10.85.49.218 ii. Identify the Sub Group IDs of all uplinks on that host. N1000v(config)# module vem 3 execute vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 2 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip> You can see from the output, SGID 1 = vmnic1, 2 = vmnic2 and 3 = vmnic3 iii. Find the VMs pinned Sub Group ID from the same ouput. N1000v(config)# module vem 3 execute vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 2 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip> From this we can see that the Test VM is assigned to SGID 2, which will use vmnic2 for external communication. 6. Exercise Force your Test VM to utilize vmnic1. a. First determine what the SGID of vmnic1 is. From our previous output, this would be SGID 1. b. Configure either the port profile or the individual interface to prefer this Sub Group. N1000v(config)# port-profile rhel-pp N1000v(config-port-prof)# pinning id 1 c. Verify the change. N1000v(config-port-prof)# module vem 3 execute vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 1 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip> 7. Explore & play with various features - ACLs, QoS, PVLANs, etc!