Network Virtualization Deep dive and Network Troubleshooting in a virtualized Environment Alexander Paul paulalex@de.ibm.com IBM Certified Advanced Technical Expert (C.A.T.E.) for Power Systems Certified Cisco Systems Instructor CCSI #32044
Physical networking I m a physical switch. You can touch me! 2
Physical networking A physical network consists of real hardware devices with embedded logic: Switches Routers Network interface cards Cable [ ] Physical network devices have their own operating environment. Physical network devices are independently manageable via a console port or an ip based management interface. [ ] 3
Hypervisor based Virtual Networking Virtual networks are in software realized networks. a consolidation of networking logic. minimizing the need for physical links. centrally supervised. dependent on a central operating environment. Benefits of network virtualization? Decreases the time spent by cabling physical servers. The number of adapters, switchports, wires can be reduced. Platform for low latency in-the-box communication 4
Virtual Ethernet Virtual Ethernet Standard technologie in near all host virtualization products. Hypervisor implemented layer 2 switch. In-box packet delivery by memory-to-memory copy Generated MAC addresses Virtual I/O Server Ent0 (Phy) Shared Ethernet Adapter ent1 (Vir) Client 1 en0 (if) ent0 (Vir) Hypervisor Client 2 VLAN-Aware Ethernet Switch en0 (if) ent0 (Vir) Ethernet Switch 5
Virtual Ethernet - MAC Address calculation LPAR ABCDEF123456 Create Virtual Ethernet Adapter ent0 (Virt) HMC Frame is blocked by Hypervisor Calculating MAC Address 7 2 : E C : F C : F 5 : B 6 : 0 B 20 Bit CEC ID 20 Bit Random 8 Bit Slot ID 6
Virtual Ethernet Performance Virtual Ethernet scales with processor entitlement. MTU=1500 270 Mbits/sec Jumbo Frames 879 Mbits/sec LPAR 1 LPAR 2 Traffic direction Virt. Eth. Virt. Eth. Hypervisor PVID 1 PVID 1 7
Integrated Virtual Ethernet Base Offering: #5636 2 Port 1 Gb 2 x 1Gb Eth Serial 2 Serial 1 VPD card 10Gb Upgrade Offering: #5637 2 Port SX 10 Gb Serial 2 VPD card 10Gb Eth 10Gb Eth 4 x 1Gb Upgrade Offering: #5639 4 Port 1 Gb 4 x 1Gb Eth Serial 2 VPD card Physical adapter with virtualization capabilities No hypervisor work for frame bridging Network virtualization without the need of hypervisor bridging Removes software packet forwarding overhead from hypervisor Provides low latency, low cpu consuming in-box communication 8
IVE Logical Components Diagram Hypervisor AIX 1 AIX 2 AIX 3 en0 (if) en1 (if) en0 (if) en1 (if) en0 (if) en1 (if) ent0 lphea ent1 lphea ent0 lphea ent1 lphea ent0 lphea ent1 lphea lhea0 lhea1 lhea0 lhea1 lhea0 lhea1 Virtual Layer 2 Switch Virtual Layer 2 Switch Logical Ports (LHEA) HEA Physical Port Physical Port 9
IVE System Architecture Low Latency Design GX+ bus attachment Immediate data in descriptors (reduced memory access) Direct user space per-connection queuing (OS bypass) Up to 3X throughput improvement over current 10 Gbps solutions Additional acceleration functions to reduce host code path length. Provides direct I/O virtualization support Allows 10 Gbps port to replace up to 10 dedicated PCI 1 Gbps adapters in a partitioned system IVE offers the following virtualization functions Sixteen MAC addresses are assigned to each IVE port group Each logical port can be owned by a separate LPAR Direct data path to LPAR Default send and receive queues per LPAR Ethernet MIB and RMON counters per LPAR VLAN filtering per logical port (4096 VLANs * 32 Logical Ports) Internal layer 2 switch for LPAR to LPAR data traffic Multicast / Broadcast redirection to Multicast / Broadcast manager P5IOC2 System Memory POWER6 Chip GX interface IVE 2 x 10 Gbps or 4 x 1 Gbps Ethernet 10
Throughput Benchmark Host Ethernet Adapter LPAR 1 AIX 6.1 SP 1 EC 0.3, capped p570 no 1 LPAR 2 HEA MCS 4 HEA MCS 1 Virt. Eth. HEA MCS 4 HEA MCS 1 Virt. Eth. Quad Port IVE T1 T2 T3 T4 Quad Port IVE T1 T2 T3 T4 HEA MCS 4 HEA MCS1 Virt. Eth. HEA MCS 4 HEA MCS 1 Virt. Eth. LPAR 1 p570 no 2 LPAR 2 AIX 6.1 SP 1 EC 0.3, capped 11
Throughput benchmark results 700 600 500 Throughput [Mbit/s] 400 300 200 100 0 0 10 20 30 40 50 TCP sessions HEA MCS=4 HEA MCS=1 Virtual Ethernet 12
Virtual LANs Each VLAN appears as a independent and isolated network. Each VLAN represents a dedicated security domain. VLAN membership is controlled from a central point and is transparent to the client. Broadcast traffic affects only clients within the same VLAN. Physical network topology Logical network topology 13
VLAN trunking Link enablement to transfer multiple VLAN traffic through a single port. In outgoing direction each frame must be labelled with its VLAN membership. VLAN trunk VLAN trunks are good for Switch uplinks connecting to Routers connecting to Firewalls connecting to Hypervisors to support Virtual 14 VLAN 1 VLAN 2 VLAN 3
Inter VLAN routing VLAN trunks are used to carry frames to the router Switch1 Virtual Gateway VLAN2 Virtual Gateway VLAN2 Router Dot1q VLAN trunks Virtual Gateway VLAN1 Dynamic routing updates Switch2 15 Core Router VLAN 1 VLAN 2 VLAN 3
Data & Control Plane virtualization: VRF The VRF: Virtual Routing and Forwarding instance VLAN Trunk, physical interfaces, tunnels, etc. VRF 1 VRF 2 Logical or Physical Int (Layer 3) VRF 3 Each VRF = separate forwarding table Logical or Physical Int (Layer 3) 16
Spanning Tree Hierarchical Star Network Architectures Easy to implement Little reliable Single point of failure (in the star centre) Need for high reliability Alternate paths demand Problem: bridging loops Path costs: Bandwidth Cost 4 Mbps 250 10 Mbps 100 16 Mbps 62 45 Mbps 39 100 Mbps 19 155 Mbps 14 622 Mbps 6 1 Gbps 4 10 Gbps 2 Physical network topology STP network topology 17
Gateway redundancy I cannot reach my Gateway! The Problem... but i could route you to the backbone! Backbone 18
Gateway redundancy The idea Virtual Router Cluster Backbone 19
Gateway redundancy Typical Redundancy techniques in mission critical applications: Local Area Networks Backup layer 2 paths with Spanning Tree Protocol Wide Area Networks Backup layer 3 paths and dynamic routing algorithms Default Gateways can become single point of failures: Gateway Redundancy Protocols: HSRP (Hot Standby Router Protocol): proprietary - Cisco NSRP (NetScreen Redundancy Protocol): proprietary - Juniper VRRP (Virtual Redundancy Routing Protocol): Standard GLBP (Gateway Load-Balancing Protocol): Cisco/Standard 20
Link Aggregation: EtherChannel More then one link can be grouped to form a Channel Generally used for switch interconnection Sometimes used to connect a switch to a router. a server. a hypervisor. Etherchannel is seen by the switch as a single physical link Benefits More bandwidth available with load sharing Redundancy & Better availability Fast recovery in case of failure Negotiation Negotiation ent0 ent1 ent2 AIX 21
Packet distribution for Cisco port channels Availability of other load balancing methods depends on switch model Cat_3560_2(config)#port-channel load-balance? dst-ip Dst IP Addr dst-mac Dst Mac Addr src-dst-ip Src XOR Dst IP Addr src-dst-mac Src XOR Dst Mac Addr src-ip Src IP Addr src-mac Src Mac Addr Model Protocol Source Destination Source XOR Destination Layer 2 Models (C2900) MAC IP X X X X X X Multi Layer Models (C3560, C3750, C4900. C6500) TCP / UDP X X X Cisco 6500 PFC3C/XCL IP and TCP / UDP X X X Cisco 6500 PFC3C/XCL Layer 3 + VLAN ID X 22
If you do it wrong Mar 22 18:57:46 ent12 I ECH_CHAN_RCVRY Mar 22 18:57:42 ent3 I GOENT_RCVRY_EXIT Mar 22 18:57:39 ent12 P ECH_CHAN_FAIL Mar 22 18:57:39 ent3 T GOENT_LINK_DOWN Mar 22 18:23:55 ent12 I ECH_CHAN_RCVRY Mar 22 18:23:51 ent3 I GOENT_RCVRY_EXIT Mar 22 18:23:48 ent12 P ECH_CHAN_FAIL Mar 22 18:23:48 ent3 T GOENT_LINK_DOWN %SW_MATM-4-MACFLAP_NOTIF: Host 001a.6484.b012 in vlan 65 is flapping between port Gi1/0/19 and port Gi2/0/19 ping www.google.de PING www-tmmdi.l.google.com (216.239.59.103): 56 data bytes 64 bytes from 216.239.59.103: icmp_seq=0 ttl=49 time=46 ms 64 bytes from 216.239.59.103: icmp_seq=2 ttl=49 time=46 ms 64 bytes from 216.239.59.103: icmp_seq=4 ttl=49 time=44 ms 64 bytes from 216.239.59.103: icmp_seq=6 ttl=49 time=45 ms --- www-tmmdi.l.google.com ping statistics --- 8 packets transmitted, 4 packets received, 50% packet loss 23
Cisco Virtual Switching System (VSS 1440) Operational Manageability Two Catalyst 6500s share a single point of management, single gateway IP address, and single routing instance Non-Stop Communications Delivers deterministic, sub-200 millisecond layer 2 link recovery through interchassis stateful failovers and the predictable resilience of Etherchannel Scales to 1.4 Tbps 24 Scales system bandwidth capacity to 1.4 Tbps Up to 132 ports of 10 GbE per system
Multichassis EtherChannel (MEC) with Virtual I/O Server Layer 2 multipathing technology Creates simplified loop-free topologies Supported Protocols: Cisco Port Aggregation Protocol (PAgP) 802.3ad Link Aggregation Control protocol (LACP) ON Manual Etherchannel Virtual I/O Server virt SEA LA Active phy MEC phy Active VSL 25
Data Center Traffic Flow 26
Traffic Flow with Service Modules in a Looped Access Topology 27
How does Quality of Service (QoS) work? 28
How does Quality of Service (QoS) work? QoS relies on two concepts: Traffic marking Priority of network traffic is maintained by additional header information at frame or packet level: Layer 2 marking: IEEE 802.1P via Class of Service (CoS) Layer 3 marking: Differentiated Service (ToS, DSCP or DiffServ) Network traffic is marked accordingly to the type of service it needs. Very important traffic should have a high priority value. Less important traffic should have a lower priority value. Best effort traffic can be forwarded without any marking. Traffic queuing Traffic is scheduled with different importance. Importance of traffic depends on priority: CoS value on Layer 2 ToS value on Layer 3 Different queuing methods can be used: FIFO Weighted Fair Weighted round robin Low Latency Shared Ethernet Adapter supports bandwidth adoption in conjunction with IEEE 802.1p. 29
Power Hypervisor supports Layer 2 CoS marking Destination MAC Source MAC 802.1q Type Tag Data FCS TPID PRI CoS CFI VLAN ID CoS 0 1 2 3 4 5 6 7 Typical Application best effort background spare excellent effort controlled load video < 100 ms latency and voice jitter < 10 ms latency and jitter network control 30
How does priority queuing works? FIFO Queue Sending Direction (one packet at a time) (relative time of arrival) 31
How does priority queuing works? Priority Queue Best Effort Queue (one packet at a time) (relative time of arrival) 32
Shared Ethernet Adapter CoS queuing VIOS en2 (if) LPAR 1 LPAR 2 SEA supports CoS queuing Tag Tag Tag LPAR LPAR 112 ent0 (Phy) Tag LPAR 2 1 ent2 (SEA) ent1 (Vir) Low CoS priority Virt High CoS priority Virt VLAN X 33
Converged I/O Today Today Today Management LAN SAN A SAN B Parallel LAN/SAN Infrastructure Inefficient use of Network Infrastructure 5+ connections per server higher adapter and cabling costs Adds downstream port costs; cap-ex and op-ex Each connection adds additional points of failure in the fabric Longer lead time for server provisioning Multiple fault domains complex diagnostics 34 FCoE Ethernet FC Management complexity firmware, driver-patching, versioning
Converged I/O Today Converged I/O Converged I/O: Management LAN SAN A SAN B Reduction of server adapters Simplification of access layer and cabling Gateway free implementation fits in installed base of existing LAN and SAN FCoE Switch L2 Multipathing Access Distribution Lower TCO Fewer Cables Investment Protection (LANs and SANs) FCoE Ethernet FC Consistent Operational Model 35
#5708 10Gb FCoE PCIe Dual Port Adapter for IBM Power Systems #5708 is a CNA (Converged Network Adapter) Dual 10Gb ports Physically are Ethernet ports Each port can run all NIC, all FC, or mixed NIC/FC traffic SR optical fiber cabling SOD for NPIV function through VIOS AIX & Linux support AIX 5.3 with the 5300-11 Technology Level, or later AIX 6.1 with the 6100-04 Technology Level, or later SUSE Linux Enterprise Server 10 Service Pack 3 or later Red Hat Enterprise Linux 5.4 or later VIOS support VIOS 2.1.2.0 or later PCIe 8x Gen 1 Adapter CCIN = 2B3B 36
Thank you! Alexander Paul paulalex@de.ibm.com 37