White Paper Advanced Server Network Virtualization (NV) Acceleration for VXLAN August 2012
Overview In today's cloud-scale networks, multiple organizations share the same physical infrastructure. Utilizing common processing and networking resources on an as-needed basis has become a standard business practice. Some cloud networks support implementations with dedicated physical servers for each customer, while other cloud network implementations support dedicated virtual servers per customer (on a common physical server). A single network environment that hosts multiple customers (tenants) allows the customers to reduce upfront costs for processing or networking resources, yet provides them with the flexibility to increase or reduce the resources as needed. Such multitenant environments are increasingly using these new architectures due to the advantages of server virtualization. 97in. 97in. 96in. 96in. Tenant 1 Tenant 2 Tenant 3 Tenant 4 VM VM VM VM VM VM VM VM VM VM VM VM Hypervisor Figure 1: Virtualized Cloud/Data Center with a Multitenant Environment VXLAN-WP101-R Page 2
Key Requirements for Virtualized Cloud-Scale Networks A virtualized, multitenant environment must allow the unlimited transparent migration of workloads across physical servers while controlling the cost and maintaining the Quality of Service (QoS) the customer requires. Most importantly, virtualized data centers need the flexibility of provisioning resources that span multiple geographic locations. At the same time, the virtualized data centers must maintain isolation between tenants and still allow seamless management of the multitenant environment. Virtualized cloud networks must also accomplish the following: Handle MAC address growth in conjunction with the explosive growth of VMs in the cloud data center Accommodate a larger number of VLANs to handle VM traffic segregation Provide isolation of the physical L2 network Figure 2 highlights some of the benefits of a virtualized cloud-scale network as well as the shortcomings this environment commonly faces. The benefits are outlined in red, and the shortcomings are outlined in black. Figure 2: Benefits and Shortcomings of Virtualization in a Private Cloud Environment VXLAN-WP101-R Page 3
Solutions for Virtualized Cloud-Scale Networks To provide workload mobility and migrate across geographic locations, one cloud network solution is to decouple the physical and logical addressing schemes. The tenant uses the logical address while the network infrastructure sees the physical address. This decoupling enables the flexibility required by the virtualized cloud data center for creating a faster, fatter, and flatter network. Isolation of multitenant environments can be enabled through logical network configuration of multiple VLANs and IP subnets. Scaling Cloud Networks with VXLAN Scaling the cloud network with VXLAN is the first step towards enabling logical, software-based networks that can be created on demand, allowing enterprises to leverage capacity wherever it is available. In other words, VXLAN can now help companies build true global clouds that are the sum of their parts rather than distinct sets of parts. A true global cloud essentially decouples the physical network design from the logical network design. VXLAN accomplishes its goals by using an overlay of tunnels. Unicast traffic uses unicast tunnels, and multicast traffic is sent to a per-tenant IP multicast group. VXLAN running on networking infrastructure powered by Broadcom silicon (switches and controllers on servers) establishes a collection of pipes over the physical network. The physical network can now be efficiently designed to support multitenancy with virtualization in cloud and mega data centers. The VXLAN and Broadcom cloud-scale network solution provides: L2-in-L3 tunneling for tackling issues with virtual networking. A method to tunnel MAC in IP over UDP (L2-IP-UDP-VXLAN-L2). Efficient use of UDP. VXLAN takes advantage of UDP by using UDP encapsulation and pseudorandom values in the UDP source port, resulting in automatic equal-cost load balancing in every switch device that uses a 5-tuple hash calculation to load balance. VXLAN-WP101-R Page 4
In Figure 3, the shaded circles indicate the issues that VXLAN helps network architects overcome in a virtualized private cloud environment. Figure 3: Cloud-Scale Networks with VXLAN VXLAN-WP101-R Page 5
Enabling VXLAN with Broadcom Technology VXLANs are expected to become the default standard for multitenant cloud-scale networks, enabling multitenant data center fabrics. With Broadcom NetXtreme II controllers providing native Server NV Acceleration and the Broadcom switch providing Smart NV technology, cloud and enterprise operators are able to deploy faster, fatter, and flatter network implementations of the virtualized VXLAN cloud-scale networks. Cloud and enterprise IT architects can leverage the combination of VXLAN and Broadcom controller/switch solutions for high-performing virtualized cloud-scale networks. Figure 4: Broadcom Controller/Switch-Enabled VXLAN Cloud-Scale Network VXLAN-WP101-R Page 6
Broadcom Server NV Acceleration for VXLAN The standard VXLAN solution is not designed for multicore, multiprocessor server environments deploying multiple VMs. VXLAN performance is hampered by the fact that all the VMs are throttled by the singlethreaded nature. Broadcom controllers, with advanced Server NV Acceleration technology, enable necessary QoS for multiple VMs in a muticore environment by removing the single-queue restrictions and enabling multiple hardware queues for multiple VMs. For VMware ESXi Hypervisor, NV Acceleration is enabled by the RSS support available in vsphere 5.1. Broadcom Server NV Acceleration technology enables efficient distribution of network receive processing for VXLAN traffic across servers that have a multiprocessor system with multiple CPU cores. With Broadcom Server NV acceleration, the controller and driver provide the ability to schedule workload distribution efficiently across all processors. Broadcom Server NV Acceleration technology improves network performance by reducing processing delays and distributing receive processing from the controller across multiple CPUs for VXLAN traffic. This helps ensure that no CPU core is heavily loaded while the other is idle. The following acceleration technologies are enabled to enhance VXLAN performance: Distributed processing of VXLAN traffic In-order processing of VXLAN traffic Dynamic load balancing of VXLAN traffic MSI-X support VXLAN-WP101-R Page 7
Figure 5: Broadcom Controller with Server NV Acceleration Enabled in Virtualized Environment VXLAN-WP101-R Page 8
Performance Metrics of VXLAN Acceleration In a basic configuration with a Dell PowerEdge R720 server with dual 2.699 GHz CPUs and eight VMs running on the Broadcom 10G BCM57810 controller, tests show a performance increase of 55% with the VXLAN receive traffic over a configuration without Server NV Acceleration. The bidirectional performance rates are as follows: VXLAN bidirectional traffic (receive and transmit) = 15.6 Gbps VXLAN bidirectional traffic (receive and transmit) with Server NV Acceleration = 22.4 Gbps The performance improvement = 55% Figure 6: VXLAN Bidirectional Traffic Performance The performance improvement is noticeable on the receive VXLAN traffic as well. With Server NV Acceleration enabled, a 30% increase in throughput can be obtained. The receive performance rates are as follows: VXLAN receive traffic on one port = 6.78 Gbps VXLAN receive traffic on one port with Server NV Acceleration = 8.95 Gbps The performance improvement = 31% Figure 7: VXLAN Receive Traffic Performance VXLAN-WP101-R Page 9
Server NV Acceleration VXLAN Benefits Using a multitenant environment with virtualization and Broadcom-based VXLAN solution adds many benefits when designing a virtual L2 network that contains multiple physical L2 networks. Lower Capital Expenditure (CAPEX) Costs Broadcom NetXtreme II controllers are deployed on tier-1 OEM servers for HP, Dell, IBM, and Cisco. Existing server infrastructure can be used for Server NV Acceleration with VXLAN, lowering CAPEX costs and increasing cloud-scale network performance. See also: http://go.broadcom.com/dell12g See also: http://go.broadcom.com/hpgen8 Figure 8: Broadcom NetXtreme II Controllers on Servers NetXtreme II controllers that enable Server NV Acceleration are available in modular LOM, mezzanine, and NIC form factors supporting blade, rack, and tower servers. The controllers include two or four 10GbE ports and support L2 networking, iscsi, and FCoE. VXLAN-WP101-R Page 10
VM Scalability Cloud networks can increase the number of VMs being deployed on the servers by over 55% additional bandwidth provided by Server NV Acceleration for VXLAN deployments. Figure 9: Example of VM Scalability with Server NV Acceleration For example, if a server is provisioned with three VMs that each have 5 Gbps bandwidth, with Server NV Acceleration, two additional VMs with 5 Gbps bandwidth can be supported at no additional cost. VXLAN-WP101-R Page 11
Increased ROI for VXLAN Deployments Cloud networks can increase the number of VMs with Server NV Acceleration thereby increasing the number of tenants with the same physical infrastructure. Figure 10: Example of Additional Tenants with Server NV Acceleration For example, if a server is provisioned with three VMs that each have 5 Gbps bandwidth, with Server NV Acceleration, two additional VMs with 5 Gbps bandwidth can be supported at no additional cost. VXLAN-WP101-R Page 12
Additional Bandwidth for Performance-Intensive Applications With Server NV Acceleration, network administrators can provision additional bandwidth for resourceintensive applications or VMs. Network administrators can also overprovision VMs with bandwidth for highpeak scenarios. Figure 11: Example of Usage for Bandwidth-Intensive Applications Additional bandwidth provided by Server NV Acceleration can be allocated to high-performing VMs or bandwidth-intensive applications such as backup and mirroring. Workload Mobility VXLAN enables workload mobility through VM migration across multiple geographic locations. To enable VM migration, VXLAN separates the logical address space and physical address space through overlay networks. Server NV Acceleration aids the bandwidth-intensive VM migration necessary for workload mobility on cloudscale migrations. VXLAN-WP101-R Page 13
Summary Broadcom offers the most complete, end-to-end VXLAN product portfolio. Additionally, Broadcom offers the unique functionality necessary for broad deployment. Using the Broadcom controller Server NV Acceleration can enhance VXLAN performance by more than 55%. Broadcom NetXtreme II Controller solutions are available from tier-1 OEMs. Authors Abhijit Aswath is Manager, Product Marketing for Broadcom Ethernet Controllers. Sujal Das is Director, Product Marketing for Broadcom Switch. Broadcom, the pulse logo, Connecting everything, and the Connecting everything logo are among the trademarks of Broadcom Corporation and/or its affiliates in the United States, certain other countries and/or the EU. Any other trademarks or trade names mentioned are the property of their respective owners. Broadcom Corporation reserves the right to make changes without further notice to any products or data herein to improve reliability, function, or design. Information furnished by Broadcom Corporation is believed to be accurate and reliable. However, Broadcom Corporation does not assume any liability arising out of the application or use of this information, nor the application or use of any product or circuit described herein, neither does it convey any license under its patent rights nor the rights of others. BROADCOM CORPORATION 5300 California Avenue Irvine, CA 92617 2012 by BROADCOM CORPORATION. All rights reserved. VXLAN-WP101-R August 2012 Phone: 949-926-5000 Fax: 949-926-5203 E-mail: info@broadcom.com Web: www.broadcom.com