Implementing High-Throughput and Low-Latency 10 Gb Ethernet for Virtualized Data Centers
Implementing High-Throughput and Low-Latency 10 Gb Ethernet for Virtualized Data Centers Introduction Adoption of high performance networking solutions is rapidly increasing due to the increasing needs of I/O intensive applications in virtualized data centers. This growing demand has drawn networking infrastructure vendors to develop a new crop of solutions that provide a substantial boost in network performance (10 Gb Ethernet offers 10x the throughput of Gigabit Ethernet and dramatically lower latency). Additionally, the industry has developed a full suite of innovative capabilities that enables consolidation of mixed applications into fewer systems without compromising enterprise-class service levels. The result is a new generation of Ethernet switch and adapter solutions that offer unparalleled performance and scalability, coupled with superior ROI and TCO. Problem Statement I/O-intensive tasks (such as backup and recovery or data warehousing) and I/O sensitive applications (such as on-line transaction processing) suffer when the network infrastructure is a bottleneck. This performance problem is exacerbated when an administrator must consolidate one or more I/O intensive applications, such as an Oracle database and an associated data mining application, on a single virtualized system. Virtualized servers may also require different kinds of applications to share a single server and which means that several applications are now competing for access to the network. The result is a serious I/O bottleneck. It is this bottleneck that is deterring IT managers from consolidating more of their applications, which limits the benefits achieved by consolidation and virtualization. A virtualized system, which will typically host multiple enterprise applications, frequently share a limited number of physical network I/O connections, which is directly tied to the fact that a limited number of network adapters can be supported in a given system. Without proper planning, application consolidation will lead to I/O constrained servers, and in turn, poor performance in applications. For this reason many server administrators are reluctant even downright unwilling - to consolidate and run their I/O-intensive applications on virtualized server platforms. Server administrators, who typically own responsibility for the successful operation of all hardware in their data centers, cannot afford to compromise the reliability or throughput or latency caused by the server or the network resources. Previous Options Traditional server I/O methods often fall short of delivering sufficient I/O performance - and thus application performance - in a virtualized server environment. One approach to overcoming the I/O bottleneck on virtualized servers is to dedicate a physical Ethernet port for each critical virtual machine. For example, an installation running eight virtual servers might dedicate a 1 GbE adapter to each. But this solution suffers from several limitations:
It is expensive in terms of hardware acquisition and management costs. 1 GbE of bandwidth per virtual machine may not be sufficient to meet the peak demands of I/Ointensive applications. The number of running VMs will quickly outstrip the number of physical PCI slots in the system, so the solution is temporary at best. Finally, this approach leads to an increase in the complexity of physical I/O configurations, and particularly cabling layouts. Neterion and Arista Networks Solution Fortunately, a new solution is available to server administrators to address these problems. Arista s high-density, low-latency 10 GbE switches, coupled with Neterion s high-performance, low-latency 10 GbE adapters with support for multiple physical hardware channels, addresses the challenges outlined above without increasing cost or complexity. This available-today technology enables corporate users to virtualize a whole new class of applications, while fully leveraging existing IP-based networks and administration tools. The solution enables high-performance I/O for multiple virtual machines on consolidated systems. By helping server administrators to overcome application throughput and latency limitations, and at the same time, simplify network I/O cabling and management, a combined Neterion and Arista solution can significantly ease the pain points that typically result from consolidating applications on virtualized server platforms. Arista 10 GbE Switch The Arista 7100 Series of Ethernet switches are optimized for new generation of data centers by featuring the industry's highest density 10 Gigabit Ethernet switching solution with a 1U rack unit (ru) with up to 48 ports. It is the first with an extensible modular network operating system. With breakthrough price-performance and power efficiency the Arista 7100 Series enables 10 Gigabit Ethernet to be deployed everywhere in the data center, which can significantly improve server utilization and consequently data center power efficiency. In addition to offering a highly cost-effective network interconnect, the market-leading 10 GbE enterprise switches has been designed from the outset to meet the most demanding requirements for increased levels of aggregated traffic in the data center, as well as the core and access/aggregation tiers of the network. Switch features that support 10 GbE IP storage for consolidation and virtualization of storage resources include: High port density simplifies the design of data center networks and allows a large number of servers to readily share IP storage resources High-Performance, Low Latency 10 Gigabit Ethernet Switching: Layer 2/3/4 switching up to 960 Gbps and 720 Mpps. Delivers low 600 nanosecond latency at all packet sizes Arista EOS: Extensible operating system customizable to customer needs. Fine-grain modular protected memory architecture Neterion 10 GbE Adapters
Best-in-class 10 GbE adapters have I/O virtualization functionality designed into their core. This is a key requirement for enabling the virtualization of I/O intensive applications. Neterion s virtualized architecture represents a major breakthrough in server utilization. One of the keys to the high performance of Neterion s multi-channel architecture is its hardware state machine based implementation. Neterion s state machine architecture enables lower power and better latency characteristics than competing firmware-based implementations. Additional innovations contributing to superior performance include contextual interrupts, which reduce VM processing overhead; and independent interrupt moderation schemes for each I/O channel. These channel-specific interrupt schemes enable server administrators to select the appropriate interrupt scheme for each context, which in turn allows the adapter to optimize its operation for throughput, latency or quality of service as required. The architectural design also includes a number of network and system related offloads, which help reduce I/O bottlenecks and improve application performance. One 10 GbE with multiple I/O channels eliminates need for multiple 1 GbE cards and significantly ease management complexity, by simplifying cabling and consolidating adapter cards. Benefit 1: QoS at the adapter level IOV (I/O virtualization) solutions pool and dynamically allocate a set of high-speed I/O resources, such as network bandwidth across multiple physical I/O channels. Multi-channel Neterion adapters have independent hardware-based transmit and receive paths that may be prioritized for true quality-ofservice at the IO level. As well, they implement independent interrupt schemes for each virtual machine and can be reset independently. This enables the virtualization of more I/O intensive applications per physical server. From an application s perspective, IOV appears as traditional I/O no change to I/O management is required. Benefit 2: High Performance 10 Gbps throughput is essential to providing the I/O headroom that multiple, concurrently running applications need in a virtualized environment. Neterion adapters are designed to provide maximum efficiency and minimize user response times for critical, I/O intensive applications, delivering the industry s lowest latency without requiring proprietary offloads. Stateless-offload adapters are fully capable of carrying 10 GbE traffic with reduced CPU utilization rates, with full compliance with all operating systems without an overhaul of TCP/IP stacks, and preserving the security and reliability of established TCP/IP stacks Benefit 3: Reliability Neterion has a proven enterprise track record that is based on the industry s best RAS (Reliability, Availability, Scalability) capability. This is a result of being the first high volume network adapter supplier for server and storage OEMs. Thousands of enterprise customers have implemented Neterion products. Additionally, broad operating system support, including Windows, Linux, VMware, AIX, HP- UX, Solaris Mac OS, ensures a solution for any environment Performance Data
In this section, we outline the network performance that may be achieved using best-in-class network hardware, including Neterion 10GbE adapters, and Arista low-latency 10GbE switches. In joint benchmark testing, Neterion and Arista were able to produce line-rate throughput and best-in-class server-to-server latency without needing to resort to the use proprietary offloads such as TOE (which is not generally accepted by the server management community). Our demonstration included the following hardware: An Arista 7148S 48-port 10 GbE switch A Neterion 10 GbE adapter connected through the switch to multiple 10 GbE endpoints. The Arista 7148S is the highest density switch in the market to deliver less than 600 nanoseconds switching latency, which is comparable to proprietary interconnects. It is an ideal match to Neterion's Xframe and X3100 series of 10 GbE adapters, which are available for PCI-X 1.0, 2.0 and PCI Express slots. In our testing, the Neterion and Arista products were benchmarked to evaluate their throughput and latency. The results were measured using Chariot and Netpipe, two standard network performance profiling tools, please refer to Figure 1. Figure 1. Neterion and Arista Performance Profiling Number of Bytes Board to Board (µsec) Thru Arista Switch (µsec) Difference (µsec) 1 11.55 12.09 0.54 2 11.62 12.18 0.56 4 11.60 12.17 0.57 8 11.77 12.34 0.57 12 11.75 12.36 0.61 16 11.69 12.29 0.60 24 11.80 12.35 0.55 32 11.77 12.42 0.65 48 11.77 12.43 0.66 64 11.86 12.56 0.70 Average 11.72 12.32 0.60 Results of the benchmark are presented in two ways. The board to board configuration does not include a switch as the boards in two different servers are directly connected. The second set of numbers in the table below includes a single hop through the Arista switch. Latency is indicated in microseconds (µsec) while throughput is presented in gigabits per second (Gbps). A key point is that our best latency (under 12 µsec) was achieved with standard TCP/IP running on a standard installation of Windows 2008. No proprietary offloads, including TOE or iwarp, were required.
Only mainstream, readily available hardware and protocols were used. The Arista switch only contributed an impressive 0.6 µsec (600 nanoseconds). The chart below is a screen shot of Chariot I/O throughput performance tool. It highlights the average through put of the Neterion 10 Gb Ethernet adapter running at a sustained rate of 9.869 Gbps. Summary Server virtualization provides server administrators with a strategic approach to solving some tough problems, including the issues of underutilized systems, data center space and power limitations, and run-away hardware and operating costs. But, as is typical of new technologies, server virtualization also creates some new challenges, such as application I/O bottlenecks that can lead to both dissatisfied users and reduced overall productivity. Since these problems cannot be solved by using traditional methods, server administrators need a new approach to solving the I/O problem in virtualized server environments, so that they can once again deliver strong application performance to their end users.
Neterion 10 GbE adapters, when deployed with Arista s high-density 10 GbE switches provide server administrators with a reliable, high performing I/O connectivity solution, which restores the balance between server CPU and I/O resources. This approach enables an environment that is more flexible and nimble, simpler to support and easier to manage. At the end of the day, a Neterion 10 GbE deployment, based on the company s advanced I/O virtualization technology, provides server administrators with the competitive edge they need to meet server reliability and performance requirements, while staying one-step ahead of their users. About Neterion: Neterion is the leader in I/O Virtualization for 10 Gigabit Ethernet adapters that provide high speed solutions for customers' server and storage systems. Xframe and X3100 Series are the industry's highest performing line of 10 GbE adapters, allowing IT managers to virtualize many more applications per physical server - including the most I/O-intensive ones drastically decreasing costs when deploying virtualized servers. The company was founded in 2001 and is headquartered in Cupertino, California with an additional office in Ottawa, Canada. More information on Neterion, Inc., can be found at http://www.neterion.com About Arista Networks: Arista Networks, based in Menlo Park, California, was founded to deliver scalable networking interconnects for large-scale datacenters and cloud computing. Arista offers bestof-breed 10 Gigabit Ethernet solutions for Cloud Networking TM that redefine scalability, robustness, and price performance. At the core of Arista's platform is the Extensible Operating System (EOS TM ), pioneering new software architecture with self-healing and live in-service software upgrade capabilities. Arista's team is comprised of seasoned management and engineering talent bringing over a thousand man-years of expertise from leading network and server companies. For more information, please visit: http://www.aristanetworks.com Arista Networks, the Arista logo, Cloud Vision, Cloud Networking and EOS are trademarks of Arista Networks, Inc. in the U.S. and other countries. The trademarks, logos and service marks ("Marks") displayed in this document are the property of Arista Networks, Inc. or other third parties. Other names and brands may be claimed as the property of others.