NEC * Virtualized EPC Innovation Powered by Multi Core Intel Architecture Processors

Transcription

1 NEC * Virtualized EPC Innovation Powered by Multi Core Intel Architecture Processors

2 Table of Contents Abstract Introduction Moves towards network virtualization Network Virtualization and Software-defined Networking (SDN) NEC virtualized EPC (Evolved Packet Core) Market background and requirement to EPC NEC virtualized EPC NEC Carrier Grade HyperVisor Intel Data Plane Development Kit Forwarding Technology for NEC vepc Intel Virtualization Technology (Intel VT) Intel Data Plane Development Kit Intel Data Plane Development Kit Architecture Overview Intel Data Plane Development Kit for Virtualized Environment Intel Data Plane Development Kit Performance in Virtual Machines Further Development (OpenVSwitch with Intel Data Plane Development Kit)

3 Abstract This paper introduces NEC* s virtualized EPC (Evolved Packet Core) Innovation that runs on an Intel architecture server platform. NEC, who has been in the mobile core network business with ATCA hardware equipment for around ten years, is now going to provide carrier-grade, mobile network equipment that will work on the common Intel architecture server platform. Virtualized EPC is the first commercial offering of this effort. Over the past few years, Intel has been promoting a 4:1 workload consolidation strategy that enables multiple workloads (e.g., applications, control plane, data plane and signal processing) to simultaneously run on a single platform. The introduction of NEC s virtualized EPC solution on Intel architecture demonstrates a leap forward in this direction. EPC has been traditionally deployed in various platforms using dedicated hardware for specific workloads within the core network. Advances in Intel microarchitecture, software and networking solutions have made the consolidation of these specific workloads onto a common Intel architecture server platform possible. This paper will describe how NEC deploys virtualized EPC on an Intel architecture server platform to overcome network virtualization challenges using the Intel Data Plane Development Kit (Intel DPDK) and achieve the carrier grade service on a common Intel architecture server platform. Among the discussion items are NEC s virtualized EPC architecture, Intel architecture, Intel Virtualization Technology (Intel VT), Intel Ethernet Controllers and data plane processing software for Intel architecture. 1.0 Introduction The key to enabling successful virtualization of each EPC component is to deliver good networking performance and maintain virtual machines (VMs) isolation. This section introduces the concepts that are continuously being refined to successfully implement a virtual EPC. 1.1 Moves towards network virtualization As traffic keeps growing by as much as ten times in five years network operators are addressing this demand with continual investments in network appliances. However, this additional equipment requires more space, and the growing varieties of hardware are increasing complexity for operators building up capacity, enhancing performance and replacing equipment at end of life, which happens sooner as technology innovation accelerates. Virtualization technologies can help reduce the complexities caused by diverse hardware by running vendor applications on common commercial, off-theshelf (COTS) hardware. In addition, the performance and capability of each application can be dynamically adjusted to satisfy changes in demand. Some of these benefits were highlighted in ETSI s Network Functions Virtualization (NFV) forum activities, where major global network operators gathered to discuss how to realize such solutions. The solutions are expected to significantly reduce CAPEX and OPEX for operators by simultaneously simplifying capacity management and enhancing network capacity. NEC architected the individual functions for each EPC node (e.g., Mobility Management Entity (MME), Packet Gateway (P-GW) and Serving Gateway (S-GW)) to run in a virtual machine (VM). Each virtualized function is allocated to a VM. Each VM is totally isolated from the other VMs and runs independently of the other VMs required performance. However, all the VMs in a logical node mutually interconnect in order to forward packets correctly. The details of the implementation will be discussed in a later section. 3

4 Individual Functions Virtual Machine 1 Individual Functions Virtual Machine 2 Individual Functions Virtual Machine VM Isolation Hypervisor 3 2 VM to VM Packet Forwarding Intel Virtualization Technology (Intel VT) for IA-32, Intel 64 and Intel Architecture (Intel VT-x) Intel Virtualization Technology (Intel VT) for Directed I/O (Intel VT-d) Intel Virtualization Technology (Intel VT) for Connectivity (Intel VT-c) 3 VM to Port Packet Forwarding Figure 1. Image of virtualized network functions 1.2 Network Virtualization and Software-defined Networking (SDN) SDN, which plays a complementary role to network virtualization, facilitates the abstraction of network infrastructure. As a result, network operators can control virtualized network resources across network applications, service types, application types, service providers, etc. by orchestrating various networks from the transport to the application network. The bandwidth and capability of each function can be changed dynamically based on requested application or service, or current demand. SDN will enhance virtualized network capabilities to monitor and manage VM status, and control the scaling (out/down) of each VM according to the performance and capability demands across multiple services, tenants, applications, etc. Conversely, network virtualization will maximize the benefits of SDN orchestration through its native ability to provide on-demand flexibility and hardware independence. 2.0 NEC virtualized EPC (Evolved Packet Core) 2.1 Market background and requirement to EPC LTE Core network architecture has fundamentally changed from 2G/3G, and it has shifted to a flat architecture that directly manages each base station (i.e., enodeb). Simplistically, a MME could represent a busy controller of mobility-related signaling traffic, and a S-GW and a P-GW are likely to focus on data plane (DP) processing. But in reality, their functions are more complex since the S-GW and P-GW also play a vital role in the control plane (CP), handling functions such as inter-enobeb mobility anchoring. 4

5 Since the naissance of the smart device, mobile network carriers have been facing challenges in expanding performance and enhancing capacities to adapt the network to satisfy the needs and demands of users who expect the LTE experience to be similar to fixed lines or WiFi services. For instance, planning, configuring and tuning the network to handle surging and fluctuating traffic is becoming more complicated. Furthermore, M2M services kicking into high gear in the mobile network will create a new nature of traffic. Network node deployments and configurations could go beyond recognition or financial sense, considering the need to keep up with the traffic using EPC systems bound on static capabilities and a nonflexible performance balance between CP and DP. It is becoming essential to have elastic network performance and capabilities, as well as flexible performance balance between CP and DP in order to dynamically handle such traffic fluctuations. Obviously, carrier-grade quality, together with the necessary performance and capabilities, is fundamental to the mobile network. 2.2 NEC virtualized EPC The NEC vepc has been realized on commercial offthe-shelf (COTS) servers as virtualized networking functions. Most of NEC vepc software reuses that of existing ATCA-based, non-virtualized EPC products, which have a rich experience and proven quality in commercial networks. Also, in order to maintain carrier-grade qualities on a virtualization platform and maximize virtualization benefits, NEC CGHV (Carrier-Grade HyperVisor) has been introduced to vepc, as described in section 2.3, and the Intel DPDK technology, as described in section 2.4. Figure 2 shows some of the NEC vepc main features, which realize capacity elasticity for easy scale out SDN Controller Resource Control Virtualized Mobile Core (EPC) COTS server #1 COTS server #2 COTS server #3 OpenFlow Control P/SGW (CP) MME (control) P/SGW (DP) P/SGW (DP) P/SGW (DP) MME (S1 INTF) P/SGW (DP) P/SGW (DP) NEC CGHV NEC CGHV NEC CGHV MME (control) Easy Scale-out Server Pool Figure 2. Features of NEC s vepc 5

6 vmme vp/sgw Internet RAN CP DP M2M RAN CP DP Streaming RAN NEC CGHV NEC CGHV CP DP Figure 3. Resource balance between CP and DP for each service from small start to full system, and for CP and DP performance flexibility. Each virtualized logical node consists of multiple virtual machines (VMs), and each VM is allocated independently to CP or DP of the logical node. This feature enables operators to design each logical node capacity, including CP and DP, independently and flexibly. In addition, the SDN Controller manages and deploys VMs according to service and traffic requirements, and allocates a suitable server, considering each VM s functional features, necessary resources and system resource availability. VMs are dynamically interconnected for configuring a virtualized logical node through the SDN Controller using OpenFlow technology. NEC added rich virtualization features to help drastically shorten the time needed to scale capacity and make configuration changes. Using COTS servers for every logical function makes it faster than dedicated hardware, and the SDN Controller s sophisticated virtualization functions make it easier and more simplified to procure hardware, and setup and configure additional functions. Leveraging these features, resource allocation suitable for each service and its quick launch are realized as shown in Figure NEC Carrier Grade HyperVisor Generally, many hypervisors are regarded as suitable for IT services, but inappropriate for carrier-grade mobile core networks, which must deliver real-time processes and stable services for a very large number of subscribers, even under heavy load conditions. The NEC Carrier Grade Hypervisor resolves issues related to virtualization with its management functions, realtime performance, high availability, fault tolerance and easy analysis, whose configurations are described in Figure 4. 6

7 VM P/S-GW (DP) Application VM VM VM VM VM VM P/S-GW (DP) P/S-GW (CP) MME (S1 INTF) MME (CP) HSS PCRF Network Stack Intel Data Plane Development Kit (Intel DPDK) NEC CGHV Guest OS COTS Server Figure 4. NEC s vepc configuration The NEC Carrier Grade HyperVisor integrates advanced technologies, such as intelligent priority control of CPU resources, disk I/O and network I/O, and memory access optimization to minimize virtualization overhead and hardware conflicts caused by accesses from multiple VMs. Fail-safe logic and program improvements are applied to enhance fault tolerance capabilities, high availability and failover. Analysis features, such as VM resource usage tracing and comprehensive log collection, reduce the time needed for failure analysis and virtualization software troubleshooting. Furthermore, NEC s Carrier Grade HyperVisor is based on the Kernel-based Virtual Machine (KVM) and supports its open application programming interface (API), enabling third party applications to maximize operators virtualization benefits on the single virtualization platform. 2.4 Intel Data Plane Development Kit Forwarding Technology for NEC vepc The biggest challenge of virtualization was to achieve both a data plane with high-performance forwarding and carrier grade quality (i.e., low latency and jitter) in a virtualized environment. NEC adopted the Intel DPDK for its vepc in order to significantly improve the DP forwarding performance in a virtualization environment. Figure 4 shows the Intel DPDK deployed with a network stack on the guest operating system (OS) for vepc Data-Plane applications. Furthermore, NEC enhanced the NEC Carrier Grade HyperVisor and the Intel DPDK to overcome DP forwarding quality problems caused by virtualization overhead, conflicts among processes and memory accesses by multiple VMs, thus ensuring low latency and jitter of DP forwarding. Through the improvements NEC made to its Carrier Grade HyperVisor and the Intel DPDK, high performance DP forwarding with carrier grade quality has been achieved on a virtualization platform. 3.0 Intel Virtualization Technology (Intel VT) The fundamental success for Cloud Computing is, arguably, the ability of the system to virtualize a hardware environment. Intel Virtualization Technology offers comprehensive hardware platform capabilities to achieve native performance, with best in class reliability, security and scalability. These capabilities extend from x86 Architectures (Intel Virtualization 7

8 Intel Virtualization Technology (Intel VT) Intel Virtualization Technology (Intel VT) for IA-32, Intel 64 and Intel Architecture (Intel VT-x) Native performance of virtualized workloads with security and reliability Intel Virtualization Technology (Intel VT) for Directed I/O (Intel VT-d) Performance, reliability and security through dedication of system resources Intel Virtualization Technology (Intel VT) for Connectivity (Intel VT-c) Performance, and scalability through a dynamically sharable converged high-capacity interconnect Processor Chipset Network Figure 5. Intel Virtualization Technology (Intel VT) Technology (Intel VT) for IA-32, Intel 64 and Intel Architecture (Intel VT-x)), Intel chipsets (Intel Virtualization Technology (Intel VT) for Directed I/O (Intel VT-d)) and Intel Network Devices (Intel Virtualization Technology (Intel VT) for Connectivity (Intel VT-c)). Intel is continously improving VM performance with each new architecture and process technology. One of the main causes of performance loss is the overhead required for VM entry and VM exit, also referred to as context switching. Addressing this issue, Figure 6 on the next page shows how context switching latency has been dramtically reduced as new Intel processor and new Intel microarcitectures have been introduced. This has been possible because of Intel s Tick Tock model, which enables new CPUs to be introduced each year with significant context switching improvement. The success of NEC s virtualized EPC that is fully deployed on Intel architecture is largely attributed to the Intel Virtualization Technology (Intel VT) advancements. More information on Intel VT can be found on Intel s online resources listed below. 1) Server Virtualization. Fundamentals of CPU resource virtualization and Intel VT-x hardware accelerators virtualization/processors-extend-virtualizationbenefits.html 2) Network Device Virtualization. Fundamentals of network virtualization advancements; example VM Device ques and SR-IOV and Intel VT-c capabilities within Intel Ethernet Server Adapters virtualization/virtualization-for-improved-networkflexibility.html 8

9 8000 Context Switching Latency in Clocks Context Switching Figure 6. Context Switching Latency in Clocks 4.0 Intel Data Plane Development Kit 4.1 Intel Data Plane Development Kit Architecture Overview The Intel DPDK is a set of optimized software libraries and drivers that run in a Linux* userspace environment and enable high-performance data plane packet processing on Intel architecture. The Intel DPDK consists of the following components: Memory/Buffer Manager_ Responsible for allocating pools of objects in memory. A pool is created in huge page memory space and uses a ring to store free objects. It also provides an alignment helper to ensure objects are padded to spread them equally on all DRAM channels. It reduces a significant amount of the time the operating system spends allocating and deallocating buffers. The Intel DPDK pre-allocates fixed size buffers, which are stored in memory pools. Queue Manager_ Implements safe lockless queues, instead of using spinlocks, which allow different software components to process packets while avoiding unnecessary wait times. Flow Classification_ Provides an efficient mechanism for incorporating Intel Streaming SIMD Extensions (Intel SSE) to produce a hash based on tuple information so that packets may be placed into flows quickly for processing, thus greatly improving throughput. 9

10 Poll Mode Drivers_ The Intel DPDK includes Linux User Space Poll Mode Drivers for 1 GbE and 10 GbE Ethernet controllers, designed to work without asynchronous, interrupt-based signaling mechanisms, which greatly speeds up the packet pipeline. Application Application Application Environment Abstraction Layer_ Provides an abstraction to platformspecific initialization code which eases application porting effort. It also contains the run-time libraries for launching and managing Intel DPDK software threads. Figure 7 shows the components of the Intel DPDK and how they interact directly from user-space with the platform hardware through an abstraction layer. Note, the kernelspace part of the hardware abstraction layer is mostly used for initializing access to hardware and subsequent hardware access is done directly without using the kernel-space abstraction layer (i.e., without involving context switching). Intel Data Plane Development Kit (Intel DPDK) Buffer Management Queue/Ring Functions Flow Classification NIC Poll Mode Driver Environment Abstraction Layer User Space Kernel Space 4.2 Intel Data Plane Development Kit for Virtualized Environment The Intel DPDK has included support for data plane packet processing within the VM. Among the included features are a VM virtual function driver for Intel 82599EN/82599EB/82599ES 10 Gigabit Ethernet Controller, virtual NIC and virtual E1000e drivers. Please refer to the official Intel DPDK roadmap for further details. Environment Abstraction Layer Platform Hardware Linux* Kernel Figure 7. Intel Data Plane Development Kit (Intel DPDK) Overview 10

11 4.2.1 Intel Data Plane Development Kit Performance in Virtual Machines The performance shown below in Figure 8 compares Intel DPDK performance running native (no Virtualization), Intel DPDK with direct passthrough (Intel VT-d), Intel DPDK with SR-IOV (Intel VT-c) and Pure Software (without using Intel DPDK). The results show the Intel DPDK running on Intel VT-d and SR-IOV achieved native performance for packet sizes 256B and above Further Development (OpenVSwitch with Intel Data Plane Development Kit) Current ongoing development includes an effort to improve the performance of software switches, which is now gaining prominence and importance. The Intel DPDK will be integrated with OpenVSwitch architecture to replace the current OpenVSwitch forwarding engine with Intel DPDK. This effort will boost the performance of software packet switching and software packet forwarding. Packet Forwarding Performance Estimates for 256B Packet Size 8 Million Packets/s Native Intel DPDK KVM Intel DPDK with Intel VT-d KVM Intel DPDK with SR-IOV Pure Software Estimated results reference for Intel Xeon processor E5 series with 10GB Ethernet Controller Figure 8. Intel Data Plane Development Kit Performance Comparison for 256B Packet Size 11

12 INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A Mission Critical Application is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked reserved or undefined. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or go to: All information provided related to future Intel products and plans is preliminary and subject to change at any time, without notice. All dates provided are subject to change without notice. Intel may make changes to specifications and product descriptions at any time, without notice. Intel, the Intel logo, Xeon, and Xeon Inside are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Other vendors are listed by Intel as a convenience to Intel s general customer base, but Intel does not make any representations or warranties whatsoever regarding quality, reliability, functionality, or compatibility of these devices. This list and/or these devices may be subject to change without notice. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. This document contains information on products in the design phase of development. INFORMATION WITH RESPECT TO NEC CORPORATION S PRODUCTS OR TECHNOLOGIES CONTAINED IN THIS DOCUMENT IS PROVIDED AS IS, AND EXCEPT AS OTHERWISE EXPRESSLY PROVIDED IN NEC CORPORATION S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, NEC CORPORATION MAKES NO REPRESENTATION AND WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT OF INTELECTUAL PEOPERTY RIGHTS. The information contained in this document is subject to change without notice. NEC Corporation assumes no liability for any technical or editorial errors or omissions that may exist in this document. The logo of NEC is a registered trademark of NEC Corporation in Japan and other countries. Other names and brands used in this document may be trademarks or registered trademarks of their respective owners. Copyright 2013 Intel Corporation and NEC Corporation. All rights reserved. *Other names and brands may be claimed as the property of others. Printed in USA MS/VC/0613 Order No US 12