Network Functions Virtualization (NFV); Testing Best Practices

Transcription

1

2 2014 QualiTest Group Ltd. ALL RIGHTS RESERVED. This document and any and all content or material contained herein, including text, graphics, images and logos, are either exclusively owned by Qualitest Group Ltd., its subsidiaries and/or affiliates ("QualiTest") or are subject to rights of use granted to QualiTest, are protected by national and/or international copyright laws and may be used by the recipient solely for its own internal review. Any other use, including the reproduction, incorporation, modification, distribution, transmission, republication, creation of a derivative work or display of this document and/or the content or material contained herein, is strictly prohibited without the express prior written authorization of QualiTest. The information, content or material herein is provided "AS IS", is designated confidential and is subject to all restrictions in any law regarding such matters, and the relevant confidentiality and non-disclosure clauses or agreements issued prior to and/or after the disclosure. All the information in this document is to be safeguarded and all steps must be taken to prevent it from being disclosed to any person or entity other than the direct entity that received it directly from QualiTest. The text and drawings herein are for the purpose of illustration and reference only. QualiTest reserves the right to periodically change information that is contained in this document; however, QualiTest makes no commitment to provide any such changes, updates, enhancements or other additions to this document to you in a timely manner or at all. Publication Date: November 2014 Web Site: P a g e 2

3 Contents 1. Scope References Abbreviations Introduction... 7 NFV Framework... 7 NFV Use Cases Assuring compliance and conformance to ETSI ISG Standards... 9 Test Procedure... 9 MANO Compliance Criteria... 9 VNF Compliance Criteria NFV Test Environment Build NFV Test Environment Test Appliances Virtual Test Appliances vs. Physical Test Appliances Simulating real workload traffic Metrics Test Automation in NFV The Need for Heterogeneous SDN/NFV Self-Service Automation... Error! Bookmark not defined. Network Orchestration & Automation Object Oriented Approach for NFV Test Automation... Error! Bookmark not defined. 8. Testing MANO NFV Orchestrator VNF Manager: Virtualised Infrastructure Manager (VIM) and NFV Infrastructure (NFVI): Orchestrator VNF Manager (Or-Vnfm) Virtualised Infrastructure Manager VNF Manager (Vi-Vnfm) Orchestrator Virtualised Infrastructure Manager (Or-Vi) OSS/BSS NFV MANO (Os-Nfvo) VNF VNF Manager Scalability and Performance Testing Scalability Performance Stress testing Failover Convergence Testing Test Setup P a g e 3

4 11. VNF Migration Testing P a g e 4

5 1. Scope The document provides a list of best practices to be used for ensuring smooth migration of Network Elements and Services to NFV environment. It is recognized that certain portion of the best practices and recommendation are not required in all cases of NFV implementations. Some part of testing, such as security and usability, we not addressed in this document, but still should be addressed when defining the right strategy for NFV testing. This document may apply to: - ISVs developing VNF and NFV based network services and functions - Telco operators in a process of implementing NFV 2. References This document is based on ETSI ISG NFV Standards. The following referenced documents are served as a baseline for this document: GS NFV 001 GS NFV 003 NFV GS NFV 004 GS NFV-PER 001 Network Functions Virtualisation (NFV); Use Cases Network Functions Virtualisation (NFV); Terminology for Main Concepts Network Functions Virtualisation (NFV); Virtualisation Requirements Network Functions Virtualisation (NFV); NFV Performance & Portability Best Practises 3. Abbreviations BHTR Busy Hour Traffic Rates BIOS Basic Input/Output System CDN Content Delivery Network CIFS Common Internet File System CPU Central Processing Unit DPI Deep Packet Inspection DUT Device Under Test FTP File Transfer Protocol HTML HyperText Markup Language HTTP HyperText Transfer Protocol HW Hardware IMS IP Multimedia Subsystem IP Internet Protocol KPI Key Performance Indicator MANO MANagement and Orchestration MOS Mean Opinion Score P a g e 5

6 MOS-AV N/A NF NFV NFVI NIC NID OS P2P PPP PPPoE PSNR QoE QoS SLA TC TCP VF VM VNF VQM VTA DNS NTP SSH NFS TA ISV MOS-Audio & Video Not Applicable Network Function Network Functions Virtualisation Network Functions Virtualisation Infrastructure Network Interface Card Network Interface Device Operating System Peer-to-Peer Point-to-Point Protocol Point-to-Point Protocol over Ethernet Peak Signal-to-Noise Ratio Quality of Experience Quality of Service Service Level Agreement Test Case Transmission Control Protocol Virtual Function Virtual Machine Virtualised Network Function Video Quality Metric Virtual Test Appliance Domain Name System Network Time Protocol Secure SHell Network File System Test Agent Independent Software Vendor P a g e 6

7 4. Introduction Network Functions Virtualization (NFV) is a concept introduced by network operators in 2012 [Network Functions Virtualisation Introductory White Paper]. Network Functions Virtualization (NFV) is a network architecture concept that proposes using IT virtualization related technologies to virtualize entire classes of network node functions into building blocks that may be connected, or chained, together to create communication services. NFV relies upon, but differs from traditional server virtualization techniques such as those used in enterprise IT. A virtualized network function, or VNF, may consist of one or more virtual machines running different software and processes, on top of industry standard high volume servers, switches and storage, or even cloud computing infrastructure, instead of having custom hardware appliances for each network function. The European Telecommunications Standards Institute (ETSI) has formed an Industry Specification Group on Network Function Virtualization (ISG NFV). The contributors of the NFV Introductory white paper as well as the ETSI ISG have identified Testing and QoE monitoring as one of the main use cases and subjects to address when implementing NFV environment. NFV Framework The NFV framework consists of three main components. 1. Virtualized Network Functions (VNF) are software implementations of network functions that can be deployed on a Network Function Virtualization Infrastructure (NFVI). 2. NFV Infrastructure (NFVI) is the totality of all hardware and software components which build up the environment in which VNFs are deployed. The NFV-Infrastructure can span across several locations. The network providing connectivity between these locations is regarded to be part of the NFV-Infrastructure. 3. Network Functions Virtualization Management and Orchestration Architectural Framework (NFV-MANO Architectural Framework) is the collection of all functional blocks, data repositories used by these functional blocks, and reference points and interfaces through which these functional blocks exchange information for the purpose of managing and orchestrating NFVI and VNFs. P a g e 7

8 Virtualised network Functions (VNFs) VNF VNF VNF VNF VNF NFV Management and Orchestration (NFV-MANO) NFV Orchestrator (NFVO) NFV Infrastructure (NFVI) Virtual Compute Virtual Storage Virtual Network VNF Manager (VNFM) Virtualisation Layer Compute Storage Network Hardware resources Virtualised Infrastructure Manager (VIM) High level NFV framework The building block for both the NFVI and the NFV-MANO is the NFV platform. In the NFVI role, it consists of both virtual and physical compute and storage resources, and virtualization software. In its NFV-MANO role it consists of VNF and NFVI managers and virtualization software operating on a hardware controller. The NFV platform implements carrier-grade features used to manage and monitor the platform components, recover from failures and provide effective security - all required for the public carrier network. NFV Use Cases The first standard issued by ETSI identified 9 use cases for NFV in GS NFV 001: Use Case #1: Use Case #2: Use Case #3: Use Case #4: Use Case #5: Use Case #6: Use Case #7: Use Case #8: Use Case #9: NFV Infrastructure (NFVI) as a Service Virtual Network Functions as a Service (VNFaaS) Virtual Network Platform as a Service (VNPaaS) VNF Forwarding Graphs Virtualisation of Mobile Core Network and IMS Virtualisation of Mobile Base Station Virtualisation of the Home Environment Virtualisation of CDNs (vcdn) Fixed Access Network Functions Virtualisation Each of those use cases requires different level of techniques and has different set of QoS and QoE KPIs. Nevertheless, from System Test point of view, same workload scenarios may apply for all use cases. P a g e 8

9 5. Assuring compliance and conformance to ETSI ISG Standards Assuring compliance aim to find the deviations from the ETSI ISG Standards. It determines whether the NFV is meeting the defined standards. Compliance is performed on two levels: - MANO - VNF/NE When performing compliance testing on NFV layer, each component is analyzed independently (MANO, VNFM, VIM, NFVI and interfaces). VNF/NE compliance is archived mainly by comparing the VNF and NE descriptor to the main Test Procedure Compliance tests are done using the following steps: Step 1: Analyze ISG current standards Step 2: Prepare compliance requirements Step 3: Analyze each NFV component and interfaces Step 4: Define gap points and analyze Step 5: Prepare correction plan Step 6: Monitor correction plan execution Step 7: Repeat steps 1-6 for each VNF and NE MANO Compliance Criteria Following criteria are evaluated: - VNF Descriptor format - VNF redundancy model - VNF state transitions - Host affinity and anti-affinity rules for deployment of VNFC instances - Authorization of the lifecycle management request - Validation of the lifecycle management request - VNF Package - Support multiple SWA-1 interfaces - Support multiple SWA-5 interfaces - Allocation of addresses - Dynamic allocation and dellocation of addresses - Scaling event types and format - Auto-scaling - Maintain records on VNF Packages P a g e 9

10 VNF Compliance Criteria Following criteria are evaluated: - VNF Design Patterns o VNF Internal Structure o VNF Instantiation o VNFC States o VNF Load Balancing Models o VNF Scaling Models o VNF Component Re-Use - VNF Update and Upgrade o Automatic procedure o Control Update and Upgrade process o Requesting virtual resources o Roll-back - VNF's Properties o Hardware Independence o Virtualization and Container Awareness o Elasticity o VOID o VNF Policy Management o Migration operations o VNF State o VNF Internal Structure o Reliability o Location Awareness o Application Management o Diversity and Evolution of VNF Properties - VNF Topological Characteristics o Deployment Behaviour o Virtualisation containers o NFVI Resources o Components and Relationship o Location - VNF States and Transitions o States and Transitions as Architectural Patterns o The VNF Descriptor o VNF Instantiation o VNFC Instantiation o VNFC Instance Termination o VNF Instance Termination P a g e 10

11 o VNF Instance Scaling o Start and Stop VNF o VNF Instance Configuration - VNF Fault Management o Virtualised resource faults o VNF faults P a g e 11

12 6. NFV Test Environment Build NFV Test Environment NFV provides great flexibility when building Test Environment. Using NFV, the end user can: - Build parallel test environments to address different needs - Scale up and down compute resources allocated based on type of testing (performance, functional etc.) - Copy any existing environment, either from development environment (once testing starts) or from production environment (to investigate production problems) When building Test Environment for NFV, the following guidelines should be kept: - Management and Orchestration should be similar to the production, including all elements of NFV (MANO, VNFM, VIM, NFVI) - NS and VNF Instantiation should be similar to production environment (hypervisors, computing, storage and network resources) - If possible, based on the available resources, same level of resources should be allocated. However, in case not enough free resources are available, a load environment should be established when running performance and scalability testing Test Appliances Test Appliances are required in order to: - Simulate valid workload traffic on the client and server - Simulate both data plane and control plane traffic - Measure key metrics, both data plane metrics and control plane metrics - Assuring the appropriate compute, storage and network resources are allocated - Active and passive monitoring of a single function, a set of functions or the entire service chain to ensure services adhere to established QoS and QoE metrics - Monitoring production SLA (not covered in this document) Virtual Test Appliances vs. Physical Test Appliances Existing physical test appliances may be used, but most test equipment vendors have developed virtual test appliances that work on separated VMs in the NFV Test Environment. Virtual test appliances offer equivalent capabilities for almost all scenarios and meets the flexibility required when testing NFV on multiple, geographically disparate. Virtual test appliances are usually much more cost effective. Physical test appliances are mainly recommended for the testing virtual environments that require the highest levels of data-plane performance (line-rate) or microsecond-level timing accuracy. P a g e 12

13 Simulating real workload traffic When simulating real workload traffic, consider the following steps: Step 1 measure production busy hours traffic rates (BHTR) Step 2 - Simulate a traffic mix on the client and server including: - HTTP - FTP - DNS - Streaming video - NTP - SSH - Syslog - NFS - Messaging - VoIP - Social Networks - Unicast Video - P2P - CIFS - Background Traffic Step 3 create different workload scenarios based on percentage of BHTR. The following traffic rates are recommended: Level % of BHTR Used for Low 5% Functional Testing Average 50% Scalability, on-going, fail-over Busy 100% Performance Stress 150% Stress Testing, auto-scaling Metrics NFV Service Metrics - VM Provisioning Latency, instantiation latency. Time between VM instantiation and first available packet - VM Stall (event duration and frequency) - VM Scheduling Latency QoS and Data-plane metrics: - Latency on each of tens of thousands of data streams P a g e 13

14 - Throughput and forwarding rate - Frame loss rate - Packet-delay variation and short-term average latency - Dropped frames and errored frames - Service Disruption Time for Fail-over Convergence QoE and Control-plane metrics: - HTTP: page load time, load time variance - Video: MOS-AV score, range = 2-5 with 5 being the best - HTML5 video - AS score, 100 % score as the maximum - Direct metrics: o Peak Signal to Noise Ratio (PSNR) o Structural Similarity (SSIM) - compare the original image with the received image o Video Quality Metric (VQM) o Mean Opinion Score (MOS) - This metric combines delays, perceived jitter at application layer, codec used for communication and packet loss at application layer - Indirect metrics: o Startup time: Time difference between sending the request for content and the time when the user actually received the content o Delivery synchronization - In a multicast many-to-many scenario it is important that the content is received by all participants at the same time. Consider online gaming or video conferencing o Freshness: The time difference between the time when the content is actually generated and the time when the users receives it, e.g. celebrating a goal with friends while watching a sports event o Blocking: When the buffers on the receiver are empty and the user has to wait for content. - Connections establishment rate, and transactions per second - Total number of connections, round trip time and goodput P a g e 14

15 7. Test Automation in NFV Disclaimer: The solution described in this document is based on CloadShell for SDN/NFV by QualiSystems. Similar solution may be built using a different tool. The Need for Heterogeneous SDN/NFV Self-Service Test Infrastructure Automation Software Defined Networking introduces the concept of network programmability from applications that interact with centralized SDN controllers via northbound APIs. This API-driven network paradigm opens the way for agile SDN application development, but also creates the need for networking organizations to deliver access to end-to-end network environments to application delivery stakeholders for supporting a DevOPS process. Network Function Virtualization (NFV) adds to the complexity of this picture by allowing agile service chaining of network functions hosted on virtual machines rather than hardware appliances. Automation of access to network sandboxes is complicated by the fact that SDN and NFV will gradually phase into networks, leaving significant portions of the network operating in a legacy mode, potentially for years. Networking teams need a self-service automation platform that can handle both SDN/NFV and legacy networks in a unified manner. Network DevOps Orchestration & Automation Testing NFV requires a comprehensive automation platform including resource management, provisioning, test automation and integrated reporting and business intelligence that is ideal for delivering self-service automation and continuous network certification processes for SDN and NFV. The following capabilities should be included: - Centralized inventory management of all legacy and SDN/NFV network resources allowing engineers to gain visibility to any components needed to design and publish network topologies required by developers and testers - Integration with all the existing and future infrastructure, including legacy network devices, SDN-enabled switches, SDN controllers and virtualized network functions - Visual network topology and service chain design and publishing - Visual workflow and test automation creation to build continuous integration - Integrated reporting and business intelligence - Sustainable object-based automation architecture - Easy to use web-based self-service portal P a g e 15

16 NFV Test infrastructure automation framework based on CloudShell Object-Based Test Automation Approach with TestShell TestShell allows all automation elements to be captured as small-scope objects to enable high reusability for test workflow construction. Objects are divided into the following categories: - NFV-Mano an automation object for each MANO operation, such as instantiation of Network Service, Disable VNF package etc. - NFV Infrastructure - an automation object for each NFVI operation, such as create, shutdown, destroy and update virtual machine (VM) - VNF Each VNF shall have dedicated objects based on its internal workflow. - Legacy Network Services each legacy network service shall have dedicated objects - Test Appliances each Test Appliance should have a list of Test Automation objects, such as simulate control plane traffic and retrieve QoE parameters P a g e 16

17 8. Testing MANO The first step is to test NFV Management and Orchestration Architectural Framework. NFV Orchestrator (NFVO) NS Catalog VNF Catalog Or-Vnfm Managem ent and Orchestrat ion Architectu re VNF Manager (VNFM) NFV Instance s Virtualised Infrastructure Manager Vi-Vnfm (VIM) NFV Instance s Or-Vi Manage ment and Orchest ration Architec ture NFV Orchestrator NFV Management and Orchestration Architecture - On-boarding of new Network Service (NS) and VNF Packages - Instantiate Network Service i.e. create a Network Service using the NS on-boarding artefacts - Query VNF retrieve VNF instance state and attributes - Check VNF instantiation feasibility - Scale Network Service, i.e. grow or reduce the capacity of the Network Service - Update Network Service by supporting Network Service configuration changes of various complexity such as changing inter-vnf connectivity or the constituent VNF instances - Disable, enable, update, query, delete VNF package - Create, delete, query, and update of VNF Forwarding Graphs (VNF-FG) associated to a Network Service - Create, delete, query, and update of Virtual Links (VL) - Query Network Service and assure all attributes retrieved properly P a g e 17

18 - Terminate Network Services, i.e. request the termination of constituent VNF instances, request the release of NFVI resources associated to NSs, and return them to NFVI resource pool if applicable - Get VNF performance metrics and notification - Simulate Notification and different fault information: o physical infrastructure (compute, storage, and networking related faults) o virtualised infrastructure (e.g., VM-related faults) o application logic (i.e. VNF instance related faults) - Monitoring and collection of information related to resource usage, including mapping of usage - Scheduled request regarding VNF instances - Global resource management, validation and authorization of NFVI resource requests - Resources sharing between VNFs - Create, update, delete, query, activate and de-activate policy (e.g., policies related with affinity/anti-affinity, management of VNF or NS scaling operations, access control, resource management, fault management, NS topology etc.) - Constraints management - SLA parameters - Network capacity adaptation to load - Coexistence with legacy network equipment - Controlling and managing Inventory of versions, releases and patches of all units of hardware and software - Maintenance, hardware and software exchange, SW upgrades, Firmware upgrades, repair - Manage log of all changes to inventory, unexpected events and maintenance activities - Administration domains and permissions VNF Manager: - VNF instantiation, including VNF configuration if required by the VNF deployment template (e.g., VNF initial configuration with IP addresses before completion of the VNF instantiation operation) - VNF instantiation feasibility checking - VNF instance software update/upgrade - VNF instance modification - VNF instance scaling out/in and up/down - VNF instance-related collection of NFVI performance measurements and faults/events information, and correlation to VNF instance-related events/faults - VNF instance assisted or automated healing - VNF instance termination - VNF lifecycle management change notifications - Configuration and event reporting between NFVI and the E/NMS P a g e 18

19 - Liveness checking of an VNF, e.g. watchdog timer or keepalive - Failure detection - Fault remediation of each VNF resiliency category - Virtualised Infrastructure Manager (VIM) and NFV Infrastructure (NFVI): - Resource catalog management - controlling and managing Inventory of software (hypervisors), computing, storage and network resources - Collection and forwarding of performance measurements and faults/events - NFV Infrastructure faults collection and remediation - Create, shutdown, destroy and update virtual machine (VM) - Create list of virtual VMs, query, reboot, suspend, resume, save and restore VM - Create, modify, list, delete and query storage pool - Create, delete, list, and query VM storage - Allocate, query, update scale, migrate, operate, release of NFVI resources, and managing the association of the virtualised resources to the compute, storage, networking resources, e.g. increase resource to VMs, improve energy efficiency and resource reclamation - Create, query, update and release resource reservation - Create, query, update, delete and notify of VNF Forwarding Graphs, e.g., by creating and maintaining Virtual Links, virtual networks, sub-nets, and ports - Add, delete, update, query and copy of software images - Root cause analysis of performance issues from the NFV infrastructure perspective - Mechanism for time-stamping of hardware (e.g. network interface cards, NICs and NIDs) - Create, update, list, query and delete hypervisor policies - Create, delete, update, list and query virtual network - Create, update, list, query, and delete subnet - Create, update, list, query and delete port Orchestrator VNF Manager (Or-Vnfm) - Resource related requests, e.g. authorization, validation, reservation, allocation by VNF Manager(s) - Sending configuration information to the VNF Manager, so that the VNF can be configured appropriately to function within the VNF Forwarding Graph in the NS - Collecting state information Virtualised Infrastructure Manager VNF Manager (Vi-Vnfm) - Resource allocation requests - Virtualised hardware resource configuration and state information (e,g. events) exchange P a g e 19

20 Orchestrator Virtualised Infrastructure Manager (Or-Vi) - Resource reservation and allocation - Virtualised hardware resource configuration and state information (e,g. events) OSS/BSS NFV MANO (Os-Nfvo) - Requests for network service lifecycle management - Requests for VNF lifecycle management - Forwarding of NFV related state information - Policy management exchanges - Data analytics exchanges - Forwarding of NFV related accounting and usage resords - NFVI capacity and inventory information exchanges VNF VNF Manager - Requests for VNF lifecycle management - Exchanging configuration information - Exchanging state information P a g e 20

21 9. Scalability and Performance Testing Scalability refers to the maximum number of control plane sessions that can be established. Examples include the number of PPPoX sessions or number of routing peers. For routers, number of routes per session and total number of routes in the routing table are also a measure of scalability. In NFV environments, VNF should have auto-scale feature, so resources (compute, memory, network etc.) should be scaled automatically in response to the varying network function performance needs. The purpose of this test is: - Ensure auto-scale works properly - Ensure resources consumption is efficient - Measure performance meets SLA in high performance workloads Scalability - Disable auto-scale and run low (5%), Average (50%) and busy (100%) traffic. Measure metrics for each traffic level. - Manually allocate resources for average traffic to meet SLA. Increase traffic to busy and manually scale resources until SLA is met. Compare resources allocations and analyze results - Enable auto-scale and measure SLA in average and busy traffic. Analyze resources allocation results - Run a dynamic test to shift between different traffic rates each 5 minutes randomly. Analyze results Performance - Run tests with low traffic. Increase traffic by 5% every 1 hour and analyze metrics - Run on-going busy traffic rate for at least 5 days and measure metrics over time - Run on-going busy traffic rate and execute all MANO tests Stress testing - Run stress traffic rate (150%) over 5 hours and measure metrics - Define resource limit to support full traffic and run stress traffic rate (150%) over 5 hours and measure metrics P a g e 21

22 10. Failover Convergence Testing Convergence time is one of the key metrics for validating SLAs and high availability in service provider networks. The requirements for failover convergence times can be in the order of milliseconds depending on the services and applications that are under consideration. These constraints are equally valid in an NFV environment as well. In NFV deployments, there is an added factor of variability where failover convergence time for a VNF can be impacted by the number of VNFs on the physical server that is converging to alternate routes. Convergence measurement involves the measurement of processing time of the trigger event in the control plane and the traffic switchover time. It is important in a multiple VNF deployment scenario that the convergence time of any VNF is not impacted by the other VNFs on the same physical server, so that the VNF continues to satisfy the SLAs for which it was provisioned. Test Setup Fail-over Convergence Test Topology (source: ETSI GS NFV-PER 001) - Provision a VNF for DUT with routing protocol functionality. - Provision two virtual test endpoints (VTA) connected to the virtual DUT. In order to avoid the performance impact of resource sharing between DUT and virtual test boxes, install the virtual test endpoints on a separate physical server. - Configure the pre-determined routing protocol on the two virtual test appliances and advertise same set of routes from both virtual test appliances. Use a preferred metric in the routes advertised from one of the virtual test appliances (primary next hop/path). - Configure L3 traffic between endpoints advertised by VTA 3 and endpoints advertised by VTA 1 - Configure liveness detection mechanisms such as BFD protocol on the VNF as well as VTA1 and VTA2. P a g e 22

23 11. VNF Migration Testing Each VNF deployed in the NFV environment requires independent testing. VNF Testing shall cover: - Static testing: static testing involves reviewing requirements and specifications to ensure completeness or appropriateness for the VNF - Conformance testing: see VNF Compliance Criteria - VNF-Mano Integration: o VNF Instantiation o VNFC States o VNF Scaling Models o VNF Component Re-Use o VNF Update and Upgrade o Virtualization and Container Awareness o Elasticity o VNF Policy Management o Migration operations o VNF State o The VNF Descriptor o Start and Stop VNF o VNF Instance Configuration o Virtualised resource faults o VNF faults - VNF Internal functionality testing specific functions for VNF - Scalability/Performance run Performance testing and validate resources allocation - End-to-end/system test: Testing VNF performance together with other VNFs and validate the introduction of the new VNF doesn t affect other VNFs functionality and performance P a g e 23