Operational Practices

Transcription

1 Operational Practices Deploying, Monitoring & Troubleshooting Performance Assured Business Services over DOCSIS An Operational Practice Prepared for the Society of Cable Telecommunications Engineers By Germain Levesque, Greg Spear, Scott Sumner Accedian Networks, Inc Blvd Alfred-Nobel, Suite N-410 Saint-Laurent (Montreal), Quebec, H4S 2A9 Canada 2015 SCTE

2 Title Table of Contents Page Number Introduction 4 1. Executive Summary 4 2. Scope 4 3. Background 5 4. BSoD Lifecycle Overview 5 5. NFV-Based vcpe System Architecture Architecture Overview Performance Assurance Functions Operations Integration Considerations Low-Touch vcpe Module Provisioning vcpe Management Communication Options 11 VNF Controller & vcpe Deployment & Setup Controller Provisioning Requirements Supported Hypervisors NFVI Resource Requirements Required Equipment NFVI Preparation Controller Provisioning Procedure Initial Preparation Confirming Installation & Setting Management Parameters Troubleshooting Establishing vcpe Connectivity Required Equipment vcpe Discovery & Configuration Configuring the vcpe Modules for Use Confirming Installation Troubleshooting 25 BSoD Service Deployment Key Performance Attributes Class of Service Model Required Equipment Calibration and Equipment Preparation Detailed Procedure Initial Preparation SAT Test on EVC SAT Test per CoS Recording of Results Analysis of Results and Examples Troubleshooting 41 BSoD Performance Monitoring Required Equipment Calibration and Equipment Preparation Detailed Procedure Configuring the TWAMP Generator (vcpe1) Configuring the TWAMP Reflector (vcpe2) Analysis of Results Troubleshooting 48 BSoD Troubleshooting Considerations Summary of Per-Metric Performance Objectives SCTE 2

3 2. Sampling Rate 49 Conclusions and Recommendations Conclusions Areas for Further Investigation or to be Added in Future Versions 51 Abbreviations 52 Bibliography & References 53 Annexes 54 Annex A. SAT Report EVC Test for 64B frames 54 Annex B. SAT Report Per-CoS Test 55 Title List of Figures Page Number FIGURE 1 - METRO ETHERNET FORUM SERVICE LIFECYCLE. 6 FIGURE 2 - VCPE: TRADITIONAL VS. VIRTUALIZED CUSTOMER PREMISES EQUIPMENT EXAMPLE7 FIGURE 3 - VIRTUALIZATION OF NID ARCHITECTURE USING NFV 8 FIGURE 4 - NFV-BASED VCPE CONTROL TUNNEL & FUNCTION LOCATIONS 8 FIGURE 5 - TYPICAL BSOD PERFORMANCE ASSURANCE FUNCTIONALITY & IMPLEMENTATION 9 FIGURE 6 - INSTALL-TRIGGERED VCPE MODULE PROVISIONING USING DHCP OPTION 60 & 43 WITH FQDN NAMING 11 FIGURE 7 - ILLUSTRATION OF DUAL MAC/IP VCPE MODULE ADDRESSING SCHEME 12 FIGURE 3 SERVICE ACTIVATION TESTING METHODOLOGY 26 FIGURE 4 POINT-TO-POINT TWO-WAY SAT DIAGRAM 29 Title List of Tables Page Number TABLE 1- COS MODEL (EXAMPLE) 27 TABLE 2 - CONFIGURATION/PERFORMANCE TEST RESULTS FORMAT 39 TABLE 3 - POLICING TEST RESULTS FORMAT 39 TABLE 4 - TYPICAL COS PERFORMANCE OBJECTIVES (CPOS) PER APPLICATION 49 TABLE 5 - STANDARD DEVIATION FOR VARIOUS REAL LOSS VALUES AND NUMBER OF SAMPLES SCTE 3

4 1. Executive Summary Introduction This document describes the operational practices recommended to deploy, monitor, and troubleshoot performance assured Business Services over DOCSIS (BSoD). Consistent with the performance Service Level Agreements (SLAs) that normally accompany such commercial services, the procedures, methods and typical values for Key Performance Indicators (KPIs) associated with Service Activation Testing (SAT), real-time Performance Monitoring (PM), SLA reporting, and in-service troubleshooting are outlined, covering the full BSoD service lifecycle. Tests conducted, and operational procedures outlined, employ a virtualized Customer Premises Equipment (vcpe) approach that uses Network Function Virtualization (NFV)-powered hardware modules to add Service Operations, Administration and Maintenance (SOAM), testing and performance monitoring functions lacking in cable modems, to the customer site. A typical NFV-based vcpe module and controller architecture is presented as an overview to set the context for the operational procedures that employ it. Performance assured BSoD can be efficiently deployed and maintained when consistent instrumentation is applied from initial deployment through monitoring and troubleshooting. Standards commonly employed in Ethernet service deployment and maintenance are used, including Ethernet Service OAM (IEEE 802.3ah), Layer 3 TWAMP monitoring (RFC-5357), and RFC-2544 turn-up testing. Operational procedures show how to validate each critical service parameter at each stage, and how to interpret the results in the context of troubleshooting and maintaining optimal service quality and to meet SLAs, complying with the service lifecycle and service definition requirements formalized in Metro Ethernet Forum (MEF) specified Carrier Ethernet standards and guidelines. Areas of future work are explored, including the development of operational guidelines to govern the installation, provisioning, management, and automation of NFV-based vcpe solutions, as well as extension of monitoring methods to virtualized infrastructure to strengthen the hybrid-cloud connectivity over BSoD business case. 2. Scope This document is intended as a reference for operational, product management, engineering, and technical personnel directly involved in the definition, deployment, maintenance, optimization, and evolution of BSoD. Background on NFV-based vcpe architectures is provided as context for the detailed operational procedures that employ this method and architecture to deliver and assure the performance of BSoD services. Detailed procedures start with covering considerations and practical methods involved in setting up a virtualized control environment for the test infrastructure and connecting vcpe to the controller, and then guide the reader through detailed steps of service activation, performance monitoring and SLA reporting referencing the common KPIs that govern common enterprise-grade commerical services. A section dedicated to BSoD troubleshooting explores some general guidelines that can assist Multiple Systems Operator (MSO) operations personnel interpret metrics collected, and offers general guidelines as to what limits should be respected to ensure BSoD meet SLA commitments SCTE 4

5 3. Background Commercial connectivity services command premium pricing, and cable MSOs have the infrastructure in position to take advantage of this segment. However, reliability and performance are more important than bandwidth, given that cloud computing, Software as a Service (SaaS) and data center connectivity are increasingly critical to operations. Over-the-top video, conferencing services and other bandwidthconsuming applications often share the same link. MSOs can grow BSoD revenue by offering assured Quality of Service (QoS) and SLAs committing to guaranteed uptime, bandwidth availability, and rapid Mean Time To Repair (MTTR). Such capabilities allow MSOs to offer sophisticated connectivity and managed services for important commercial segments including healthcare, education, hospitality, and financial services. DOCSIS 3.x cable modems do not offer integrated performance monitoring, service turn up testing, demarcation and other features required for efficient business services delivery. Also, the BSoD market is cost sensitive, eliminating the option to use Network Interface Devices (NIDs) that typically terminate and assure fibered enterprise connections. Complementing the capabilities of cable modems, vcpe strategies employing NFV can deliver full NID functionality using low cost, standards based vcpe modules, ensuring MSOs obtain the visibility and reporting capabilities required in the most operationally and cost efficient manner. This paper aims to provide concrete methods and procedures MSOs can employ to leverage these NFVbased vcpe strategies to deploy, monitor and maintain BSoD services alongside their existing operational infrastructure. 4. BSoD Lifecycle Overview Specific operational practices pertain to each phase of the BSoD service lifecycle: 1. Deployment: Provisioning and service activation testing 2. Performance Monitoring & SLA Reporting: collecting and presenting key performance metrics 3. Troubleshooting: Techniques to identify, isolate and troubleshoot service issues These three phases are consistent with the MEF definition of the Carrier Ethernet service lifecycle, which serves as an established model for commercial connectivity SCTE 5

6 Figure 1 - Metro Ethernet Forum service lifecycle. 1 This paper is structured to address the operational practices associated with each stage of this service lifecycle, as it applies to BSoD. 5. NFV-Based vcpe System Architecture The operational practices and procedures described in ths paper are implemented using a networkembedded architecture that employs small footprint, programmable performance assuranace hardware modules (vcpe modules) augmented by virtualized performance assurance functions hosted on a centralized controller(s). Section 5 here introduces this architecture, as well as operational considerations that facilitate integration of this approach with existing Operational Support Systems (OSS) and Network Management Systems (NMS) Architecture Overview In the context of this document, the term vcpe will refer to the strategy of virtualizing as many customerlocated networking functions as possible, while retaining the minimum hardware necessary for service delivery, consistent with performance, reliability, and Quality of Experience (QoE) expectations. An example of a vcpe strategy where onsite hardware appliances performing firewall, PBX, and routing funcitons have been virtualized is illustrated in Figure 2, below. Virtualization is accomplished by transferring local networking functionality to software-based, Virtual Network Functions (VNFs) which can be hosted on low-cost, Commericial Off-The-Shelf (COTS) servers or cloud-infrastructure. 1 MEF 39: Service OAM Performance Monitoring YANG Module Technical Specification 2015 SCTE 6

7 Figure 2 - vcpe: Traditional vs. Virtualized Customer Premises Equipment Example In the context of BSoD, this approach can be used to introduce customer premises-located performance monitoring, turn-up test, SOAM and troubleshooting functionality which in the case of fiber business services is normally provided using a NID. As BSoD is a more cost-sensitive service than full, fibered enterprise connections, a NID often costing several times more than a cable modem is not normaly a feasible CPE option. NFV-powered hardware modules can offer the same level of performance monitoring precision, loopback and full line-rate turn-up test capabilities at a fraction of the cost of a NID, making this approach an economically viable fit when deploying SLA-grade BSoD. In addiiton to supplementing the cable modem with performance assurance features, this approach has a number of other benefits: 1. Truck-rolls are reduced over the service lifecycle when compared to handheld test sets, as a single vcpe module can remotely perform turn-up testing, continuous monitoring, and on-demand troubleshooting. 2. Compatibility with existing hand-held Ethernet test sets and third-party centralized monitoring probes allows straightforward integration into existing operational practices and infrastructure. 3. By employing NFV, new functionality can be added to the vcpe module remotely, without impacting the service. This allows MSOs to introduce new, performance assured commercial services (e.g. burstable or on-demand Ethernet connections) without requiring new equipment onsite. An example of how NID functionality can be virtualized using NFV is shown in Figure SCTE 7

8 Figure 3 - Virtualization of NID Architecture Using NFV The connection between the performance assurance VNF controller and each module needs to be reliable, secure and lossless (e.g. TCP-based) to ensure that the vcpe module can assume the same level of functionality as a NID. This management tunnel is critical to support performance assurnace VNFs, as raw data is returned to the controller for test results calculation, performance montoring, and fault reporting, in addition to performance monitoring session control, module management, synchronization information, etc. In an NFV-based vcpe architecture, the lossless control sessions allow each remote module to virtually become a remote port of the controller, which is analagous to a virtualized NID that can support many remote endpoints. Figure 4 - NFV-based vcpe Control Tunnel & Function Locations Some vcpe architectures virtualize customer-premises functions with a simple COTS server at the customer site. This is impractical in BSoD deployments because: 1. The cost of a COTS server is significantly higher than compact vcpe modules. 2. Performance assurance functions implemented purely in software lack sufficient time stamping precision and packet transmission scheduling control to meet the requirements of: a. Full line-rate test traffic generation and loopback for SAT and troubleshooting. b. Precise traffic generation sequencing required by common turn-up test standards (where inter-packet delay needs to be controlled for burst testing, for example). c. Microsecond-level latency measurement precision required to monitor and report on commerical services SLAs SCTE 8

9 Aside from incorporating these capabilties, vcpe modules also offer a number of other advantages over traditional test set and centralized probe solutions: 1. Modules can monitor and test between themselves: for site-to-site monitoring, end-to-end turn-up testing and troubleshooting between customer service endpoints. Most probe-based solutions are limited to loopbacks or monitoring tests from a central location to a service endpoint. This huband-spoke topology does not test the actual service path between customer locations, or between a customer and a remotely hosted data center, for example. 2. Test sets require trained technician dispatch to each service endpoint requiring service activation testing or troubleshooting, which is much less responsive and much more costly than a remotely initiated test using vcpe modules that are initially installed during service provisioning Performance Assurance Functions NFV-based vcpe solutions must be capable of all performance assurance functions required to support the business services lifecycle, as described in Section 4. These include, but are not limited to: 1. Standards-based service activation testing supporting commonly employed IEEE RFC-2544 and ITU-T Y.1564 turn-up testing approaches. 2. Ethernet Connectivity Fault Management (CFM), as defined by IEEE 802.3ag to ensure service availability meets SLA definitions, and to measure continuity and latency using CCM and DMM/DMR messages, respectively. 3. Standards-based performance monitoring for Layer 2 (Ethernet) and Layer 3 (IP) services, typically implemented using ITU-T Y.1731 / IEEE 802.3ah Ethernet SOAM and RFC-5357 Two- Way Active Measurement Protocol (TWAMP), respectively. 4. Bandwidth utilization monitoring, per port and per service flow (as defined by the MSO: VLAN, CoS, source or destination MAC or IP address, etc.) for usage-based billing, trending, and troubleshooting. Figure 5 - Typical BSoD Performance Assurance Functionality & Implementation 2015 SCTE 9

10 5.3. Operations Integration Considerations Although outside the scope of this paper to cover in depth, implementation of an NFV-based vcpe solution should interwork with existing operational support systems to permit integration with existing management practicies and procedures, and to make deployment of vcpe modules as well as the monitoring and maintenance of the services they support as operationally efficient as possible. Main areas to consider include: Deployment and management of the solution itself, to facilitate and automate element management of remote vcpe modules and BSoD service provisioning. Integration with SLA reporting platforms and fault management systems to harmonize monitoring, and reporting and within existing tools. As introduction and general guidelines, the following opeational aspects should be considered during solution selection and deployment Low-Touch vcpe Module Provisioning Ideally, vcpe modules should be ready to install in factory default configuration, without requiring prestaging by the MSO. To make that possible, the modules must be discoverable by an inventory system that can attribute the module to a particular customer site. This may be accomplished by relating the module to the MAC or IP address of the customer s cable modem, for example. To come under management control without requiring pre-staging or on-site configuration by a trained technician, the units require a method to discover the mangement environment, have their managmeent IP address defined, have the desired configuration provisioned on the unit, etc. Auto-Provisioning Example using DHCP Option 60/43 & FQDN One commonly employed method to bring devices under managmeent involves using Dynamic Host Configuration Protocol (DHCP) with Options 60 & 43: 1. When a vcpe module is connected to the network, it announces itself using a DHCP request with option 60 enabled, which communicates the unit s identifier (device type) to the DHCP server. 2. When properly configured, the server assigns a temporary (dynamic) IP address to the unit, and responds with Option 43, providing the address of the inventory node responsible for managing the unit. 3. Once under management control, a static IP can be assigned (if desired). A Fully Qualified Domain Name (FQDN) can also be assigned to the module. The FQDN must remain in sync with any link-state change (per RFC-2131), typically realized using automated DNS queries by the module inventory node. Supporting FQDN allows the vcpe module to then be integrated into the cable modem inventory management system, alongside the cable modem supporting the BSoD service SCTE 10

11 Figure 6 - Install-Triggered vcpe Module Provisioning Using DHCP Once the unit is under management control, automation may be used to trigger an immediate or scheduled turn-up test to validate the service, then provision customer/sla-specific monitoring sessions, etc. to allow customer-level self-install vcpe Management Communication Options The vcpe module should be capable of adapting to an MSO s particular management and device addressing methodology. This requires the unit to distinguish customer/test traffic from management communication. A variety of methods are commonly used, each implying the vcpe module must offer support for each of these schemes: Layer 2 addressing: separate management and customer MAC addresses. In this case, the vcpe module must support two MAC addresses, one for management traffic, and another for customer, test, CFM, SOAM and active performance monitoring traffic. Layer 2 addressing: separate management and customer VLANs. A single MAC address is used in combination with Q-in-Q VLAN support (C/S tagging, IEEE 802.1ad). Layer 3 addressing: similar to the Layer 2 scheme described above; separate management and customer traffic IP addresses may need to be supported by the vcpe module, depending on the method used by the operator. VLAN support, including Q-in-Q (S-VLAN), may also be required to support this scenario. Layer 3, transparent IP addressing: an operator may elect not to assign new IP address(s) to the vcpe module, instead using the address of a device located behind the module. This is practical when the operator has a known device at the customer premises (e.g. a set-top box, Wi-Fi controller, security gateway, etc.) In this case the vcpe module detects management traffic using 2015 SCTE 11

12 a combination of this other device s IP address in combination with other identifiers (such as a management VLAN tag). Figure 7 - Illustration of Dual MAC/IP vcpe Module Addressing Scheme VNF Controller & vcpe Deployment & Setup The NFV-based vcpe used as a reference configuration for procedures outlined in this paper are managed by a performance assurance controller, embodied as a virtual appliance. This section will show how to install the VNF controller (VCX) on a VMware ESXi hypervisor to the point where it is ready to login to and use for the procedures in subsequent sections of this paper, as well as pointing out common management configuration requirements. This is followed by a section describing the method to establish connectivity to the two NFV-based vcpe modules that will be used to conduct tests outlined in the remainder of this paper. 1. Controller Provisioning Requirements The following hardware, operating system and hypervisor guidelines outline the requirements to host the performance assurance VNF controller (specifically, in the procedures outlined in further sections of this paper, the Accedian SkyLIGHT VCX Controller, release 1.2): 1.1. Supported Hypervisors The VCX Controller can be installed on VMWare ESXi v5.5 or higher, or KVM Cent OS 6. VMWare will be used for subsequent operational procedures, as well as those outlined in this section SCTE 12

13 1.2. NFVI Resource Requirements The recommended system resources for the VCX Controller are: CPU: 4 single cores minimum up to 8 quad cores depending on the capabilities and resources. HDD: Minimum of 2 Gb dynamic with thin provisioning. RAM: Minimum of 1 Gb, recommended 4 Gb. Network Ports: Minimum of one (1) for network access, but shall be a minimum of two (2) in practice to allow management and network accesses. If possible, all ports shall be dedicated NICs accessible to the Virtual Machine (VM) instance to allow load distribution Required Equipment X86-based PC or server running an operating system that supports the VMware ESXi 5.5 hypervisor 2, as specified on the VMware compatibility page: vmware.com/resources/compatibility VMware ESXi install files ( vsphere Virtual Machine management application ( Licensing required to support the scale of the particular MSO BSoD implementation ( VCX Controller OVA virtual appliance image file, example: VCX_1.1_3149_VMWare.ova 2. NFVI Preparation The NFV Infrastructure (or NFVI) environment must be prepared before installing the controller virtual appliance (application running as a VM instance) that will be used. This example uses the VMware ESXi hypervisor version 5.5 to run the VCX application. This can be set up on any server that supports VMware. In this example the server being configured is an HP DL160, certified and supported by VMware, with a 250 GB drive and 16 Meg of RAM. Procedure: 1) Power up the server and boot from an external CD or USB with the appropriate ESXi version. 2) Follow the on-screen installation instructions. 3) Once completed, the ESXi hypervisor is now ready to support the controller virtual appliance application as a VM. 4) The vsphere application is used to simplify management of virtual machines running on on the ESXi hypervisor. Install the vsphere application using the instructions provided by VMware. 5) The vsphere application can also be used to set network parameters, including the IP address, of the virtual appliance. 2 Alternatively, an X86-based PC or server running an operating system that supports the KVM Cent OS 6. As stated previously, for this example we use VMware SCTE 13

14 Figure 8 - vsphere Interface: Network Parameter Configuration Screen 3. Controller Provisioning Procedure 3.1. Initial Preparation 1) The vsphere application will be used to load and start the VCX Conroller virtual appliance (the.ova file). Select file deploy OVF template and follow the instructions as prompted during install SCTE 14

15 2) Right-click on the VCX tab and select Power On. The VCX appliance entry provisioned have a green triangle icon next to it, indicating it is running SCTE 15

16 3) To configure the VCX Controller, open a console to the VCX by right-clicking the entry and selecting Open Console. 4) Set the IP address of the application using the Command Line Interface (CLI). The screenshot below is an example of commands used to disable DHCP and set the IP address. Refer to the VCX Controller User Manual for additional information SCTE 16

17 5) Now the VCX Contoller can be accessed by using the assinged IP address in a web brower: 3.2. Confirming Installation & Setting Management Parameters The VM and the VCX application are now running. To verify the installation is successful, proceed with the following steps: Log out of the VM and exit the vsphere application SCTE 17

18 Exit and log back into the VCX using the address configured in the previous steps. The VCX Controller Graphical User Interface (GUI) can now be used to configure all the usual required fields typically required for management: Discovery options Adding remote devices SNMP traps Remote Syslog Timing: NTP 1588 or manually configured Adding users Configuring routing requirements (default route, gateways etc ) Please consult the VCX Controller User Manual for complete details on how to configure each of these management parameters consistent with the target operating environment Troubleshooting If you unable to connect to the VCX: Verify you have a valid IP address. Verify the link status on both devices. Check if a firewall is preventing or blocking the connection. Verify there is not a duplex mismatch (one port at half duplex and the other full duplex). Check for physical errors on the network. Ensure you are not using a duplicate IP address. 4. Establishing vcpe Connectivity 4.1. Required Equipment Functioning VCX Controller as specified above. Two compatible NFV-based vcpe modules: Accedian antmodules (model ANT-1000-TX, part number ) vcpe Discovery & Configuration The remote vcpe modules will be automatically discovered by the VCX Controller in a Layer 2 or Layer 3 network. The vcpe modules can get their addresses automatically via DHCP, or can be staged with a predefined address. For this paper, simple Layer 2 discovery will be used. For a more complete overview of the discovery options, please refer to the user guide. The following diagrams show how the discovery process works. No pre-configuration of the vcpe modules is required SCTE 18

19 Figure 9 - Layer 2 Discovery Method as Employed by the VCX Controller 1) Go to the Discovery/configuration tab. Then, select Add. 2) Select the interface that is connected to the network. In this example we are using the Management interface. Any other configured interfaces will be available from the pull down menu: 2015 SCTE 19

20 3) Once the discovery is setup, switch to the Inventory tab to see if the unit was discovered. 4) In this example we see ANN-1000-CX was identified and added to the VCX Controller inventory. 5) To start managing the units, select the unit using the box on the left and click Add: 2015 SCTE 20

21 6) Once under management, the unit will be available in the Remote Devices tab. 7) The unit will initialize and update if required. The update could take a few minutes. 8) Once the unit is linked and authorized you can click on the unit and configure a unique name and click Apply. 9) This is the view in Remote Devices once the name is configured and the updating is complete: 10) Verify the module s ports are activated using Port tab SCTE 21

22 4.3. Configuring the vcpe Modules for Use Depending on the test to be performed, other configuration steps are required. For example, a simple traffic generator test requires setting up the traffic generator and pointing it to the MAC address of the destination/remote vcpe module reflecting the traffic. This example will test from the MyNanoNID vcpe module to the central reflector vcpe module located at a head-end location: The traffic generator will be set up in the MyNanoNID vcpe module. 1) Go to the SAT tab, configure as needed, and then click Add. 2) Below is the configuration to generate one test flow at 50 Mbps for 30 seconds. Once parameters are entered, click Apply to save the configuration SCTE 22

23 3) To run the test go to the Result tab, and select Detailed Report: 2015 SCTE 23

24 This simple traffic generation test is a simple way to quickly validate that the vcpe module is under management and is in good working order Confirming Installation To this point we have confirmed the vcpe module was discovered, is under management control, and is functioning properly as shown using a simple traffic generator test. To complete confirmation that the vcpe module has the correct firmware and configuration required for tests and procedures outlined in this paper, follow the steps below. 1) Visit the Configuration tab and verify the unit has both PMON (performance monitoring) and TGEN features available, and select the Active Firmware for each as shown in the series of screen captures below: 2015 SCTE 24

25 2) Verify that the values in the pull-down menu include PMON and TGEN. 3) The vcpe modules are now ready to use for the procedures outlined in the following sections of this paper Troubleshooting If you are unable to discover the remote unit the most common cause is issues with network connectivity the path between the VCX and the vcpe module must be open and reliable. Is the remote device powered up and is the port active? Verify the LED on the vcpe module. If the unit is discovered but you are not able to manage it, it may already be managed by another VCX. If a unit is discovered and managed but will not perform a test it may be in the wrong mode. Set the module Out Of Service (OOS) setting to In Service in the Remote Devices screen and try again SCTE 25

26 BSoD Service Deployment SAT is a key part of the Carrier Ethernet service operations life cycle. Before a service provider deploys an Ethernet service to a customer, it must be validated to ensure the participating network devices have been configured properly and the service meets the defined SLA, or as commonly referred to by the MEF, the Service Level Specification (SLS). This testing also provides baseline service measurements to create a Carrier Ethernet service birth certificate as a benchmark for troubleshooting, and to report a successful service turn-up to the customer. The service activation testing methodology, as illustrated in Figure 10, addresses these requirements. Figure 10 Service Activation Testing Methodology 3 5. Key Performance Attributes The following key performance attributes will be measured in a Service Activation Test: Frame Transfer Delay (FTD) The time required to transmit a Service Frame from ingress Service Activation Measurement Point (SAMP) to egress SAMP. Frame Delay Variation (FDV) The absolute value of the difference between the FTD of two consecutive Service Frames belonging to the same Class of Service (CoS) stream. Frame Loss Ratio (FLR) 3 Source: MEF CE 2.0 Service Management Life Cycle White Paper 2015 SCTE 26

27 A characterization of the number of lost frames between the ingress SAMP and the egress SAMP for Service Frames that have the same CoS. The Frame Loss Ratio Performance is expressed as a percentage. 6. Class of Service Model The choice of a specific CoS model belongs to the operator. The MEF 23.1 standard specifies a set of three CoS Names called Labels that can be used by operators to indicate the performance expectations to be associated with a given set of frames that comprise a CoS Frame Set. CoS Labels are H, M and L, which informally refer to High, Medium, and Low. The order of the CoS Labels is based on the traffic classes and their associated PCP values. Operators chose how specific applications are mapped to these CoS labels. They can also define how the priority (PCP or DSCP) will be mapped at a User Network Interface (UNI) for multi-cos Ethernet Virtual Connections (EVCs) that support all 3 MEF CoS Labels. Finally, the CoS Performance Objectives (CPO) is defined for each Performance metric and per CoS Label. Defining a CoS model for specific applications is beyond the scope of this paper. Operators should refer to MEF 23.1 (see References section) for more information. The following CoS model will be used as an example in this paper to illustrate the principles of SAT. PCP mapping and Committed Information Rate (CIR) values are provided. Table 1- CoS Model (Example) CoS Label PCP CIR H 5 2 Mbps M 3 3 Mbps L 1 5 Mbps 7. Required Equipment One Y.1564 generator unit. One Y.1564 reflector unit In this procedure, both units are Accedian antmodule vcpe units, under the control of the SkyLIGHT VCX Controller. 8. Calibration and Equipment Preparation The Y.1564 generator shall be configured to compute rates at Layer 2. To do so, proceed as follows: 1) Log in to the GT Performance Element. 2) Access the page Traffic Configuration. 3) Set the Generator working rate to Layer 2 (as shown in the following picture) and click Apply SCTE 27

28 9. Detailed Procedure 9.1. Initial Preparation In the context of SAT, the performance measurements may be performed one-way or two-way (roundtrip). One-way measurements require time synchronization at each Service Activation Measurement Point. For the sake of simplicity, two-way SAT will be used in the following procedures. The Y.1564 methodology shall be preferred over RFC-2544 because of its ability to test multiple Classes of Service simultaneously. An operator may also want perform a SAT test on a circuit or EVC before configuring the various Classes of Service to make sure it conforms with the overall Service Level Specifications (SLS). This initial SAT test can help identify basic issues and troubleshoot them before performing more detailed per-cos configuration and SAT testing. Following the initial EVC SAT test, more sophisticated tests shall be performed par CoS to validate the performance objective for each service SAT Test on EVC The purpose of this test is to validate the performance objectives of the circuit/evc. No CoS shall be configured at this point. The test shall be run for various frame sizes, as required by the Operator. In this example, we will test the EVC at the CIR rate sequentially for 64B, 128B, 256B, 512B, 1024B, 1518B frames SCTE 28

29 Figure 11 Point-to-Point Two-way SAT Diagram 1) Set up the Y.1564 generator at the measurement point where the test traffic will be generated. 2) Set up the Y.1564 reflector (loopback) unit at the far-end measurement point. 3) Configure the SAT test as follows: a) Get the far-end vcpe MAC address Log in to the SkyLIGHT VCX Controller Access the page Remote Devices Configuration. This displays the screen shown in the figure below. Take note of the MAC address for vcpe module (2) that will be used as a reflector at the far end. This information will be required to configure the Y.1564 test generator. b) Set up the Y.1564 Test on the far-end vcpe module Log in to the VCX Controller and access the page SAT Y.1564 Configuration. Click the Add button to create a new test profile. Enter the test configuration as shown in the following figure. In the MAC Destination field, type the MAC address of the vcpe module (2) that will be used as a reflector. Click Apply to save the test configuration SCTE 29

30 Explanations: The Configuration Test is enabled and configured to run for 10 seconds. This test validates each defined service to make sure that the configuration is correct. The Performance Test is enabled and configured to run for 15 minutes. This test validates the quality of the services over a medium to long time duration. The specific duration is to be determined by the Operator according to their operational pratices. The Layer-2 test frame type is selected for this test. With this configuration, the generator sends an LBM frame to the reflector device (MAC 00:15:AD:1B:57:60). The reflector swaps the MAC destination and source addresses and returns an LBR frame. c) Set up the Y.1564 Services Go to SAT Y.1564 Configuration. This displays a summary of all test profiles. Click the Name of the test (EVC Test) to edit its settings. Click the Name of a service (e.g. Service_1) from the Service List to edit its settings. This displays the screen shown in the figure below SCTE 30

31 Enter the service configuration parameters for the 64B frame test flow as shown in the figure above and then click Apply. Explanations: The CIR is set to 10 Mbps in order to measure the overall throughput. The Configuration Test is enabled. This will add the following rates to the configuration test: 25% CIR, 50% CIR and 75% CIR. The acceptance criteria shall be configured as per the Service Level Specifications SCTE 31

32 Repeat the service configuretion steps as described above for each other frame size that needs to be tested, namely for 128B, 256B, 512B, 1024B and 1518B. Do not enable the state of these services for now. The following figure shows the six services (frames sizes) once they are configured: Six services (frames sizes) have been configured in the test profile. The test shall be run for the first service (64B frames), then for the second, and so on until all frame sizes have been tested. d) Run the Y.1564 Test Go to the page SAT Y.1564 Results. This displays a summary of all test reports. Click the Start new test button to start the test and generate a report for the first service (64B). This displays the screen shown in the figure below SCTE 32

33 Enter the a test report description and any other optional information as needed and and click Run. e) View a Summary of the Test Results Go to the page SAT Y.1564 Results. This displays a summary of all test reports. To view detailed results from a test, click the name of the test report. While the test is running, this displays the screen shown in the figure below. To abort a test while it is running, click Stop. When the test is completed, the following screen will be displayed: 2015 SCTE 33

34 To export the detailed report into a text file and save it on the management station, click Export. See Appendix A to view the completed report from this example. Go back to the page SAT Y.1564 Configuration and click the name of the test (EVC Test) to edit its settings. To test the EVC with 128B frames, disable the first service (Frame Size 64B) and enable the second one (Frame Size 128B). Run the test again and save the report as described previously. Repeat these steps for each frame size SAT Test Per CoS The purpose of this test is to validate the performance objectives per CoS. All CoS shall be configured in the network at this point in order to be tested simultaneously. Configure the SAT test as follows: a) Set up the Y.1564 Test on the near-end vcpe (1) module Log in to the SkyLIGHT VCX. Access the page SAT Y.1564 Configuration. Click the Add button to create a new test profile. Enter the test configuration as shown in the following figure. In the MAC Destination field, type the MAC address of the vcpe (2) that will be used as a reflector. Click Apply to save the test configuration SCTE 34

35 Explanations: The Parallel Configuration Test is enabled in orther to validate all defined service simultaneously. From the Service List, three CoS shall be configured and enabled. This will allow running the Performance Test on all services simultaneously. b) Set up the Y.1564 Services Go to SAT Y.1564 Configuration. This displays a summary of all test profiles. Click the Name of the test (EVC Test) to edit its settings. Click the Name of a service (e.g. Service_1) from the Service List to edit its settings. This displays the screen shown in the figure below SCTE 35

36 Explanations: The CIR and VLAN Priortity (PCP) are configured as defined in the CoS model. The Policing Test is enabled. The purpose of this test is to send traffic above the rate of CIR+EIR in order to verify that the bandwidth policer is operating correctly and discards frames that exceed CIR+EIR. The Frame Size type is set to EMIX, which allows testing with a pattern of variable frame sizes. The available sizes are: a=64b; b=128b; c=256b; d=512b; e=1024b; f=1280b; g=1518b; h=port MTU; u=user-defined size SCTE 36

37 Repeat the service configuration steps as described above for other two CoS, with their respective CIR and PCP values: c) Run the Y.1564 Test Go to the page SAT Y.1564 Results. This displays a summary of all test reports. Click the Start new test button to start the test and generate a report for the first service (64B). This displays the screen shown in the figure below SCTE 37

38 Enter the a test report description and any other optional information as needed and and click Run. d) View a Summary of the Test Results Go to the page SAT Y.1564 Results. This displays a summary of all test reports. To view detailed results from a test, click the name of the test report. While the test is running, this displays the screen shown in the figure below. This time, we see that the three CoS are being tested simulteneously. When the test is completed, the following screen will be displayed: To export the detailed report into a text file and save it on the management station, click Export. See Appendix A to view the completed report from this example. 10. Recording of Results The SAT test produces a detailed report that can be exported to a local workstation or remote server in.txt or.xml format. The following tables show the format used in the report for each test. Examples of the complete test reports produced in the previous examples are available in Annex A and B SCTE 38

39 Table 2 - Configuration/Performance Test Results Format IR FL FTD FDV Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR PASS e PASS e PASS e PASS e The first columns display the information rate (CIR/EIR) that is generated. The second column (Pass/Fail) indicates whether the test is a PASS or FAIL, based on the configured acceptance criteria. The next three columns, under IR, display the measured Information Rate values. The next two columns, under FL, display the measured frame loss. The Cnt field shows a count of the number of lost frames, if any, up to Higher counts would be marked >100k. FLR displays the measured frame loss ratio. The last six columns display the measured Frame Transfer Delay (FTD) and Frame Delay Variation (FDV) values. Table 3 - Policing Test Results Format IR FL FTD FDV Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) POLICING PASS e CIR*(1-FLRsac) <= IR <= CIR+EIR+M 2*( ) <= <= All column headers are the same as in the Configuration and Performance tests results. The difference is that a successful Policing test is expected to show a certain FLR that is greater than 0 and even excessive FTD and FDV, since its purpose is to validate that the bandwidth policer is doing its job when we attempt to exceed the allowed bandwidth SCTE 39

40 11. Analysis of Results and Examples Annex A and B show examples of successful SAT tests. These tests used the following Service Acceptance Criteria: FTD FTD type FDV FDV type FLR M factor : us : max : 2500 us : max : 1.00e-06 : 1 Mbps The following test report excerpts show a few examples of failed tests and their cause: Failed test due to excessive Frame Loss CONFIGURATION Started at : :47:42-04:00 IR FL FTD FDV Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR FAIL e Failed test due to excessive Frame Transfer Delay CONFIGURATION Started at : :13:00-04:00 IR FL FTD FDV Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR FAIL e Failed test due to excessive Frame Delay Variation CONFIGURATION Started at : :28:35-04:00 IR FL FTD FDV 2015 SCTE 40

41 Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR FAIL e Failed test due to inadequate bandwidth policing IR FL FTD FDV Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR PASS e PASS e PASS e PASS e CIR/EIR N/A POLICING FAIL e CIR*(1-FLRsac) <= IR <= CIR+EIR+M 5*( ) <= <= Troubleshooting This section describes a few typical setup or configuration issues that can cause a SAT test to fail, and how to troubleshoot them. Error Message: Test Overall Result: PEER NOT FOUND This error is usually encountered when the peer unit (reflector) fails to return any test frames. 1. Verify that all physical connections are made properly end to end. 2. Verify the port status at both Service Activation Measurement Points. All ports must be up. 3. Verify the SAT test configuration and make sure that the MAC Destination address is the reflector vcpe s MAC address. 4. Make sure that the Y.1564 generator is configured with the proper type of test frame (Layer 2) SCTE 41

42 The test fails for specific frame sizes 1. Verify the MTU on all devices, including any network element that may be installed between the SAT generator and the reflector vcpe modules. The MTU must large enough to support the largest configured test frame. 2. Verify if any network element adds a VLAN tag anywhere on the path that is being tested. A VLAN tag adds 4 bytes to a frame and may cause the MTU to be exceeded. BSoD Performance Monitoring Once a service is validating using the SAT tests described in the previous section, real-time performance monitoring allows the operator to measure and report SLA metrics, and monitor KPIs for troubleshooting, proactive fault mitigation, trending, and many other important aspects of maintaining a performanceassured service that meets customer expectations and contractual obligations. In this example the previously configured units will be used to demonstrate performance monitoring configuration steps, reviewing results, and explore some common troubleshooting steps. The most common method of measuring performance in the BSoD model is TWAMP, Two Way Active Measurement Protocol (RFC-5357), for Layer 3 monitoring. This will be the method used for this example. Other standard methods are also available -- Y.1731 for example -- and will provide similar results for Layer 2 measurements. The operator will normally select their preferred method based on the type of network: Layer 2 or Layer 3, and the supported standards in the equipment deployed in the network. These standard employ active test sessions by sending test traffic between two endpoints and monitoring the results, where thresholds will trigger an alarm if a value is exceeded. As an example, if packet loss (PL) is approaching an SLA limit, the operator can be notified by an alarm. This will allow the provider to fix any network issues before the SLA (Service Level Agreement) with the customer is violated, if the alarm thresholds are set accordingly. 1. Required Equipment One Y.1564 generator unit. In this procedure, we are using an Accedian GT Performance Element. One Y.1564 reflector unit. In this procedure, we are using an Accedian ant Performance Module and Skyligt VCX Controller. 2. Calibration and Equipment Preparation The equipment listed above was prepared in the SAT section, and is ready for PM monitoring configuration. No further preparation is required SCTE 42

43 3. Detailed Procedure Figure 12 - TWAMP Test Setup Diagram TWAMP is a Layer 3 test protocol and requires an IP address at both ends. For this example, the generator will be assigned and the reflector will be assigned In a production network the addresses should be validated with a Ping test to make sure the network routing is in order before configuring the TWAMP probes Configuring the TWAMP Generator (vcpe1) 1) Login to the GUI of the VCX using the IP address assigned earlier. 2) We will assign IP address to the device, for this example the devices are called Location A (vcpe1) and Location Z (vcpe2), 3) The device names can be edited in the remote device tab. 4) The screen will look like the screenshot shown below. Click Add to configure the device SCTE 43

44 5) The configuration will appear as shown below. After entering the necessary information,click Apply. In this example there is no gateway, but in a production network this will probably be required. 6) Two ports appear in the pull down menu: LocationX UNI and LocationX NNI. NNI is the port facing the VCX usually the network side Configuring the TWAMP Reflector (vcpe2) 1) Repeat the same steps as above for Location Z. 2) Once the configuration is done, both interfaces appear in the system/interface tab SCTE 44

45 3) The SOAM/TWAMP tab now shows the following information. Both units are ready to reflect TWAMP messages on UDP port ) To send the TWAMP message, the probe must first be configured. To do so, go to the SOAM/TWAMP/Generator/Configuration screen, and click Add SCTE 45

46 5) To see if the configuration is working, go to the Status tab in the TWAMP section (see image below). I indicates inactive, A indicates active. The same view shows alarms. For example, an active alarm may show a monitoring KPI above threshold. 6) Example: A TAD alarm states that the Two Way Delay is above the threshold. If the CC Continuity Check is inactive, the probe is working. 7) Results can be further analyzed by clicking on the Index to get more detailed statistics SCTE 46

47 8) Full results show all KPIs together, so that various other metrics can be used to analyze the root cause of a particular alarm: 2015 SCTE 47