Operational Practices

Operational Practices Deploying, Monitoring & Troubleshooting Performance Assured Business Services over DOCSIS An Operational Practice Prepared for the Society of Cable Telecommunications Engineers By Germain Levesque, Greg Spear, Scott Sumner (glevesque@accedian.com, gspear@accedian.com, ssumner@accedian.com) Accedian Networks, Inc. 2351 Blvd Alfred-Nobel, Suite N-410 Saint-Laurent (Montreal), Quebec, H4S 2A9 Canada 2015 SCTE

Title Table of Contents Page Number Introduction 4 1. Executive Summary 4 2. Scope 4 3. Background 5 4. BSoD Lifecycle Overview 5 5. NFV-Based vcpe System Architecture 6 5.1. Architecture Overview 6 5.2. Performance Assurance Functions 9 5.3. Operations Integration Considerations 10 5.3.1. Low-Touch vcpe Module Provisioning 10 5.4. vcpe Management Communication Options 11 VNF Controller & vcpe Deployment & Setup 12 1. Controller Provisioning Requirements 12 1.1. Supported Hypervisors 12 1.2. NFVI Resource Requirements 13 1.3. Required Equipment 13 2. NFVI Preparation 13 3. Controller Provisioning Procedure 14 3.1. Initial Preparation 14 3.2. Confirming Installation & Setting Management Parameters 17 3.3. Troubleshooting 18 4. Establishing vcpe Connectivity 18 4.1. Required Equipment 18 4.2. vcpe Discovery & Configuration 18 4.3. Configuring the vcpe Modules for Use 22 4.4. Confirming Installation 24 4.5. Troubleshooting 25 BSoD Service Deployment 26 5. Key Performance Attributes 26 6. Class of Service Model 27 7. Required Equipment 27 8. Calibration and Equipment Preparation 27 9. Detailed Procedure 28 9.1. Initial Preparation 28 9.2. SAT Test on EVC 28 9.3. SAT Test per CoS 34 10. Recording of Results 38 11. Analysis of Results and Examples 40 12. Troubleshooting 41 BSoD Performance Monitoring 42 1. Required Equipment 42 2. Calibration and Equipment Preparation 42 3. Detailed Procedure 43 3.1. Configuring the TWAMP Generator (vcpe1) 43 3.2. Configuring the TWAMP Reflector (vcpe2) 44 3.3. Analysis of Results 48 4. Troubleshooting 48 BSoD Troubleshooting Considerations 48 1. Summary of Per-Metric Performance Objectives 48 2015 SCTE 2

2. Sampling Rate 49 Conclusions and Recommendations 50 3. Conclusions 50 4. Areas for Further Investigation or to be Added in Future Versions 51 Abbreviations 52 Bibliography & References 53 Annexes 54 Annex A. SAT Report EVC Test for 64B frames 54 Annex B. SAT Report Per-CoS Test 55 Title List of Figures Page Number FIGURE 1 - METRO ETHERNET FORUM SERVICE LIFECYCLE. 6 FIGURE 2 - VCPE: TRADITIONAL VS. VIRTUALIZED CUSTOMER PREMISES EQUIPMENT EXAMPLE7 FIGURE 3 - VIRTUALIZATION OF NID ARCHITECTURE USING NFV 8 FIGURE 4 - NFV-BASED VCPE CONTROL TUNNEL & FUNCTION LOCATIONS 8 FIGURE 5 - TYPICAL BSOD PERFORMANCE ASSURANCE FUNCTIONALITY & IMPLEMENTATION 9 FIGURE 6 - INSTALL-TRIGGERED VCPE MODULE PROVISIONING USING DHCP OPTION 60 & 43 WITH FQDN NAMING 11 FIGURE 7 - ILLUSTRATION OF DUAL MAC/IP VCPE MODULE ADDRESSING SCHEME 12 FIGURE 3 SERVICE ACTIVATION TESTING METHODOLOGY 26 FIGURE 4 POINT-TO-POINT TWO-WAY SAT DIAGRAM 29 Title List of Tables Page Number TABLE 1- COS MODEL (EXAMPLE) 27 TABLE 2 - CONFIGURATION/PERFORMANCE TEST RESULTS FORMAT 39 TABLE 3 - POLICING TEST RESULTS FORMAT 39 TABLE 4 - TYPICAL COS PERFORMANCE OBJECTIVES (CPOS) PER APPLICATION 49 TABLE 5 - STANDARD DEVIATION FOR VARIOUS REAL LOSS VALUES AND NUMBER OF SAMPLES 50 2015 SCTE 3

1. Executive Summary Introduction This document describes the operational practices recommended to deploy, monitor, and troubleshoot performance assured Business Services over DOCSIS (BSoD). Consistent with the performance Service Level Agreements (SLAs) that normally accompany such commercial services, the procedures, methods and typical values for Key Performance Indicators (KPIs) associated with Service Activation Testing (SAT), real-time Performance Monitoring (PM), SLA reporting, and in-service troubleshooting are outlined, covering the full BSoD service lifecycle. Tests conducted, and operational procedures outlined, employ a virtualized Customer Premises Equipment (vcpe) approach that uses Network Function Virtualization (NFV)-powered hardware modules to add Service Operations, Administration and Maintenance (SOAM), testing and performance monitoring functions lacking in cable modems, to the customer site. A typical NFV-based vcpe module and controller architecture is presented as an overview to set the context for the operational procedures that employ it. Performance assured BSoD can be efficiently deployed and maintained when consistent instrumentation is applied from initial deployment through monitoring and troubleshooting. Standards commonly employed in Ethernet service deployment and maintenance are used, including Ethernet Service OAM (IEEE 802.3ah), Layer 3 TWAMP monitoring (RFC-5357), and RFC-2544 turn-up testing. Operational procedures show how to validate each critical service parameter at each stage, and how to interpret the results in the context of troubleshooting and maintaining optimal service quality and to meet SLAs, complying with the service lifecycle and service definition requirements formalized in Metro Ethernet Forum (MEF) specified Carrier Ethernet standards and guidelines. Areas of future work are explored, including the development of operational guidelines to govern the installation, provisioning, management, and automation of NFV-based vcpe solutions, as well as extension of monitoring methods to virtualized infrastructure to strengthen the hybrid-cloud connectivity over BSoD business case. 2. Scope This document is intended as a reference for operational, product management, engineering, and technical personnel directly involved in the definition, deployment, maintenance, optimization, and evolution of BSoD. Background on NFV-based vcpe architectures is provided as context for the detailed operational procedures that employ this method and architecture to deliver and assure the performance of BSoD services. Detailed procedures start with covering considerations and practical methods involved in setting up a virtualized control environment for the test infrastructure and connecting vcpe to the controller, and then guide the reader through detailed steps of service activation, performance monitoring and SLA reporting referencing the common KPIs that govern common enterprise-grade commerical services. A section dedicated to BSoD troubleshooting explores some general guidelines that can assist Multiple Systems Operator (MSO) operations personnel interpret metrics collected, and offers general guidelines as to what limits should be respected to ensure BSoD meet SLA commitments. 2015 SCTE 4

3. Background Commercial connectivity services command premium pricing, and cable MSOs have the infrastructure in position to take advantage of this segment. However, reliability and performance are more important than bandwidth, given that cloud computing, Software as a Service (SaaS) and data center connectivity are increasingly critical to operations. Over-the-top video, conferencing services and other bandwidthconsuming applications often share the same link. MSOs can grow BSoD revenue by offering assured Quality of Service (QoS) and SLAs committing to guaranteed uptime, bandwidth availability, and rapid Mean Time To Repair (MTTR). Such capabilities allow MSOs to offer sophisticated connectivity and managed services for important commercial segments including healthcare, education, hospitality, and financial services. DOCSIS 3.x cable modems do not offer integrated performance monitoring, service turn up testing, demarcation and other features required for efficient business services delivery. Also, the BSoD market is cost sensitive, eliminating the option to use Network Interface Devices (NIDs) that typically terminate and assure fibered enterprise connections. Complementing the capabilities of cable modems, vcpe strategies employing NFV can deliver full NID functionality using low cost, standards based vcpe modules, ensuring MSOs obtain the visibility and reporting capabilities required in the most operationally and cost efficient manner. This paper aims to provide concrete methods and procedures MSOs can employ to leverage these NFVbased vcpe strategies to deploy, monitor and maintain BSoD services alongside their existing operational infrastructure. 4. BSoD Lifecycle Overview Specific operational practices pertain to each phase of the BSoD service lifecycle: 1. Deployment: Provisioning and service activation testing 2. Performance Monitoring & SLA Reporting: collecting and presenting key performance metrics 3. Troubleshooting: Techniques to identify, isolate and troubleshoot service issues These three phases are consistent with the MEF definition of the Carrier Ethernet service lifecycle, which serves as an established model for commercial connectivity. 2015 SCTE 5

Figure 1 - Metro Ethernet Forum service lifecycle. 1 This paper is structured to address the operational practices associated with each stage of this service lifecycle, as it applies to BSoD. 5. NFV-Based vcpe System Architecture The operational practices and procedures described in ths paper are implemented using a networkembedded architecture that employs small footprint, programmable performance assuranace hardware modules (vcpe modules) augmented by virtualized performance assurance functions hosted on a centralized controller(s). Section 5 here introduces this architecture, as well as operational considerations that facilitate integration of this approach with existing Operational Support Systems (OSS) and Network Management Systems (NMS). 5.1. Architecture Overview In the context of this document, the term vcpe will refer to the strategy of virtualizing as many customerlocated networking functions as possible, while retaining the minimum hardware necessary for service delivery, consistent with performance, reliability, and Quality of Experience (QoE) expectations. An example of a vcpe strategy where onsite hardware appliances performing firewall, PBX, and routing funcitons have been virtualized is illustrated in Figure 2, below. Virtualization is accomplished by transferring local networking functionality to software-based, Virtual Network Functions (VNFs) which can be hosted on low-cost, Commericial Off-The-Shelf (COTS) servers or cloud-infrastructure. 1 MEF 39: Service OAM Performance Monitoring YANG Module Technical Specification 2015 SCTE 6

Figure 2 - vcpe: Traditional vs. Virtualized Customer Premises Equipment Example In the context of BSoD, this approach can be used to introduce customer premises-located performance monitoring, turn-up test, SOAM and troubleshooting functionality which in the case of fiber business services is normally provided using a NID. As BSoD is a more cost-sensitive service than full, fibered enterprise connections, a NID often costing several times more than a cable modem is not normaly a feasible CPE option. NFV-powered hardware modules can offer the same level of performance monitoring precision, loopback and full line-rate turn-up test capabilities at a fraction of the cost of a NID, making this approach an economically viable fit when deploying SLA-grade BSoD. In addiiton to supplementing the cable modem with performance assurance features, this approach has a number of other benefits: 1. Truck-rolls are reduced over the service lifecycle when compared to handheld test sets, as a single vcpe module can remotely perform turn-up testing, continuous monitoring, and on-demand troubleshooting. 2. Compatibility with existing hand-held Ethernet test sets and third-party centralized monitoring probes allows straightforward integration into existing operational practices and infrastructure. 3. By employing NFV, new functionality can be added to the vcpe module remotely, without impacting the service. This allows MSOs to introduce new, performance assured commercial services (e.g. burstable or on-demand Ethernet connections) without requiring new equipment onsite. An example of how NID functionality can be virtualized using NFV is shown in Figure 3. 2015 SCTE 7

Figure 3 - Virtualization of NID Architecture Using NFV The connection between the performance assurance VNF controller and each module needs to be reliable, secure and lossless (e.g. TCP-based) to ensure that the vcpe module can assume the same level of functionality as a NID. This management tunnel is critical to support performance assurnace VNFs, as raw data is returned to the controller for test results calculation, performance montoring, and fault reporting, in addition to performance monitoring session control, module management, synchronization information, etc. In an NFV-based vcpe architecture, the lossless control sessions allow each remote module to virtually become a remote port of the controller, which is analagous to a virtualized NID that can support many remote endpoints. Figure 4 - NFV-based vcpe Control Tunnel & Function Locations Some vcpe architectures virtualize customer-premises functions with a simple COTS server at the customer site. This is impractical in BSoD deployments because: 1. The cost of a COTS server is significantly higher than compact vcpe modules. 2. Performance assurance functions implemented purely in software lack sufficient time stamping precision and packet transmission scheduling control to meet the requirements of: a. Full line-rate test traffic generation and loopback for SAT and troubleshooting. b. Precise traffic generation sequencing required by common turn-up test standards (where inter-packet delay needs to be controlled for burst testing, for example). c. Microsecond-level latency measurement precision required to monitor and report on commerical services SLAs. 2015 SCTE 8

Aside from incorporating these capabilties, vcpe modules also offer a number of other advantages over traditional test set and centralized probe solutions: 1. Modules can monitor and test between themselves: for site-to-site monitoring, end-to-end turn-up testing and troubleshooting between customer service endpoints. Most probe-based solutions are limited to loopbacks or monitoring tests from a central location to a service endpoint. This huband-spoke topology does not test the actual service path between customer locations, or between a customer and a remotely hosted data center, for example. 2. Test sets require trained technician dispatch to each service endpoint requiring service activation testing or troubleshooting, which is much less responsive and much more costly than a remotely initiated test using vcpe modules that are initially installed during service provisioning. 5.2. Performance Assurance Functions NFV-based vcpe solutions must be capable of all performance assurance functions required to support the business services lifecycle, as described in Section 4. These include, but are not limited to: 1. Standards-based service activation testing supporting commonly employed IEEE RFC-2544 and ITU-T Y.1564 turn-up testing approaches. 2. Ethernet Connectivity Fault Management (CFM), as defined by IEEE 802.3ag to ensure service availability meets SLA definitions, and to measure continuity and latency using CCM and DMM/DMR messages, respectively. 3. Standards-based performance monitoring for Layer 2 (Ethernet) and Layer 3 (IP) services, typically implemented using ITU-T Y.1731 / IEEE 802.3ah Ethernet SOAM and RFC-5357 Two- Way Active Measurement Protocol (TWAMP), respectively. 4. Bandwidth utilization monitoring, per port and per service flow (as defined by the MSO: VLAN, CoS, source or destination MAC or IP address, etc.) for usage-based billing, trending, and troubleshooting. Figure 5 - Typical BSoD Performance Assurance Functionality & Implementation 2015 SCTE 9

5.3. Operations Integration Considerations Although outside the scope of this paper to cover in depth, implementation of an NFV-based vcpe solution should interwork with existing operational support systems to permit integration with existing management practicies and procedures, and to make deployment of vcpe modules as well as the monitoring and maintenance of the services they support as operationally efficient as possible. Main areas to consider include: Deployment and management of the solution itself, to facilitate and automate element management of remote vcpe modules and BSoD service provisioning. Integration with SLA reporting platforms and fault management systems to harmonize monitoring, and reporting and within existing tools. As introduction and general guidelines, the following opeational aspects should be considered during solution selection and deployment. 5.3.1. Low-Touch vcpe Module Provisioning Ideally, vcpe modules should be ready to install in factory default configuration, without requiring prestaging by the MSO. To make that possible, the modules must be discoverable by an inventory system that can attribute the module to a particular customer site. This may be accomplished by relating the module to the MAC or IP address of the customer s cable modem, for example. To come under management control without requiring pre-staging or on-site configuration by a trained technician, the units require a method to discover the mangement environment, have their managmeent IP address defined, have the desired configuration provisioned on the unit, etc. Auto-Provisioning Example using DHCP Option 60/43 & FQDN One commonly employed method to bring devices under managmeent involves using Dynamic Host Configuration Protocol (DHCP) with Options 60 & 43: 1. When a vcpe module is connected to the network, it announces itself using a DHCP request with option 60 enabled, which communicates the unit s identifier (device type) to the DHCP server. 2. When properly configured, the server assigns a temporary (dynamic) IP address to the unit, and responds with Option 43, providing the address of the inventory node responsible for managing the unit. 3. Once under management control, a static IP can be assigned (if desired). A Fully Qualified Domain Name (FQDN) can also be assigned to the module. The FQDN must remain in sync with any link-state change (per RFC-2131), typically realized using automated DNS queries by the module inventory node. Supporting FQDN allows the vcpe module to then be integrated into the cable modem inventory management system, alongside the cable modem supporting the BSoD service. 2015 SCTE 10

Figure 6 - Install-Triggered vcpe Module Provisioning Using DHCP Once the unit is under management control, automation may be used to trigger an immediate or scheduled turn-up test to validate the service, then provision customer/sla-specific monitoring sessions, etc. to allow customer-level self-install. 5.4. vcpe Management Communication Options The vcpe module should be capable of adapting to an MSO s particular management and device addressing methodology. This requires the unit to distinguish customer/test traffic from management communication. A variety of methods are commonly used, each implying the vcpe module must offer support for each of these schemes: Layer 2 addressing: separate management and customer MAC addresses. In this case, the vcpe module must support two MAC addresses, one for management traffic, and another for customer, test, CFM, SOAM and active performance monitoring traffic. Layer 2 addressing: separate management and customer VLANs. A single MAC address is used in combination with Q-in-Q VLAN support (C/S tagging, IEEE 802.1ad). Layer 3 addressing: similar to the Layer 2 scheme described above; separate management and customer traffic IP addresses may need to be supported by the vcpe module, depending on the method used by the operator. VLAN support, including Q-in-Q (S-VLAN), may also be required to support this scenario. Layer 3, transparent IP addressing: an operator may elect not to assign new IP address(s) to the vcpe module, instead using the address of a device located behind the module. This is practical when the operator has a known device at the customer premises (e.g. a set-top box, Wi-Fi controller, security gateway, etc.) In this case the vcpe module detects management traffic using 2015 SCTE 11

a combination of this other device s IP address in combination with other identifiers (such as a management VLAN tag). Figure 7 - Illustration of Dual MAC/IP vcpe Module Addressing Scheme VNF Controller & vcpe Deployment & Setup The NFV-based vcpe used as a reference configuration for procedures outlined in this paper are managed by a performance assurance controller, embodied as a virtual appliance. This section will show how to install the VNF controller (VCX) on a VMware ESXi hypervisor to the point where it is ready to login to and use for the procedures in subsequent sections of this paper, as well as pointing out common management configuration requirements. This is followed by a section describing the method to establish connectivity to the two NFV-based vcpe modules that will be used to conduct tests outlined in the remainder of this paper. 1. Controller Provisioning Requirements The following hardware, operating system and hypervisor guidelines outline the requirements to host the performance assurance VNF controller (specifically, in the procedures outlined in further sections of this paper, the Accedian SkyLIGHT VCX Controller, release 1.2): 1.1. Supported Hypervisors The VCX Controller can be installed on VMWare ESXi v5.5 or higher, or KVM Cent OS 6. VMWare will be used for subsequent operational procedures, as well as those outlined in this section. 2015 SCTE 12

1.2. NFVI Resource Requirements The recommended system resources for the VCX Controller are: CPU: 4 single cores minimum up to 8 quad cores depending on the capabilities and resources. HDD: Minimum of 2 Gb dynamic with thin provisioning. RAM: Minimum of 1 Gb, recommended 4 Gb. Network Ports: Minimum of one (1) for network access, but shall be a minimum of two (2) in practice to allow management and network accesses. If possible, all ports shall be dedicated NICs accessible to the Virtual Machine (VM) instance to allow load distribution. 1.3. Required Equipment X86-based PC or server running an operating system that supports the VMware ESXi 5.5 hypervisor 2, as specified on the VMware compatibility page: vmware.com/resources/compatibility VMware ESXi install files (https://my.vmware.com/web/vmware/evalcenter?p=free-esxi6) vsphere Virtual Machine management application (https://my.vmware.com/web/vmware/evalcenter?p=free-esxi6) Licensing required to support the scale of the particular MSO BSoD implementation (https://www.vmware.com/support/support-resources/licensing/) VCX Controller OVA virtual appliance image file, example: VCX_1.1_3149_VMWare.ova 2. NFVI Preparation The NFV Infrastructure (or NFVI) environment must be prepared before installing the controller virtual appliance (application running as a VM instance) that will be used. This example uses the VMware ESXi hypervisor version 5.5 to run the VCX application. This can be set up on any server that supports VMware. In this example the server being configured is an HP DL160, certified and supported by VMware, with a 250 GB drive and 16 Meg of RAM. Procedure: 1) Power up the server and boot from an external CD or USB with the appropriate ESXi version. 2) Follow the on-screen installation instructions. 3) Once completed, the ESXi hypervisor is now ready to support the controller virtual appliance application as a VM. 4) The vsphere application is used to simplify management of virtual machines running on on the ESXi hypervisor. Install the vsphere application using the instructions provided by VMware. 5) The vsphere application can also be used to set network parameters, including the IP address, of the virtual appliance. 2 Alternatively, an X86-based PC or server running an operating system that supports the KVM Cent OS 6. As stated previously, for this example we use VMware. 2015 SCTE 13

Figure 8 - vsphere Interface: Network Parameter Configuration Screen 3. Controller Provisioning Procedure 3.1. Initial Preparation 1) The vsphere application will be used to load and start the VCX Conroller virtual appliance (the.ova file). Select file deploy OVF template and follow the instructions as prompted during install. 2015 SCTE 14

2) Right-click on the VCX tab and select Power On. The VCX appliance entry provisioned have a green triangle icon next to it, indicating it is running. 2015 SCTE 15

3) To configure the VCX Controller, open a console to the VCX by right-clicking the entry and selecting Open Console. 4) Set the IP address of the application using the Command Line Interface (CLI). The screenshot below is an example of commands used to disable DHCP and set the IP address. Refer to the VCX Controller User Manual for additional information. 2015 SCTE 16

5) Now the VCX Contoller can be accessed by using the assinged IP address in a web brower: 3.2. Confirming Installation & Setting Management Parameters The VM and the VCX application are now running. To verify the installation is successful, proceed with the following steps: Log out of the VM and exit the vsphere application. 2015 SCTE 17

Exit and log back into the VCX using the address configured in the previous steps. The VCX Controller Graphical User Interface (GUI) can now be used to configure all the usual required fields typically required for management: Discovery options Adding remote devices SNMP traps Remote Syslog Timing: NTP 1588 or manually configured Adding users Configuring routing requirements (default route, gateways etc ) Please consult the VCX Controller User Manual for complete details on how to configure each of these management parameters consistent with the target operating environment. 3.3. Troubleshooting If you unable to connect to the VCX: Verify you have a valid IP address. Verify the link status on both devices. Check if a firewall is preventing or blocking the connection. Verify there is not a duplex mismatch (one port at half duplex and the other full duplex). Check for physical errors on the network. Ensure you are not using a duplicate IP address. 4. Establishing vcpe Connectivity 4.1. Required Equipment Functioning VCX Controller as specified above. Two compatible NFV-based vcpe modules: Accedian antmodules (model ANT-1000-TX, part number 772-000). 4.2. vcpe Discovery & Configuration The remote vcpe modules will be automatically discovered by the VCX Controller in a Layer 2 or Layer 3 network. The vcpe modules can get their addresses automatically via DHCP, or can be staged with a predefined address. For this paper, simple Layer 2 discovery will be used. For a more complete overview of the discovery options, please refer to the user guide. The following diagrams show how the discovery process works. No pre-configuration of the vcpe modules is required. 2015 SCTE 18

Figure 9 - Layer 2 Discovery Method as Employed by the VCX Controller 1) Go to the Discovery/configuration tab. Then, select Add. 2) Select the interface that is connected to the network. In this example we are using the Management interface. Any other configured interfaces will be available from the pull down menu: 2015 SCTE 19

3) Once the discovery is setup, switch to the Inventory tab to see if the unit was discovered. 4) In this example we see ANN-1000-CX was identified and added to the VCX Controller inventory. 5) To start managing the units, select the unit using the box on the left and click Add: 2015 SCTE 20

6) Once under management, the unit will be available in the Remote Devices tab. 7) The unit will initialize and update if required. The update could take a few minutes. 8) Once the unit is linked and authorized you can click on the unit and configure a unique name and click Apply. 9) This is the view in Remote Devices once the name is configured and the updating is complete: 10) Verify the module s ports are activated using Port tab. 2015 SCTE 21

4.3. Configuring the vcpe Modules for Use Depending on the test to be performed, other configuration steps are required. For example, a simple traffic generator test requires setting up the traffic generator and pointing it to the MAC address of the destination/remote vcpe module reflecting the traffic. This example will test from the MyNanoNID vcpe module to the central reflector vcpe module located at a head-end location: The traffic generator will be set up in the MyNanoNID vcpe module. 1) Go to the SAT tab, configure as needed, and then click Add. 2) Below is the configuration to generate one test flow at 50 Mbps for 30 seconds. Once parameters are entered, click Apply to save the configuration. 2015 SCTE 22

3) To run the test go to the Result tab, and select Detailed Report: 2015 SCTE 23

This simple traffic generation test is a simple way to quickly validate that the vcpe module is under management and is in good working order. 4.4. Confirming Installation To this point we have confirmed the vcpe module was discovered, is under management control, and is functioning properly as shown using a simple traffic generator test. To complete confirmation that the vcpe module has the correct firmware and configuration required for tests and procedures outlined in this paper, follow the steps below. 1) Visit the Configuration tab and verify the unit has both PMON (performance monitoring) and TGEN features available, and select the Active Firmware for each as shown in the series of screen captures below: 2015 SCTE 24

2) Verify that the values in the pull-down menu include PMON and TGEN. 3) The vcpe modules are now ready to use for the procedures outlined in the following sections of this paper. 4.5. Troubleshooting If you are unable to discover the remote unit the most common cause is issues with network connectivity the path between the VCX and the vcpe module must be open and reliable. Is the remote device powered up and is the port active? Verify the LED on the vcpe module. If the unit is discovered but you are not able to manage it, it may already be managed by another VCX. If a unit is discovered and managed but will not perform a test it may be in the wrong mode. Set the module Out Of Service (OOS) setting to In Service in the Remote Devices screen and try again. 2015 SCTE 25

BSoD Service Deployment SAT is a key part of the Carrier Ethernet service operations life cycle. Before a service provider deploys an Ethernet service to a customer, it must be validated to ensure the participating network devices have been configured properly and the service meets the defined SLA, or as commonly referred to by the MEF, the Service Level Specification (SLS). This testing also provides baseline service measurements to create a Carrier Ethernet service birth certificate as a benchmark for troubleshooting, and to report a successful service turn-up to the customer. The service activation testing methodology, as illustrated in Figure 10, addresses these requirements. Figure 10 Service Activation Testing Methodology 3 5. Key Performance Attributes The following key performance attributes will be measured in a Service Activation Test: Frame Transfer Delay (FTD) The time required to transmit a Service Frame from ingress Service Activation Measurement Point (SAMP) to egress SAMP. Frame Delay Variation (FDV) The absolute value of the difference between the FTD of two consecutive Service Frames belonging to the same Class of Service (CoS) stream. Frame Loss Ratio (FLR) 3 Source: MEF CE 2.0 Service Management Life Cycle White Paper 2015 SCTE 26

A characterization of the number of lost frames between the ingress SAMP and the egress SAMP for Service Frames that have the same CoS. The Frame Loss Ratio Performance is expressed as a percentage. 6. Class of Service Model The choice of a specific CoS model belongs to the operator. The MEF 23.1 standard specifies a set of three CoS Names called Labels that can be used by operators to indicate the performance expectations to be associated with a given set of frames that comprise a CoS Frame Set. CoS Labels are H, M and L, which informally refer to High, Medium, and Low. The order of the CoS Labels is based on the traffic classes and their associated PCP values. Operators chose how specific applications are mapped to these CoS labels. They can also define how the priority (PCP or DSCP) will be mapped at a User Network Interface (UNI) for multi-cos Ethernet Virtual Connections (EVCs) that support all 3 MEF CoS Labels. Finally, the CoS Performance Objectives (CPO) is defined for each Performance metric and per CoS Label. Defining a CoS model for specific applications is beyond the scope of this paper. Operators should refer to MEF 23.1 (see References section) for more information. The following CoS model will be used as an example in this paper to illustrate the principles of SAT. PCP mapping and Committed Information Rate (CIR) values are provided. Table 1- CoS Model (Example) CoS Label PCP CIR H 5 2 Mbps M 3 3 Mbps L 1 5 Mbps 7. Required Equipment One Y.1564 generator unit. One Y.1564 reflector unit In this procedure, both units are Accedian antmodule vcpe units, under the control of the SkyLIGHT VCX Controller. 8. Calibration and Equipment Preparation The Y.1564 generator shall be configured to compute rates at Layer 2. To do so, proceed as follows: 1) Log in to the GT Performance Element. 2) Access the page Traffic Configuration. 3) Set the Generator working rate to Layer 2 (as shown in the following picture) and click Apply. 2015 SCTE 27

9. Detailed Procedure 9.1. Initial Preparation In the context of SAT, the performance measurements may be performed one-way or two-way (roundtrip). One-way measurements require time synchronization at each Service Activation Measurement Point. For the sake of simplicity, two-way SAT will be used in the following procedures. The Y.1564 methodology shall be preferred over RFC-2544 because of its ability to test multiple Classes of Service simultaneously. An operator may also want perform a SAT test on a circuit or EVC before configuring the various Classes of Service to make sure it conforms with the overall Service Level Specifications (SLS). This initial SAT test can help identify basic issues and troubleshoot them before performing more detailed per-cos configuration and SAT testing. Following the initial EVC SAT test, more sophisticated tests shall be performed par CoS to validate the performance objective for each service. 9.2. SAT Test on EVC The purpose of this test is to validate the performance objectives of the circuit/evc. No CoS shall be configured at this point. The test shall be run for various frame sizes, as required by the Operator. In this example, we will test the EVC at the CIR rate sequentially for 64B, 128B, 256B, 512B, 1024B, 1518B frames. 2015 SCTE 28

Figure 11 Point-to-Point Two-way SAT Diagram 1) Set up the Y.1564 generator at the measurement point where the test traffic will be generated. 2) Set up the Y.1564 reflector (loopback) unit at the far-end measurement point. 3) Configure the SAT test as follows: a) Get the far-end vcpe MAC address Log in to the SkyLIGHT VCX Controller Access the page Remote Devices Configuration. This displays the screen shown in the figure below. Take note of the MAC address for vcpe module (2) that will be used as a reflector at the far end. This information will be required to configure the Y.1564 test generator. b) Set up the Y.1564 Test on the far-end vcpe module Log in to the VCX Controller and access the page SAT Y.1564 Configuration. Click the Add button to create a new test profile. Enter the test configuration as shown in the following figure. In the MAC Destination field, type the MAC address of the vcpe module (2) that will be used as a reflector. Click Apply to save the test configuration. 2015 SCTE 29

Explanations: The Configuration Test is enabled and configured to run for 10 seconds. This test validates each defined service to make sure that the configuration is correct. The Performance Test is enabled and configured to run for 15 minutes. This test validates the quality of the services over a medium to long time duration. The specific duration is to be determined by the Operator according to their operational pratices. The Layer-2 test frame type is selected for this test. With this configuration, the generator sends an LBM frame to the reflector device (MAC 00:15:AD:1B:57:60). The reflector swaps the MAC destination and source addresses and returns an LBR frame. c) Set up the Y.1564 Services Go to SAT Y.1564 Configuration. This displays a summary of all test profiles. Click the Name of the test (EVC Test) to edit its settings. Click the Name of a service (e.g. Service_1) from the Service List to edit its settings. This displays the screen shown in the figure below. 2015 SCTE 30

Enter the service configuration parameters for the 64B frame test flow as shown in the figure above and then click Apply. Explanations: The CIR is set to 10 Mbps in order to measure the overall throughput. The Configuration Test is enabled. This will add the following rates to the configuration test: 25% CIR, 50% CIR and 75% CIR. The acceptance criteria shall be configured as per the Service Level Specifications. 2015 SCTE 31

Repeat the service configuretion steps as described above for each other frame size that needs to be tested, namely for 128B, 256B, 512B, 1024B and 1518B. Do not enable the state of these services for now. The following figure shows the six services (frames sizes) once they are configured: Six services (frames sizes) have been configured in the test profile. The test shall be run for the first service (64B frames), then for the second, and so on until all frame sizes have been tested. d) Run the Y.1564 Test Go to the page SAT Y.1564 Results. This displays a summary of all test reports. Click the Start new test button to start the test and generate a report for the first service (64B). This displays the screen shown in the figure below. 2015 SCTE 32

Enter the a test report description and any other optional information as needed and and click Run. e) View a Summary of the Test Results Go to the page SAT Y.1564 Results. This displays a summary of all test reports. To view detailed results from a test, click the name of the test report. While the test is running, this displays the screen shown in the figure below. To abort a test while it is running, click Stop. When the test is completed, the following screen will be displayed: 2015 SCTE 33

To export the detailed report into a text file and save it on the management station, click Export. See Appendix A to view the completed report from this example. Go back to the page SAT Y.1564 Configuration and click the name of the test (EVC Test) to edit its settings. To test the EVC with 128B frames, disable the first service (Frame Size 64B) and enable the second one (Frame Size 128B). Run the test again and save the report as described previously. Repeat these steps for each frame size. 9.3. SAT Test Per CoS The purpose of this test is to validate the performance objectives per CoS. All CoS shall be configured in the network at this point in order to be tested simultaneously. Configure the SAT test as follows: a) Set up the Y.1564 Test on the near-end vcpe (1) module Log in to the SkyLIGHT VCX. Access the page SAT Y.1564 Configuration. Click the Add button to create a new test profile. Enter the test configuration as shown in the following figure. In the MAC Destination field, type the MAC address of the vcpe (2) that will be used as a reflector. Click Apply to save the test configuration. 2015 SCTE 34

Explanations: The Parallel Configuration Test is enabled in orther to validate all defined service simultaneously. From the Service List, three CoS shall be configured and enabled. This will allow running the Performance Test on all services simultaneously. b) Set up the Y.1564 Services Go to SAT Y.1564 Configuration. This displays a summary of all test profiles. Click the Name of the test (EVC Test) to edit its settings. Click the Name of a service (e.g. Service_1) from the Service List to edit its settings. This displays the screen shown in the figure below. 2015 SCTE 35

Explanations: The CIR and VLAN Priortity (PCP) are configured as defined in the CoS model. The Policing Test is enabled. The purpose of this test is to send traffic above the rate of CIR+EIR in order to verify that the bandwidth policer is operating correctly and discards frames that exceed CIR+EIR. The Frame Size type is set to EMIX, which allows testing with a pattern of variable frame sizes. The available sizes are: a=64b; b=128b; c=256b; d=512b; e=1024b; f=1280b; g=1518b; h=port MTU; u=user-defined size. 2015 SCTE 36

Repeat the service configuration steps as described above for other two CoS, with their respective CIR and PCP values: c) Run the Y.1564 Test Go to the page SAT Y.1564 Results. This displays a summary of all test reports. Click the Start new test button to start the test and generate a report for the first service (64B). This displays the screen shown in the figure below. 2015 SCTE 37

Enter the a test report description and any other optional information as needed and and click Run. d) View a Summary of the Test Results Go to the page SAT Y.1564 Results. This displays a summary of all test reports. To view detailed results from a test, click the name of the test report. While the test is running, this displays the screen shown in the figure below. This time, we see that the three CoS are being tested simulteneously. When the test is completed, the following screen will be displayed: To export the detailed report into a text file and save it on the management station, click Export. See Appendix A to view the completed report from this example. 10. Recording of Results The SAT test produces a detailed report that can be exported to a local workstation or remote server in.txt or.xml format. The following tables show the format used in the report for each test. Examples of the complete test reports produced in the previous examples are available in Annex A and B. 2015 SCTE 38

Table 2 - Configuration/Performance Test Results Format IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 0.500 PASS 0.469 0.499 0.499 0 0.0e+00 5170 5259 5376 0 43 141 1 PASS 0.969 0.999 1 0 0.0e+00 5082 5227 5358 0 46 148 1.500 PASS 1.469 1.500 1.500 0 0.0e+00 5077 5215 5358 0 57 177 2 PASS 1.969 1.999 2 0 0.0e+00 5065 5188 5366 0 59 213 The first columns display the information rate (CIR/EIR) that is generated. The second column (Pass/Fail) indicates whether the test is a PASS or FAIL, based on the configured acceptance criteria. The next three columns, under IR, display the measured Information Rate values. The next two columns, under FL, display the measured frame loss. The Cnt field shows a count of the number of lost frames, if any, up to 99 9999. Higher counts would be marked >100k. FLR displays the measured frame loss ratio. The last six columns display the measured Frame Transfer Delay (FTD) and Frame Delay Variation (FDV) values. Table 3 - Policing Test Results Format IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) POLICING 2.500 PASS 1.951 2.006 2.082 932 1.8e-01 5053 5213 6915 0 64 1734 CIR*(1-FLRsac) <= IR <= CIR+EIR+M 2*(1-0.000001) <= 2.006 <= 2+0+1 All column headers are the same as in the Configuration and Performance tests results. The difference is that a successful Policing test is expected to show a certain FLR that is greater than 0 and even excessive FTD and FDV, since its purpose is to validate that the bandwidth policer is doing its job when we attempt to exceed the allowed bandwidth. 2015 SCTE 39

11. Analysis of Results and Examples Annex A and B show examples of successful SAT tests. These tests used the following Service Acceptance Criteria: FTD FTD type FDV FDV type FLR M factor : 10000 us : max : 2500 us : max : 1.00e-06 : 1 Mbps The following test report excerpts show a few examples of failed tests and their cause: Failed test due to excessive Frame Loss CONFIGURATION Started at : 2015-06-26 14:47:42-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 2.500 FAIL 1.993 2.006 2.083 7342 2.0e-01 38 125 229 0 43 183 Failed test due to excessive Frame Transfer Delay CONFIGURATION Started at : 2015-06-26 15:13:00-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 2.500 FAIL 1.997 2.008 2.086 7320 2.0e-01 11082 11184 11267 0 23 185 Failed test due to excessive Frame Delay Variation CONFIGURATION Started at : 2015-06-26 17:28:35-04:00 IR FL FTD FDV 2015 SCTE 40

-------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 2.500 FAIL 2.489 2.499 2.506 0 0.0e+00 3571 6157 8767 0 1734 4990 Failed test due to inadequate bandwidth policing IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 1.250 PASS 1.218 1.249 1.250 0 0.0e+00 5123 5256 5390 0 46 157 2.500 PASS 2.469 2.499 2.500 0 0.0e+00 5066 5223 6104 0 62 903 3.750 PASS 3.718 3.749 3.750 0 0.0e+00 5049 5214 5484 0 77 290 5 PASS 4.967 4.999 5 0 0.0e+00 5062 5210 5389 0 73 266 CIR/EIR N/A POLICING 6.250 FAIL 6.216 6.249 6.250 0 0.0e+00 5052 5199 6252 0 65 1086 CIR*(1-FLRsac) <= IR <= CIR+EIR+M 5*(1-0.000001) <= 6.249 <= 5+0+1 12. Troubleshooting This section describes a few typical setup or configuration issues that can cause a SAT test to fail, and how to troubleshoot them. Error Message: Test Overall Result: PEER NOT FOUND This error is usually encountered when the peer unit (reflector) fails to return any test frames. 1. Verify that all physical connections are made properly end to end. 2. Verify the port status at both Service Activation Measurement Points. All ports must be up. 3. Verify the SAT test configuration and make sure that the MAC Destination address is the reflector vcpe s MAC address. 4. Make sure that the Y.1564 generator is configured with the proper type of test frame (Layer 2). 2015 SCTE 41

The test fails for specific frame sizes 1. Verify the MTU on all devices, including any network element that may be installed between the SAT generator and the reflector vcpe modules. The MTU must large enough to support the largest configured test frame. 2. Verify if any network element adds a VLAN tag anywhere on the path that is being tested. A VLAN tag adds 4 bytes to a frame and may cause the MTU to be exceeded. BSoD Performance Monitoring Once a service is validating using the SAT tests described in the previous section, real-time performance monitoring allows the operator to measure and report SLA metrics, and monitor KPIs for troubleshooting, proactive fault mitigation, trending, and many other important aspects of maintaining a performanceassured service that meets customer expectations and contractual obligations. In this example the previously configured units will be used to demonstrate performance monitoring configuration steps, reviewing results, and explore some common troubleshooting steps. The most common method of measuring performance in the BSoD model is TWAMP, Two Way Active Measurement Protocol (RFC-5357), for Layer 3 monitoring. This will be the method used for this example. Other standard methods are also available -- Y.1731 for example -- and will provide similar results for Layer 2 measurements. The operator will normally select their preferred method based on the type of network: Layer 2 or Layer 3, and the supported standards in the equipment deployed in the network. These standard employ active test sessions by sending test traffic between two endpoints and monitoring the results, where thresholds will trigger an alarm if a value is exceeded. As an example, if packet loss (PL) is approaching an SLA limit, the operator can be notified by an alarm. This will allow the provider to fix any network issues before the SLA (Service Level Agreement) with the customer is violated, if the alarm thresholds are set accordingly. 1. Required Equipment One Y.1564 generator unit. In this procedure, we are using an Accedian GT Performance Element. One Y.1564 reflector unit. In this procedure, we are using an Accedian ant Performance Module and Skyligt VCX Controller. 2. Calibration and Equipment Preparation The equipment listed above was prepared in the SAT section, and is ready for PM monitoring configuration. No further preparation is required. 2015 SCTE 42

3. Detailed Procedure Figure 12 - TWAMP Test Setup Diagram TWAMP is a Layer 3 test protocol and requires an IP address at both ends. For this example, the generator will be assigned 10.1.1.1 and the reflector will be assigned 10.1.1.2. In a production network the addresses should be validated with a Ping test to make sure the network routing is in order before configuring the TWAMP probes. 3.1. Configuring the TWAMP Generator (vcpe1) 1) Login to the GUI of the VCX using the IP address assigned earlier. 2) We will assign IP address to the device, for this example the devices are called Location A (vcpe1) and Location Z (vcpe2), 3) The device names can be edited in the remote device tab. 4) The screen will look like the screenshot shown below. Click Add to configure the device. 2015 SCTE 43

5) The configuration will appear as shown below. After entering the necessary information,click Apply. In this example there is no gateway, but in a production network this will probably be required. 6) Two ports appear in the pull down menu: LocationX UNI and LocationX NNI. NNI is the port facing the VCX usually the network side. 3.2. Configuring the TWAMP Reflector (vcpe2) 1) Repeat the same steps as above for Location Z. 2) Once the configuration is done, both interfaces appear in the system/interface tab. 2015 SCTE 44

3) The SOAM/TWAMP tab now shows the following information. Both units are ready to reflect TWAMP messages on UDP port 6000. 4) To send the TWAMP message, the probe must first be configured. To do so, go to the SOAM/TWAMP/Generator/Configuration screen, and click Add. 2015 SCTE 45

5) To see if the configuration is working, go to the Status tab in the TWAMP section (see image below). I indicates inactive, A indicates active. The same view shows alarms. For example, an active alarm may show a monitoring KPI above threshold. 6) Example: A TAD alarm states that the Two Way Delay is above the threshold. If the CC Continuity Check is inactive, the probe is working. 7) Results can be further analyzed by clicking on the Index to get more detailed statistics. 2015 SCTE 46

8) Full results show all KPIs together, so that various other metrics can be used to analyze the root cause of a particular alarm: 2015 SCTE 47

3.3. Analysis of Results With the TWAMP session configured, data is now be collected continuously to monitor the service. These metrics can be integrated into common monitoring, network performance visualization and SLA reporting tools, in addition to directly accessing the data from the VCX GUI (appropriate for ad-hoc troubleshooting). As the methods that may be employed are diverse, and are consistent with the methods MSOs use to monitoring their transport networks, details are not provided here. Data import / integration can be achieved using a number of common methods, including SNMP GETS, CSV file push, and common transfer methods like SCP/FTP/etc. 4. Troubleshooting As in previous sections, network issues are the most common source of issues that may affect continuous performance monitoring. The detailed procedure above is self-confirming, in that if results are retrieved, the test has been set up properly. Should the tests stop working after initial setup, check the following as a first step: 1. Verify that all physical connections are made properly end to end. 2. Verify the port status at both endpoints are up (from the Ports screen). 3. If both endpoints are not up, verify that the IP addresses assigned to each endpoint are still valid, and are not conflicting with another device on the network that may have been assigned the same address. BSoD Troubleshooting Considerations Both service activation test methods and continuous performance monitoring can be used to troubleshoot BSoD services when there are performance issues or SLA violations. Detailed methods of interpreting each metric, how it applies to a particular problem, and how multiple KPIs can be correlated to determine the root cause of an issue is beyond the scope of this paper. However, the sections below reference common performance objectives BSoD offerings are commonly expected to meet, serving as a guideline to ensure services meet QoS expectations over all stages of the service lifecycle. 1. Summary of Per-Metric Performance Objectives Performance objectives are essentially driven by the type of applications associated with each CoS label. The requirements for application performance are usually specified from end to end by the operator or service provider, according to their own specifications. The success of SAT and PM depend on those performance targets. Therefore, it is important to carefully define the targets to ensure tests are properly configured and executed. The MEF 23.1 standard proposes reasonable performance objective values based on a wide variety of applications. The following table provides a summary of the CPOs for several applications. More detailed information can be found in the MEF 23.1 specification (see References section). 2015 SCTE 48

Table 4 - Typical CoS Performance Objectives (CPOs) Per Application Application FTD FDV FLR VoIP Data 125 ms pref. 40 ms 3e-2 375 ms limit Video Conferencing Data 125 ms pref. 40 ms 1e-2 375 ms limit VoIP and Video Conferencing Not specified Not specified 1e-3 Signaling IPTV Data Plane 125 ms 40 ms 1e-3 IPTV Control Plane Not specified Not specified 1e-3 Streaming Media Not specified 1.5 s 1e-2 Interactive Gaming 50 ms 8 ms 1e-3 Circuit Emulation 25 ms 10 ms 1e-6 Telepresence 120 ms 10 ms 2.5e-4 CCTV 150 ms (MPEG-4) Not specified 1e-2 200 ms (MJPEG) T.38 Fax 400 ms 40 ms 3e-2 Point of Sale Transactions 2 s Not specified 1e-3 Best Effort Not specified Not specified Not specified 2. Sampling Rate Synthetic loss measurement is a sampling technique for measuring frame loss. Various methodologies are based on this technique, such as ETH-SLM (Y.1731) and TWAMP (RFC-5357). When a sampling technique is used to measure frame loss, the measured FLR will be distributed around the actual loss value according to a binomial distribution. The mean measured FLR will always be equal to the actual FLR, while the standard deviation depends on the number of samples. The standard deviation can therefore be used to illustrate the accuracy of the measured FLR result. Table 5 below shows the standard deviation for various real loss values and numbers of samples (i.e., number of synthetic frames sent). The number of samples should be chosen such that the standard deviation is low, when compared to any FLR threshold that is being used to trigger an action. This ensures the chance of false positives is low. 2015 SCTE 49

Table 5 - Standard deviation for various real loss values and number of samples 4 Actual FLR Number of samples Transmission interval Standard Deviation (FLR % points) 50% 10 100 ms 15.81% 50% 100 10 ms 5.00% 50% 1000 1 ms 1.58% 10% 10 100 ms 9.49% 10% 100 10 ms 3.00% 10% 1000 1 ms 0.95% 1% 10 100 ms 3.15% 1% 100 10 ms 0.99% 1% 1000 1 ms 0.31% 0.1% 10 100 ms 1.00% 0.1% 100 10 ms 0.31% 0.1% 1000 1 ms 0.1% 3. Conclusions Conclusions and Recommendations Performance assured BSoD can be efficiently deployed and maintained when consistent instrumentation is applied from initial deployment through monitoring and troubleshooting. Standards commonly used in Ethernet service deployment and maintenance, including Ethernet Service OAM (IEEE 802.3ah), Layer 3 TWAMP monitoring (RFC-5357), and RFC-2544 turn-up testing can be used to assure performance over the full BSoD service lifecycle. Detailed KPIs permit complete service validation and detailed problem diagnosis. Guidelines provided here give operations personnel a method to guage the severity of an issue, then offer direction towards sucessful troubleshooting. The test standards and the methods shown here can be implemented by MSOs using a wide variety of instrumentation methods, including handheld test sets, cetnralized montoring probes, and network elements including swtiches and routers with support for these standards. This document instructs MSO personnel on implementing these proceudres with NFV-based vcpe modules and their related VNF controller, a method increasingly popular with MSOs seeking to deploy BSoD with the lowest operational and capital expense, while also aiming for the fastest detection and response time to common BSoD QoS issues. Using this approach, a single test architecture covers: 1. Deployment: provisioning and service activation testing 2. Performance Monitoring & SLA Reporting: collecting and presenting key performance metrics 3. Troubleshooting: techniques to identify, isolate and troubleshoot service issues In addiiton to supplementing the cable modem with performance assurance features, this approach has a number of other advantages compared to more traditional techniques: 4 Source: ITU-T G.8013/Y.1731, OAM functions and mechanisms for Ethernet-based networks 2015 SCTE 50

1. Truck-rolls and trouble-ticket response time are reduced over the service lifecycle with turn-up testing, continuous monitoring and on-demand troubleshooting controlled remotely. 2. Compatibility with existing hand-held Ethernet test sets and third -party centralized montoring probes allow straightforward integration into existing operational practices and infrastructure. 3. NFV-based functionality allows new service assurnace featueres to be deployed remotely to support future BSoD product feature additons without requiring new equipment on-site. 4. Modules can monitor and test between themselves: for site-to-site monitoring, end-to-end turn-up testing, and troubleshooting between customer service endpoints over the actual service path. As the BSoD business case is cost sensitive, NFV-based instrumentation methods provide the full set of SLA KPIs required to deliver these premium, differentiated services, while putting MSOs on a costefficient path to compete against telecom and ISP incumbents in this compelling marketplace. 4. Areas for Further Investigation or to be Added in Future Versions As MSOs scale out BSoD, there are numerous avenues where additional, related operational practices can be developed to further assist their efforts: 1. More detailed troubleshooting and root cause analysis, outlined in detailed operational flowcharts that cover all common deployment models focusing on key metrics and how they can be used to determine the best course of corrective action. 2. Service delivery optimization methods using collected metrics for trending, and real-time KPIs as dynamic input to network management systems (including SDN controllers). 3. Definition of procedures and standards revolving around rapid device provisioning and integration with existing cable modem inventory, control, fault management and security systems, consistent with DOCSIS 3.x and DOCSIS 4.x standards. 4. Extended BSoD connection methods for hybrid-cloud infrastructure used by enterprises, with focus on critical content delivery network (CDN) interconnects, peering points, and virtual infrastructure including virtualized servers themselves providing a complete view of assured performance across all service endpoints accessed by the customer. BSoD is the beginning of an evolution that transforms MSOs into rich, intelligent connectivity providers serving the shift to cloud compute. By first developing the right base of performance assured BSoD offerings as outlined in this document, MSOs will be able to contribute to more detailed procedures, and robust standards, positioning themselves as leading connectivity providers in this emerging market. 2015 SCTE 51

Abbreviations AP access point bps bits per second BSoD Business Services over DOCSIS CBS Committed Burst Size CCM Continuity Check Message CFM Connectivity Fault Management CIR Committed Information Rate CoS Class of Service COTS Commericial off-the-shelf CPE Customer Premises Equipment CPO CoS Performance Objectives C-VLAN Customer VLAN CE Carrier Ethernet DMM Delay Measurement Message DMR Delay Measurement Response DSCP Differentiated Services Code Point EBS Excess Burst Size EIR Excess Information Rate EMIX Ethernet Mix EVC Ethernet Virtual Connection FDV Frame Delay Variation FEC forward error correction FL Frame Loss FLR Frame Loss Ratio FTD Frame Transfer Delay Gb Gigabyte GUI Graphical User Interface HD high definition HFC hybrid fiber-coax Hz hertz IR Information Rate ITU-T International Telecommunication Union Telecommunication KPI Key Performance Indicator LBM Loopback Message LBR Loopback Reply M Factor Margin Factor MAC Media Access Control Mbps Megabit per second MEF Metro Ethernet Forum MSO Multiple Systems Operator MTU Maximum Transmission Unit 2015 SCTE 52

NID NMS NFV NFVI OAM OSS PCP QoS QoE SaaS SAT SCTE SLA SLS SOAM S-VLAN TWAMP vcpe VLAN VM VNF AP Network Interface Device Network Management System Network Function Virtualization NFV Infrastructure Operations, Administration and Maintenance Operational Support System Priority Code Point Quality of Service Quality of Experience Software-as-a-Service Service Activation Testing Society of Cable Telecommunications Engineers Service Level Agreement Service Level Specifications Service OAM (IEEE Y.1731) Service VLAN Two Way Active Measurement Protocol (ITU-T RFC-5357) Virtual CPE Virtual LAN Virtual Machine Virtual Network Function access point Bibliography & References CE 2.0 Service Management Life Cycle White Paper, July 2014; Metro Ethernet Forum G.8013/Y.1731: OAM functions and mechanisms for Ethernet-based networks, November 2013; ITU-T MEF 23.1: Carrier Ethernet Class of Service Phase 2, January 2012; Metro Ethernet Forum Y.1564: Ethernet Service Activation Test Methodology, March 2011; ITU-T RFC-5357: A Two-Way Active Measurement Protocol (TWAMP), October 2008, IETF 2015 SCTE 53

Annexes Annex A. SAT Report EVC Test for 64B frames Y.1564 Service Activation Test Test description: File name : new_2015-06-25_17h37m26s Description : EVC Test - 64B Technician name : Device description: Product name : AMN-1000-GT Unit identifier : G274-0010 Firmware version : AMT_7.0.0.1_2517 Serial number : G274-0010 Assembly : 500-032-01:19:22:00 Test settings: Definitions: Name : EVC Test Description : Validate the performance of the circuit/evc Configuration test : Enabled step time : 10 (seconds) parallel testing : Disabled Performance test : Enabled testing time : 15 (minutes) Delay Measurement Type : Two Way Service 1: Name : Frame Size 64B CIR Step Load Policing : enable : disable Bandwidth Profile (L2 Rate) CIR : 10 Mbps CBS : 8 KB EIR : 0 Mbps EBS : 8 KB Size type : Fix packet size size : 64 Service Acceptance Criteria FTD : 10000 us FTD type : max FDV : 2500 us FDV type : max FLR : 1.00e-06 M factor : 1 Mbps Device Informations: Testing layer : Layer-2 2015 SCTE 54

Peer MAC address : 00:15:AD:1B:57:60 NID MAC address : 00:15:AD:11:A5:FC Ethertype : 0x8902 Opcode : 3 MEG level : 7 Host VLAN Informations VLAN 1 Protocol : S-VLAN (0x88a8) Identifier : 1000 CFI/DEI : 0 Priority : 5 CONFIGURATION Started at : 2015-06-25 17:37:35-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 2.500 PASS 2.497 2.499 2.500 0 0.0e+00 5088 5165 5243 0 30 131 5 PASS 4.997 5 5 0 0.0e+00 5063 5132 5230 0 68 144 7.500 PASS 7.496 7.500 7.501 0 0.0e+00 5042 5128 5305 0 34 179 10 PASS 9.996 9.999 10 0 0.0e+00 5043 5123 5235 0 16 146 CIR/EIR N/A POLICING N/A Ended at : 2015-06-25 17:38:14-04:00 PERFORMANCE Started at : 2015-06-25 17:38:15-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) 10 PASS 9.995 9.999 10.002 0 0.0e+00 5041 5124 5263 0 17 157 Ended at : 2015-06-25 17:50:24-04:00 Service Overall Result: PASSED Test Overall Result: PASSED Ended at : 2015-06-25 17:50:24-04:00 Annex B. SAT Report Per-CoS Test Y.1564 Service Activation Test Test description: File name : new_2015-06-26_13h16m02s 2015 SCTE 55

Description : Per-CoS Validation Test Technician name : Device description: Product name : AMN-1000-GT Unit identifier : G274-0010 Firmware version : AMT_7.0.0.1_2517 Serial number : G274-0010 Assembly : 500-032-01:19:22:00 Test settings: Definitions: Name : SAT Test per CoS Description : Validate the performance per CoS Configuration test : Enabled step time : 10 (seconds) parallel testing : Enabled Performance test : Enabled testing time : 15 (minutes) Delay Measurement Type : Two Way Service 1: Name : CoS 1 (H) CIR Step Load Policing : enable : enable Bandwidth Profile (L2 Rate) CIR : 2 Mbps CBS : 8 KB EIR : 0 Mbps EBS : 8 KB Size type : EMIX Sequence : abcdeg User size : 2000 Service Acceptance Criteria FTD : 10000 us FTD type : max FDV : 2500 us FDV type : max FLR : 1.00e-06 M factor : 1 Mbps Device Informations: Testing layer : Layer-2 Peer MAC address : 00:15:AD:1B:57:60 NID MAC address : 00:15:AD:11:A5:FC Ethertype : 0x8902 Opcode : 3 MEG level : 7 Host VLAN Informations VLAN 1 Protocol : S-VLAN (0x88a8) Identifier : 1000 CFI/DEI : 0 Priority : 5 2015 SCTE 56

CONFIGURATION Started at : 2015-06-26 13:16:12-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 0.500 PASS 0.469 0.499 0.499 0 0.0e+00 5170 5259 5376 0 43 141 1 PASS 0.969 0.999 1 0 0.0e+00 5082 5227 5358 0 46 148 1.500 PASS 1.469 1.500 1.500 0 0.0e+00 5077 5215 5358 0 57 177 2 PASS 1.969 1.999 2 0 0.0e+00 5065 5188 5366 0 59 213 CIR/EIR N/A POLICING 2.500 PASS 1.951 2.006 2.082 932 1.8e-01 5053 5213 6915 0 64 1734 CIR*(1-FLRsac) <= IR <= CIR+EIR+M 2*(1-0.000001) <= 2.006 <= 2+0+1 Ended at : 2015-06-26 13:17:13-04:00 PERFORMANCE Started at : 2015-06-26 13:17:14-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) 2 PASS 1.959 1.999 2 0 0.0e+00 5080 5209 5668 0 97 525 Ended at : 2015-06-26 13:31:45-04:00 Service Overall Result: PASSED Service 2: Name : CoS 2 (M) CIR Step Load Policing : enable : enable Bandwidth Profile (L2 Rate) CIR : 3 Mbps CBS : 8 KB EIR : 0 Mbps EBS : 8 KB Size type : EMIX Sequence : abcdeg User size : 2000 Service Acceptance Criteria FTD : 10000 us FTD type : max FDV : 2500 us 2015 SCTE 57

FDV type : max FLR : 1.00e-06 M factor : 1 Mbps Device Informations: Testing layer : Layer-2 Peer MAC address : 00:15:AD:14:DF:4C NID MAC address : 00:15:AD:11:A5:FC Ethertype : 0x8902 Opcode : 3 MEG level : 7 Host VLAN Informations VLAN 1 Protocol : S-VLAN (0x88a8) Identifier : 1000 CFI/DEI : 0 Priority : 3 CONFIGURATION Started at : 2015-06-26 13:16:12-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 0.750 PASS 0.719 0.749 0.750 0 0.0e+00 5106 5233 5392 0 77 257 1.500 PASS 1.469 1.499 1.500 0 0.0e+00 5059 5223 5406 0 86 256 2.250 PASS 2.218 2.250 2.250 0 0.0e+00 5043 5222 6344 0 87 1200 3 PASS 2.968 2.999 3 0 0.0e+00 5048 5194 7530 0 63 2374 CIR/EIR N/A POLICING 3.750 PASS 2.965 3.006 3.089 1405 1.8e-01 5048 5201 6882 0 59 1679 CIR*(1-FLRsac) <= IR <= CIR+EIR+M 3*(1-0.000001) <= 3.006 <= 3+0+1 Ended at : 2015-06-26 13:17:13-04:00 PERFORMANCE Started at : 2015-06-26 13:17:14-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) 3 PASS 2.963 2.999 3.009 0 0.0e+00 5045 5208 6060 0 93 938 Ended at : 2015-06-26 13:31:45-04:00 Service Overall Result: PASSED 2015 SCTE 58

Service 3: Name CIR Step Load Policing : CoS 3 (L) : enable : enable Bandwidth Profile (L2 Rate) CIR : 5 Mbps CBS : 8 KB EIR : 0 Mbps EBS : 8 KB Size type : EMIX Sequence : abcdeg User size : 2000 Service Acceptance Criteria FTD : 10000 us FTD type : max FDV : 2500 us FDV type : max FLR : 1.00e-06 M factor : 1 Mbps Device Informations: Testing layer : Layer-2 Peer MAC address : 00:15:AD:14:DF:4C NID MAC address : 00:15:AD:11:A5:FC Ethertype : 0x8902 Opcode : 3 MEG level : 7 Host VLAN Informations VLAN 1 Protocol : S-VLAN (0x88a8) Identifier : 1000 CFI/DEI : 0 Priority : 1 CONFIGURATION Started at : 2015-06-26 13:16:12-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) CIR 1.250 PASS 1.219 1.249 1.250 0 0.0e+00 5094 5238 5368 0 60 160 2.500 PASS 2.468 2.500 2.500 0 0.0e+00 5097 5267 7530 0 135 2363 3.750 PASS 3.718 3.750 3.750 0 0.0e+00 5074 5224 6331 0 86 1083 5 PASS 4.967 4.999 5 0 0.0e+00 5050 5212 5890 0 87 681 CIR/EIR N/A POLICING 6.250 PASS 4.960 5.006 5.091 2187 1.7e-01 5049 5203 7447 0 71 2277 CIR*(1-FLRsac) <= IR <= CIR+EIR+M 5*(1-0.000001) <= 5.006 <= 5+0+1 2015 SCTE 59

Ended at : 2015-06-26 13:17:13-04:00 PERFORMANCE Started at : 2015-06-26 13:17:14-04:00 IR FL FTD FDV -------------------------- ------------- ----------------------- ---------------------- Pass/ Min Mean Max Min Mean Max Min Mean Max Fail (Mbps) (Mbps) (Mbps) Cnt FLR (usec) (usec) (usec) (usec) (usec) (usec) 5 PASS 4.960 4.999 5.023 0 0.0e+00 5046 5188 5585 0 69 499 Ended at : 2015-06-26 13:31:45-04:00 Service Overall Result: PASSED Test Overall Result: PASSED Ended at : 2015-06-26 13:31:45-04:00 2015 SCTE 60