A cloud-based approach for rational network life cycle management Jointly presented by Telenor and Netrounds on Sep 19, 2013 at the TC3 conference arranged by the Telecom Council of Silicon Valley
About Netrounds Netrounds was founded in 2007 and is headquartered in Luleå, Sweden, with offices in London, Singapore and Boston. Netrounds is the world s leading cloud-delivered tool for network test automation. More than 150 network operators, enterprises and network consultants worldwide use Netrounds to improve operational efficiency, reduce capital expenditures and secure high end-user experience. INTRODUCTION This case study describes how Telenor Sweden, a large Swedish telecom operator, initiated a project to introduce Netrounds as a cloud-delivered network performance management solution. The main goal was to achieve automation of tasks related to verification, optimization and troubleshooting of Telenor s IP network. Traditionally, many of these tasks have been performed manually in a very labor-intensive way, requiring expensive and complex hardware and software tools. A secondary goal was to enable cross-organizational collaboration. This case study describes how a cloud-delivered tool, like Netrounds, is able to significantly rationalize this work by simplifying and automating more than 50% of manual tasks. In addition it is described how the solution helps promoting collaboration within the organization as well as with external consulting partners and suppliers. Benefits gained: Shorter regression testing cycles Increased service launch velocity Improved end-user quality More efficient and competitive organization Page 2
About Telenor Group: 149 million subscribers Operations in 11 markets 30 000 employees Telenor Sweden is a Swedish subsidiary BACKGROUND Telenor Group is one of the world s major mobile operators with 149 million mobile subscribers and a large customer base on their wired broadband network. They have operations in 11 markets and around 30 000 employees worldwide. The Swedish subsidiary, Telenor Sweden, is a successful competitor on the Swedish market. Telenor Sweden offers both wired and wireless services towards the all market segments. Telenor Sweden has similar challenges as all other network operators over the world; the complexity of the network increases with the endless introductions of new platforms and new advanced services. Furthermore, the increased market competition requires enhancements in service launch velocity as well as a cost-efficient organization. At the same time, the resources for quality assurance are restrained, and problems not found during pre-deployment testing will drive additional costs for customer service and customer support. One of the key factors Telenor identified in order to improve service launch velocity and service quality was the introduction of automation during the quality assurance process. This requirement had been around for several years, but nothing happened as most systems were really expensive, and it was considered difficult to find the time needed in order to introduce automation. Another key factor that was identified was to improve the collaboration with suppliers, such that they could do a more thorough and specific testing according to Telenor s network conditions and actual service realizations. Often, situations could occur where a new important service was planned for launch, but had to be delayed for several months since verification tests revealed bugs in some suppliers hardware or configurations. After several iterations of regression testing the service was finally ready for release. Telenor Sweden also identified that partnering with external test consultants could help focusing the available resources and further optimize the organization. Page 3
Steven Muir, responsible for Terminals and CPEs at Telenor Sweden AB, is very positive to the introduction of Netrounds about a year ago. He says: - During the past year we have been very pleased with the Netrounds platform, in terms of expanded test suites and improved usability for our engineers. - The support for automation of tests has been improved thanks to the cooperative dialogue we have with the Netrounds team, and the combination of automation with simplicity of use makes Netrounds an important strategic testing tool for us to achieve higher efficiency and quality. Therefore, Telenor Sweden started looking for tools and solutions to address their needs. Netrounds, although a new system on the market, was identified to fulfill all the basic requirements: 1. Easy to implement and simple to use 2. Supports test automation 3. Supports collaboration, which is needed for efficient work together with partners CURRENT SITUATION Telenor Sweden is going through some major organizational changes, also including collaboration with external test partners that bring special test competences, as well as giving their suppliers the duty of fully testing their releases before sending to Telenor. In this situation Netrounds is seen as a valuable tool for being able improving the quality assurance process and support functions, while still staying in control of which features should be tested and approved prior to deployment. Today, Netrounds has been identified as a potential solution for three different groups within Telenor Sweden: Terminals and CPE, Service Management Centre and Support Product Specialists. The first of the three groups have been using Netrounds for more than a year. The different groups use Netrounds prior to deployment of new hardware and/or software in their customer-placed equipment, both for their mobile and wired platforms, as well as their access platforms (both fiber-to-the-home switches, and DSLAMs). Some of the features that are tested with Netrounds are: Performance Long-term stability and quality Broadband TV (IPTV) in combination with peer-to-peer traffic as this has been a confirmed as a real-world problem for some CPEs Quality-of-Service (QoS) and prioritization mechanisms Access Security Page 4
Features tested with Netrounds: Network performance Long-term stability and quality Broadband TV (IPTV) in combination with peer-topeer traffic as this has been a confirmed as a real-world problem for some CPEs Quality-of-Service (QoS) and prioritization mechanisms Access Security Naturally, there are some tests that are not possible or feasible to automate. However, a great benefit even with partial automation is that the network engineers can free their valuable time. So instead of spending time on tests that can be automated, engineers can put extra efforts into tests that cannot be automated and do them more thoroughly. Also, since test coverage can be increased when done in an automated way, fewer problems are likely to surface in the live production network. Since Netrounds measurement devices are possible to distribute anywhere in the production network, Netrounds is also used by third line support when troubleshooting at customer sites. Here, the same tests as performed in the lab can be reused. This makes it convenient to compare live and lab results since the test execution is identical down to every single packet. Future plans: Teams can collaborate both internally and externally Necessary tests can be developed by experts and shared among groups Tests can be shared to suppliers for simple reproduction FUTURE PLANS With respect to the network life cycle, many phases are involved; ranging from development tests in the lab, predeployment verification in parts of the live network, inproduction network optimization and troubleshooting towards the customer premises. Using a solution that is applicable to all of these phases in the network life cycle brings a number of great possible advantages: Teams can collaborate both internally and externally Necessary tests can be developed by experts and shared among groups Tests can be shared to suppliers for simple reproduction Netrounds can be used by external test partners and suppliers to execute tests that Telenor have specified to be used. This makes it possible for Telenor staff to work more strategically and long-term, and still keep control of the test process. In the future the expectation is to be able to automate more than 70%. Page 5
Netrounds fulfills the need from Telenor Sweden Introduced a year ago Identified as a potential solution for three different groups An important strategic testing tool Key factors for Telenor Sweden choosing Netrounds: Ease of use Automation support Simplicity to deploy To make this happen the knowledge about Netrounds needs to be spread across several parts of the organization. All involved staff has to realize the benefits gained by using Netrounds instead of using old and manual methods. This is a challenge in a large organization, but definitely worth working for. Since Netrounds is continuously improved based on customer requirements, Telenor and Netrounds have an active discussion about the development and which features to include in the regular updates every six weeks. The next feature in Netrounds being identified as most important is IPv6 support. From having been an area of discussion and anticipation during more than ten years, IPv6 is now becoming a reality. This increases the need for tools to verify and troubleshoot, especially since it is a new technology where the knowledge is limited. Here, Netrounds is seen as an important tool to pave the way for a successful introduction. Driven by Telenor s and other customers expressed needs, Netrounds has started the design and implementation of support for IPv6 and plans to make the feature available at the end of 2013. CONCLUSION Netrounds has been introduced into Telenor about a year ago. During that time it has evolved in the direction that Telenor hoped, with more test features and also improved support for automation. In a world where the technical complexity and where the competition increases day by day, automation is seen as a key factor in order to stay competitive and to be able to secure high end-user experience. Netrounds fulfills the need from Telenor Sweden, where the ease of use, automation, and also the simplicity to deploy were key factors when making the decision to become customers. Netrounds is the world s first professional and cloud-delivered solution for troubleshooting and test automation - it is the future of network testing. Page 6
Wide range of tests With the wide range of test features, and its unique ease-of-use, Netrounds supports the entire life cycle, ranging from automated lab testing before service launch and delivery testing of L2/L3 VPN connections to 24/7 monitoring of network and triple-play performance. Netrounds also supports remote troubleshooting in the production environment. The same intelligent measurement devices can be used regardless of use case. APPENDIX I: TYPICAL USE CASES FOR A NETWORK OPERATOR Case 1: Automated lab testing Many of the manual tests typically performed during regression testing of new firmware releases for network devices can be automated with Netrounds. In this use case, 3-10 measurement devices are connected to a reference lab setup. This makes it possible to stress test customer premises equipment (CPE) such as triple-play Residential Gateways (RGWs), and access networks. Netrounds helps ensuring that expected levels for performance, quality, stability and security are all met. As much as up to 70% of manual efforts can be automated. This increases both service velocity and quality at the same time as staff resources are available for more advanced tasks that are not possible to automate. Case 2: Pilot testing Once new equipment firmware has been verified in the lab environment, commonly a few selected nodes are upgraded in the production network before the entire network is updated. The purpose is to find problems that potentially slip through the lab testing process, while only affecting a limited number of subscribers in the production network. Since Netrounds is fully distributed, it is simple to deploy Netrounds measurement devices on site at the selected pilot nodes. By reusing the identical automated test suites that were used during the pilot tests in the lab, it is possible to find and fix problems that only surface under fully realistic situations. This decreases the duration time for the pilot test and shortens time to service launch. Case 3: Verification of backhaul and business L2/L3 VPN connections Verification of mobile backhaul transmission and business VPN connections has traditionally been made using hand-held testers that perform RFC 2544 throughput measurements. However, this method has several drawbacks. Firstly, RFC 2544 is a rather old standard that primarily was developed in order to benchmark individual network equipment in a lab, rather than a complete (and quite often complex) network connection. Secondly, the method is not trivial to setup and the field staff requires considerable training before operating the test instruments. Thirdly, the test results and reports are Page 7
Unique collaboration possibilities Providing all these features in a userfriendly and modern web user interface, makes it also possible for different parts of the organization to collaborate in a way that has never been possible before. This is a key for competitive and customer-oriented operations for modern network operators. Case 4: Monitoring of triple-play and business services Netrounds supports in-production monitoring of network and services, providing periodic SLA compliance reports and alarms generation in case of quality degradation. Monitoring of quality demanding services such as IPTV, Digital Cable TV, IP Telephony monitoring and other business critical services are considered absolutely necessary for achieving professional network operations. Netounds helps solving problems proactively and provides means for quick remote troubleshooting without truck rolls. Case 5: Second line and third line troubleshooting As explained in the other use cases, Netrounds helps minimizing costly problems through automated quality assurance testing. However, unexpected problems might occur when the network is put in production. These problems have to be quickly isolated and solved. Netrounds supports this troubleshooting process in several steps as explained below, minimizing the need for field efforts. Netrounds provides a three-step process: 1. Netrounds' intelligent measurement devices have built-in responders for browser-based speed tests. This makes it possible for subscribers to trigger network performance measurements themselves using an ordinary web browser. All test results are stored at the Netrounds cloud servers with full visibility of the subscribers' achieved performance results by the network operator support staff. 2. For more detailed remote measurements, a subscriber downloads and installs a measurement agent on its own computer (Windows, Mac, Linux) in order for the support team to remotely do long-term measurements of stability and performance. 3. If the two first steps still do not provide sufficient details for the support team, a measurement appliance may have to be deployed at the subscriber location. The plug-and-play property makes it possible for a customer to pick up and connect a measurement appliance at home. Now, very powerful tests are possible to be carried out by second or third line support teams. As an alternative, a field technician easily transforms an ordinary laptop into a professional measurement appliance and brings this to the subscriber location. The actual testing and analysis are commonly made remotely by second/third line support staff, minimizing the need for highly trained personnel in the field. Page 8
Netrounds' features are possible to use in practically unlimited combinations. This enables for continuous 24/7 monitoring with automatic SLA compliance alarms and report generation. In addition to powerful monitoring capabilities, Netrounds is able to execute individual test cases or full test suites either on-demand or scheduled for later execution with pass/fall criterion based on the concept of "errored seconds". Execution can even be triggered through an API exposed towards Netrounds' cloud servers. APPENDIX II: CAPABILITIES OVERVIEW Network Performance leverages the built-in synthetic traffic generators available in Netrounds' measurement devices. Traffic streams for TCP/UDP can be crafted with custom throughput rate, packet sizes, destination ports and QoS headers. IPTV and Digital Cable TV measures the video quality of existing MPEG video streams. Multiple channels can be measured concurrently on multiple locations. SIP communications allows for measuring call quality and call setup times using the built-in SIP clients in the measurement devices. HTTP and DNS makes it possible to measure download rates and response times measured towards existing HTTP and DNS servers. Network Speed is a feature that enables operators and enterprises to set up their own speed test sever/responder that allows end-users/subscribers to assess network performance within a relevant part of the network rather than involving Internet. Access Security testing is used to quickly identify and eliminate potential security weaknesses in order to prevent loss of data or service interruption. Remote Traffic Analysis is a feature for capturing traffic remotely from measurement devices located anywhere in the network. Transparency testing contains a complete set of test cases to easily verify if a Layer-2 or Layer-3 point-to-point connection has the correct characteristics in terms of protocol transparency and Class of Service marking/remarking. Page 9
Netrounds offers cloud-delivered network performance management that helps operators and IT operations to ensure that networks and Internet services are accessible, usable and secure. More than 150 network operators, enterprises and consultants worldwide use Netrounds to increase operational efficiency, reduce capital expenditures and deliver high end-user experience Founded in 2007, Netrounds is headquartered in Luleå, Sweden, with offices in London, Singapore and Boston. Netrounds Solutions AB Storgatan 9 972 38 LULEÅ, SWEDEN Phone +46 920 420015 www.netrounds.com Copyright 2013, Netrounds. All rights reserved. This document is issued to provide information only and the content is subject to revision without notice due to continued development. The product and services described are subject to availability and change without notice. Page 10