The P3 U.S. Public Benchmark 2015

Transcription

1 2015 MAY

2 The P3 U.S. Public Benchmark 2015 In the first independent public benchmark of Voice over LTE services in the U.S., P3 measurements confirm that VoLTE brings significant improvements over existing legacy circuit switched voice services, delivering crystal clear voice quality, ultra-fast connection setup and paving the way for future rich communications services. Moreover, with VoLTE, U.S. carriers have successfully leapfrogged the world s best state-of-the-art networks in terms of voice quality and call setup speeds. The P3 U.S. Public Benchmark 2015 tested VoLTE service against legacy voice implementations with a clear outcome: VoLTE is ready for everyday use and significantly improves the customer experience. Using the latest hightech measurement equipment, P3 tested the voice services of all four U.S. wireless networks in the Washington, D.C. metro area with a primary focus on comparing the newest VoLTE services to legacy voice services. The test cases were designed to reflect average customer behavior and were all based on proven P3 measurement methodologies. Results show that the three existing U.S. VoLTE deployments in Washington, D.C. provide significant improvements over the legacy voice services and are reasonably reliable. Sprint has yet to announce its VoLTE plans. With sufficient LTE coverage and during VoLTE calls on the same carrier, the user experience improves dramatically. Overall, VoLTE channel call setup success and call drop ratios also were on an impressive level for such a new technology. In addition, all U.S. carriers still have to implement VoLTE calling across networks and VoLTE roaming, to broaden the number of customers who benefit from these improvements. 2

3 Introduction The U.S. is the largest and most advanced mobile communications market in the world. Leading U.S. carriers already operate some of the newest 4G LTE technologies and are moving very quickly to deploy them nationwide. Thus, the U.S. is a prime area of focus for P3 communications, an international leader in mobile benchmarking. This P3 U.S. Public Benchmark 2015 is designed to provide a clear, objective and technically sound overview of current VoLTE network performance. It highlights the carriers leadingedge VoLTE innovations and reveals potential weak spots in wireless service quality as perceived by customers. P3 s holistic approach to mobile network benchmarking and optimization has helped shape the wireless customer experience around the world. In the last year, P3 performed network testing and optimization projects in over 42 countries, including Australia, India, Qatar and the U.S.. P3 has performed annual public benchmarks in Germany since 2002 P3 selected Washington, D.C. because it represents a level playing field in terms of the spectrum available to all carriers and VoLTE deployment. In addition, the results can be compared to P3 s measurements in capital cities around the world. and in Austria and Switzerland since Australia and the UK were added in 2014, and P3 is expanding its measurements to additional countries in 2015 and In parallel, P3 is currently rolling out a crowd sourcing solution to provide superior network insights around the globe. P3 plans to perform a nationwide U.S. Public Benchmark in the first half of In Germany, Austria and Switzerland, P3 s annual benchmark is widely accepted as the most objective and thorough test of network performance, drawing on hundreds of thousands of voice and data measurements. It has significantly improved market transparency regarding measurement and comparison of network quality. Background U.S. wireless carriers have deployed and operated 4G LTE technology across large portions of their networks since 2010; LTE-capable smartphones are now available everywhere and dominate handset sales. In the second half of 2014, VoLTE was introduced by three of the four Tier-1 carriers in select markets across the country. But VoLTE is not yet the default technology used for voice calls; today many customers with VoLTE capable phones must opt-in to this technology (see page 6). Thus, the legacy circuit switched voice service still has a dominant impact on the wireless customer experience. Implementing good legacy voice service is still a particular challenge for the carriers when smartphones are not VoLTEcapable and use LTE for data. For each phone call non-volte devices have to switch back to legacy voice network standards like UMTS or CDMA. To do so, a device must first detect a legacy voice network and then switch to it. This process, called Circuit Switched Fall Back (CSFB), can lengthen call setup times and increase the possibility of failure. When VoLTE is used to make voice calls, this complicated CSFB procedure is not necessary. VoLTE-enabled devices remain on the 4G LTE network to initiate voice calls. The challenge for VoLTE is that an established and ongoing voice call must either remain on the 4G LTE network for the duration of the call, or seamlessly be handed off to the legacy 2G or 3G networks when moving out of 4G LTE coverage. P3 s benchmark was designed to test whether or not VoLTE implementations are providing the stable and high quality service promised and the level of improvement over legacy services. For these reasons, the P3 U.S. Public Benchmark 2015 was conducted using advanced VoLTE-capable 4G LTE devices and compared legacy circuitswitched and VoLTE technologies. 3

4 Approach The benchmark results are based on two weeks of intensive testing. Two specially-equipped measurement vehicles took thousands of test samples while travelling throughout the Washington, D.C. metro area on and inside the Capital Beltway (I-495). To ensure fair competition between carriers, each company was informed several weeks in advance about test methodologies, the main focus of the test cases, the target area and the test period. It was up to each carrier to decide if and how to best prepare for the test. At the same time, the P3 benchmark was designed to be strictly independent and impartial. The final say for the test design, the routes and the The route covered the dense downtown area of the nation s capital, as well as the quiet residential parts of Washington, D.C. to provide the most representative data possible. results interpretation was P3 s. And, needless to say, the carriers had absolutely no influence over the final results. Performance assessment for the P3 U.S. Public Benchmark 2015 was based on P3 s extensive worldwide network measurement and optimization experience. P3 communications has well over a decade of experience in network testing of this type. P3 works with clients around the world including wireless network carriers, device manufacturers and regulatory authorities. In 2014 alone, P3 s test vehicles covered more than 620,000 miles and compiled more than 40,000 measurement hours in 42 countries across 5 continents. 4

5 Timeline 2015 Feb Mar Apr May Jun Start of scoping and planning work. AT&T, T-Mobile, Sprint and Verizon Wireless were notified of the planned benchmark by P3, including preliminary information around scope, and were invited to provide feedback. P3 finalized all test parameters and selected June 2015 for benchmark publication. Testing began. All carriers were notified and asked to provide notice of any outages which might impact results. Testing concluded. P3 began post-processing of results. The P3 U.S. Public Benchmark 2015 is published. About P3 communications P3 communications is a leading international consulting, engineering and testing services company. P3 communications is part of the P3 group, which has over 2,700 employees worldwide and posted a turnover of more than $320 million in The company provides a broad portfolio of independent technical and management consulting services including network planning, engineering, end-to-end optimization, security, QoS and QoE testing, international benchmarking, device testing and acceptance services. P3 communications clients include network operators, equipment vendors, device manufacturers, public safety organizations and regulatory authorities around the world; the company s own experts combine strong technical know-how with many years of management experience. 5

6 VoLTE Today s wireless carriers are continually challenged to keep up with the explosion of wireless data traffic, first seen in fixed networks. Demand for wireless data quickly outpaced the capabilities of 2G and 3G networks and was the main driver behind the development of 4G LTE technology. The initial role of 4G LTE was to offload data traffic from the legacy networks and provide customers with a superior data experience, which worked well until now and will further improve in the next years. It also promised to make more efficient use of expensive wireless spectrum, unlock new and untapped spectrum, simplify network infrastructure and provide a clear path for future innovation. Well-functioning VoLTE service allows carriers to re-purpose ( re-farm ) valuable wireless spectrum currently used less efficiently by legacy 2G and 3G technology. Moreover, moving voice calls to the more efficient 4G LTE technology, will allow carriers to phase out 2G and 3G networks over time, further simplifying network infrastructure and handset complexity. For consumers, VoLTE comes with a promise of high definition call quality with higher fidelity voice signals and wider audio spectrum. Consumers who experience HD voice service by placing VoLTE-to-VoLTE calls within the same carrier are amazed by the clarity of the voice connection, asking How close are you? Other With nearly 400 commercial LTE networks in operation worldwide, 4G LTE has been very successful in complementing existing legacy 2G and 3G networks. However, one piece was sorely missing to really enable the carriers to start replacing their legacy infrastructure with LTE and LTE- Advanced technology: the basic voice call was not possible over LTE. This started to change with the availability of Voice-over-LTE (VoLTE) technology. VoLTE uses 4G LTE networks to carry voice calls instead of the legacy 2G or 3G networks. Based on the same Voice-over-IP technology already long used by the Internet s fixed IP connections, VoLTE makes use of the packet data bearer instead of the legacy circuit switched bearer. 4G LTE 6

7 benefits include typically shorter call setup times and the future promise of rich communications services (RCS) such as video calling. VoLTE is just the first step into a new world of communicating. The underlying IP Multimedia Subsystem (IMS) technology allows video chat and a wide range of completely new services and features including handover or offloading to WiFi networks. The VoLTE revolution has just begun. As of April 2015, only 16 commercial VoLTE services had been launched worldwide. The vast majority of VoLTE deployments are in South-East Asia, while three are in the United States AT&T, T-Mobile and Verizon Wireless. These three U.S. carries completed soft launches of VoLTE in the third quarter of 2014; carrier promotion of the service has been minimal to date and many customers with VoLTE capable phones must activate this feature manually. Verizon Wireless launched VoLTE nationwide on Day One, while AT&T and T-Mobile launched in a few markets and added markets over time. Is VoLTE ready to replace legacy voice service and Prerequisites for VoLTE are either good contiguous 4G LTE coverage or a methodology to handover an ongoing VoLTE call from 4G LTE to the circuit switched legacy service when the handset leaves 4G LTE coverage. This is one reason why carriers like AT&T, T-Mobile and Verizon invested heavily in the deployment of 4G LTE infrastructure. VoLTE now has been in operation for more than six months, which is a very long time in the fast-paced wireless industry. In this first public benchmark to include VoLTE measurements, we looked at how this new service stacks up against legacy technology and if it is delivering on its core promises of better voice quality and faster call setup times. become the new default choice for voice calls? 7

8 Methodology ant nuatr The basis for the benchmark was the proven P3 measurement methodology, used by the company for similar testing around the world. It is continuously adapted to the latest state-of-the art technologies and services. The methodology ensures a fair comparison of results between all networks tested, and yields reliable outcomes. Additionally, it enables a worldwide comparison of network performance. Smartphones Every year P3 reviews and decides which measurement device will be used for the upcoming tests. The selected smartphone must meet a detailed set of requirements. It is very important that it be widely used in the target market to ensure the benchmark results are representative of the customer experience; P3 only considers market-leading devices. The test smartphones also must accommodate installation of all necessary measurement software, making an Android operating system mandatory. Special local requirements are reflected by using local versions of the device and, if available, carrier specific firmware. The Samsung Galaxy S5 LTE Cat 4 was selected for the P3 U.S. Public Benchmark This device is based on the well-established Qualcomm Snapdragon 801 chipset, supports VoLTE and download-speeds up to 150Mbit/s. T-Mobile and Sprint device versions support circuit switched voice calling with wideband codecs. The AT&T device version used Carrier Aggregation during the benchmark. The Sprint device version does not support VoLTE. To duplicate the customer experience, P3 anonymously purchased the SIM cards, voice and data plans from normal retail outlets. It was ensured that any fair use policies, sometimes used to control data download speeds for customers with very high data usage, did not interfere with the test results. For the VoLTE measurement phones, P3 opted in to the carriers VoLTE services and enabled it on the phones. Influence of Measurement Devices on the Results VoLTE is a fairly new technology not only for the carriers and network infrastructure providers, but also for device manufacturers, which had to build in this completely new way of making phone calls. In addition, compared to the highly mature legacy voice technologies, VoLTE network protocols demand greater interaction between the network and the device. At this early stage of VoLTE maturity, the influence of the smartphones used can be significant; different smartphones can yield different results. The goal of this benchmark is to compare the new VoLTE service against the legacy voice service as a whole, which can be reasonably accomplished with a single measurement device. For an accurate comparison of individual carrier VoLTE implementations, however, more mature VoLTE smartphones will be required. All smartphones were mounted in a proprietary antenna attenuation box inside the vehicles to ensure proper test conditions with added signal attenuation to reflect a balance of realworld conditions. 8

9 Measurement System Each measurement vehicle was equipped with P3 s custom-built proprietary high-end measurement systems that provide real-world conditions one legacy voice, one VoLTE voice (except for Sprint) and one data test smartphone per carrier. All voice test calls were mobile to mobile, and set up from one car to the other. Voice measurements were taken while the cars were moving. One pair of smartphones per carrier was configured to preferably use VoLTE between cars where available and legacy (or mixed) voice where VoLTE was not (continuously) available. The other pair was set up to always use the legacy circuit switched implementation between cars. One of the legacy voice devices was configured for LTE preferred mode for data, which forced the networks to employ Circuit Switched Fallback (CSFB) if the device was in a 4G area. Voice calls in 3G-only or 2G-only coverage areas were operated as usual. To simulate typical user behavior, voice measurements were conducted while the test devices were handling small amounts of data traffic at the same time in this case, by receiving s. Speech quality was evaluated using the Perceptual Objective Listening Quality Analysis (ITU-T P.863 POLQA) wideband algorithm. POLQA-Wideband has become an important measurement standard for next-generation networks, as carriers deploy high-definition voice services and offer compatible handsets. It delivers an advanced level of benchmarking accuracy and adds significant new capabilities for wideband and super-wideband (HD) voice signals, along with support for the most recent voice coding and VoIP transmission technologies. Besides speech quality, key performance metrics included call setup times, call setup success ratios and call drop ratios. Logistics and Routes The testing was conducted on and inside the Washington, D.C. Capital Beltway (I-495). The test vehicles drove different routes within Washington, D.C. to maximize the area covered. To prevent the risk of one car distorting the measurements of the other, the drivers avoided traveling in the same exact areas I-495 Washington, D.C. Capital Beltway Testing route Annandale 2 test vans 1,300 miles Bethesda Arlington at the same time. In total, the test vehicles covered more than 1,300 miles. Across all services combined on all networks, they conducted roughly 7,000 voice calls with more than 50,000 speech samples. In all, more than 140 Gigabytes of data were transferred. Silver Spring Washington Alexandria 50,000 speech samples Hya sville 7,000 voice calls 140 GB data 9

10 Results In order to see how VoLTE performs compared to the legacy voice service, we looked at both technologies independently and compared the results. The big question: Is VoLTE delivering on its promise? VoLTE service on all three carrier networks and with the selected measurement device showed acceptable results in connecting and holding VoLTE calls (availability and retainability) comparable to the legacy services. The P3 benchmark focused on the core VoLTE promises of improved call setup times and voice quality. VoLTE Channel Call Setup Not all call attempts of the VoLTE preferred devices actually ended up as VoLTE-to-VoLTE calls. Of all attempted calls on the VoLTE-preferred channels, the shares of connected calls using VoLTE technology on both ends were between 97.3% and 97.8%. The other call attempts were set up either as pure legacy calls, with only one side on VoLTE and the other side on legacy, failed to connect or timed out. The portions of call attempts of the VoLTEpreferred channels that did not connect within the first 20 seconds were between 0.50 and 1.09%. The measurement system was configured to classify call attempts as timed-out if they take longer than 20 seconds to connect. This is plenty of time for a pure VoLTE call. Overall, the VoLTE channel call setup success ratios were on an acceptable level with slight room for improvement. Circuit Switched Fallback Circuit Switched Fallback (CSFB) is a mechanism that helps when there is a need to place a legacy voice call with a device that is currently camping or transferring data on the LTE network. This can be the case either because the device is not VoLTE enabled or for some other reason VoLTE is not available to set up the call. The device thus needs to hand down to and register in the legacy network and then proceed with the call setup. Verizon Wireless doesn t need this mechanism because their devices are registered in both LTE and legacy network at the same time. This increases device complexity, but simplifies the network infrastructure and removes a potential point of failure. How does CSFB perform for AT&T, Sprint and T-Mobile? 10

11 CSFB for legacy devices One legacy measurement device was configured to prefer LTE for data transmission, forcing it to employ CSFB to place or receive a legacy voice call, while the other was set to prefer 3G data technology, circumventing the need for CSFB. The below table shows the CSFB success ratio and the average and maximum time it took to perform the CSFB procedures. It s Carrier CSFB success ratio interesting to see that both for Sprint and T-Mobile, we observed a few very long CSFB procedure durations, which can be problematic because the device cannot be reached by the network during that time, and indicates a potential for optimization. The CSFB success ratios and durations are comparable to other carriers around the globe. Average CSFB duration Maximum CSFB duration AT&T 98.67% 1.57s 5.13s Sprint 98.91% 2.19s 10.52s T-Mobile 99.7% 1.45s 14.45s CSFB for VoLTE devices The AT&T VoLTE measurement phones only performed one CSFB procedure to place a call. The CSFB succeeded and the procedure took only 0.68 seconds. For the T-Mobile VoLTE channel, we measured 30 CSFB attempts, 16 on the calling and 14 on the receiving side. On average, the procedure took 3.91 seconds, and one CSFB attempt failed, resulting in an unsuccessful call setup. Call Setup Time Improvements Call setup time is the time it takes between dialing a call on the calling party and the ringing on the called party. Legacy networks especially when using mechanisms like CSFB typically have call setup times between 5 and 15 seconds, and sometimes even longer. A short call setup time is desirable because customers may abandon the call attempt if it is too long AT&T Sprint The legacy voice implementations showed average call setup times between 6.1 and 7.8 seconds but quite a few were longer than 15 seconds. VoLTE shows an impressive improvement for all three VoLTE carriers, especially AT&T and Verizon. T-Mobile is slightly slower but still with very good results T-Mobile Verizon Call setup time: legacy VoLTE preferred

12 Voice Quality Improvements Voice quality was measured by playing reference samples on the talking side and recording the transmitted samples on the listening side. Then, the POLQA wideband algorithm was applied to derive a Mean Opinion Score (MOS) on a scale between 1 and Higher scores mean better voice quality The circuit switched MOS showed the impact of the different technology implementations between the carriers. Verizon and AT&T don t use wideband or HD codecs in their CDMA and UMTS networks, which resulted in average MOS scores around 2.3. Both T-Mobile and Sprint s networks showed better average MOS scores of VoLTE with its HD voice codecs lifts the VoLTE-preferred channels of the three VoLTE-carriers to speech quality scores around This is an impressive improvement for customers, especially those of AT&T and Verizon, which have the lower MOS scores on their legacy networks compared to T-Mobile. Sprint is clearly missing out on an opportunity to improve the customer experience. 0 AT&T Sprint T-Mobile Verizon Voice Quality (MOS - Mean Opinion Score) The POLQA algorithm was not the only confirmation of the improvements that VoLTE brings to speech quality. Also when comparing the spectral densities of the recorded speech sample for the different technologies under good radio conditions, the diagrams clearly show that the received VoLTE sample contains much higher frequencies than the recorded legacy sample. This does not only result in a higher MOS score, but in better voice clarity and experience for customers. New VoLTE users often are surprised to hear the difference, asking How close are you? P3 engineers also examined which wideband voice codecs predominated in each VoLTE implementation. Even though T-Mobile focuses on the higher bitrate voice codec (23.85 kbit/s), the voice quality results are more or less the same as for AT&T and Verizon who use a kbit/s codec. 12

13 -30dB -35dB -40dB -45dB Speech sample received on legacy UMTS -50dB -55dB -60dB Original speech sample -65dB -70dB -75dB -80dB -85dB -90dB 1000Hz 2000Hz 3000Hz 4000Hz 5000Hz 6000Hz 7000Hz 8000Hz 9000Hz 10000Hz 11000Hz 12000Hz 13000Hz 14000Hz -30dB -35dB -40dB -45dB -50dB -55dB -60dB Speech sample received on VoLTE Original speech sample -65dB -70dB -75dB -80dB -85dB -90dB 1000Hz 2000Hz 3000Hz 4000Hz 5000Hz 6000Hz 7000Hz 8000Hz 9000Hz 10000Hz 11000Hz 12000Hz 13000Hz 14000Hz VoLTE Channel Call Retainability The overall dropped call ratios of the VoLTE channel were only slightly higher than the drop ratios of the legacy channel. This is impressive for such a new technology with minor room for improvement. Both AT&T and T-Mobile have implemented SRVCC (Single Radio Voice Call Continuity) functions in their network to hand existing VoLTE voice calls over to their legacy networks when a device leaves the LTE footprint. Ideally, SRVCC does not result in interruptions such as speech gaps noticeable to callers. All carriers showed a good LTE footprint inside the Capital Beltway. There were just five SRVCC attempts on AT&T and one on T-Mobile. All of them were successful and did not result in call interruptions. AT&T handed down the VoLTE wideband call to a narrowband 3G UMTS call and took on average 140 milliseconds, while interestingly T-Mobile handed down the wideband VoLTE call to a narrowband 2G GSM call within 215 milliseconds, and then after another 21 seconds handed it over to a 3G UMTS call. This suggests different SRVCC layer strategies between AT&T and T-Mobile. In conclusion, SRVCC successfully provides a solution for leaving the networks LTE footprints, with an expected reduction in voice quality and a slightly different implementation between AT&T and T-Mobile. Verizon Wireless has not implemented SRVCC functionality; when going out of LTE coverage, an ongoing VoLTE call would drop. The coverage requirements for the 4G network of Verizon with VoLTE enabled are thus much higher as there is no safety net of SRVCC. It needs to provide contiguous LTE coverage to keep VoLTE calls going. 13

14 International Comparison The U.S. is one of the first countries with commercially available VoLTE services. P3 wanted to see how much carriers in other countries could benefit from introducing VoLTE if they follow the U.S. implementations. P3 has carefully selected relevant metropolitan areas around the world with recent measurement results from carriers without VoLTE deployments, and compared voice quality and call setup times in these markets with P3 s U.S. VoLTE-to- VoLTE results. Voice quality in these markets is already on a fairly good level, but comparison with the U.S. VoLTE-to- VoLTE results shows improvement potential in many of these markets, especially for call setup times. It will be interesting to see if continued Bottom line: many international networks could benefit from an upgrade to the latest VoLTE technology. optimization of their VoLTE networks will move U.S. carriers even farther ahead, and how the first VoLTE implementations in the international markets will perform. 4 Average Voice Quality [MOS - Mean Opinion Score] (the higher the better) Amsterdam Berlin London Melbourne Munich Sydney Washington, D.C. Washington, D.C. Vienna Zurich legacy calls only VoLTE to VoLTE calls only 14

15 Average call setup times [s] (the faster the better) Amsterdam Berlin London Melbourne Munich Sydney Washington, D.C. Washington, D.C. Vienna Zurich legacy calls only VoLTE to VoLTE calls only Conclusions The P3 U.S. Public Benchmark 2015 evaluated the improvements that Voice-over-LTE brings over the legacy voice services, by performing extensive measurements in the Washington, D.C. metropolitan area. With VoLTE, the U.S. carriers overtake European carriers in terms of voice quality and call setup speeds. Yet, the U.S. carriers will still have to implement VoLTE calling across networks and VoLTE roaming, to broaden the number of calls that benefit from these improvements. Without a VoLTE implementation, Sprint is currently trailing behind its U.S. competitors and will greatly benefit from its introduction. The results clearly show: VoLTE is ready for everyday use and brings significant improvements to the customer experience. All carriers already offer a number of VoLTE-capable devices from Apple, HTC, LG and Samsung. Verizon Wireless also offers Motorola VoLTE phones. But, in many cases, customers still need to opt-in to enable the VoLTE services and should contact their carriers to enable this new feature. VoLTE currently is being deployed by a number of carriers around the globe. The U.S. results show what is already possible with VoLTE with regard to improved voice quality and call setup times. Further improvements can be expected over time. P3 communications will continue to follow the evolution of networks and the introduction of new technologies. 15

16 The Future The P3 U.S. Public Benchmark 2015 is intended to be the first in a regular series of such reports The mobile industry in the U.S. moves very quickly and is at the forefront of deploying and operating some of the newest LTE technologies, like VoLTE, LTE-Advanced, Carrier Aggregation and WiFi offloading. Right now, each of the carriers has a number of major cutting-edge initiatives under development to boost their networks and services in different ways. P3 expects to see significant changes by the time it performs the nationwide P3 Mobile Benchmark U.S. 2016, which will cover a significantly larger geographic scope and additional services. P3 communications, Inc, 412 Mt. Kemble Ave, Suite G20s, Morristown, NJ phone , [email protected] Headquarters: Aachen (GER), Belgrade (SRB), Morristown (NJ, USA), Sydney (AUS) All rights reserved. The information and statements provided in this document are for informational purposes only. Any measurements, recommendations, opinions or findings provided are only true and accurate for the conditions encountered at the time of any survey and P3 communications accepts no liability for any errors or omissions. In no event will P3 communications be liable for any consequential, indirect or incidental damages whatsoever, including lost profits, business, contracts, revenue or anticipated savings arising out of the use of any P3 communications products or documentation (or from errors or deficiencies therein), reliance on the data or measurements produced or information contained within. [Image Sources: Fotolia (Page 4: SeanPavonePhoto, Page 6: Petr Vaclavek, Page 11: freebird, Page 13: Nikolai Sorokin, Page 14: WoGi]