LTE Test: EE 4G Network Performance

Transcription

1 LTE Test: EE 4G Network Performance An Analysis of Coverage, Speed, Latency and related key performance indicators, from October 31 st to December 20 th, 2012.

2 Disclaimer Epitiro has used its best endeavours to make sure that all the information, data and other material (copy and images) in this document are accurate and complete. Epitiro does not accept liability for any errors or omissions. Copyright Notice Reproduction, redistribution and transmission of any information, data or other material contained in this document for any other purpose whatsoever is strictly prohibited without the prior written permission of Epitiro Group Ltd. 2

3 TABLE OF CONTENTS EXECUTIVE SUMMARY... 4 INTRODUCTION... 6 METHODOLOGY... 7 FINDINGS MOBILE BROADBAND PERFORMANCE... 8 AREAS OF SPECIAL INVESTIGATION EE 4G AVERAGE SUMMARY TABLE CONCLUSIONS AND FUTURE WORK IPQ PERFORMANCE MONITORING SOLUTION

4 EXECUTIVE SUMMARY This report is an analysis of the EE 4G mobile broadband service launched at the end of October The report is based on actual 4G QoE performance data collected by Epitiro for the following three test periods; Test Period 1: October 31 st up to November 20 th Test Period 2: December 3 rd to December 9 th Test Period 3: December 12th to December 19 th Epitiro s ipq Performance Monitoring Solution was used to measure key performance indicators (KPIs) including Coverage, Speed and Latency. Throughout this project Epitiro has also undertaken Areas of Special Investigation including; CS Fallback/Return Indoor vs. outdoor comparison testing The impact of 4G deployment on 3G call completion The behaviour of 4G data sessions during a voice call Comparison of iphone and Android 4G performance Data was collected via walk testing and drive testing city centre routes where EE have indicated 4G service availability on their published coverage map. Testing was conducted in Cardiff, Bristol and Birmingham. The 4G test data collected consists of the following sample sizes; Throughput - 1,101 samples Latency - 9,661 samples RF - 201,853 samples 3G CCSR - 4,943 samples The following has been observed; 1) 4G availability was found to be generally as indicated on the EE Website, though this study did not seek to absolutely endorse the predictive map published by EE. 2) The average time on a 4G was 6, based on all data collected. The highest average time on a 4G Bearer by city was 77.6% in Birmingham during the final test period. The lowest average time on a 4G bearer by city was also in Birmingham at 49.6% during Test Period 2. 3) The average speed measured during the study was 15.8Mbps. 29% of observed speeds were faster than 20Mbps. The fastest speed recorded was 52.4 Mbps in Birmingham. 4) The average latency recorded was 39.4ms. Birmingham provided the lowest latency measurement at 22ms. 5) A device with 4G connectivity was observed to switch to a 3G session if a voice call was made or received, returning to 4G service after the voice call is finished. 6) The average CS Fallback return time based on all data collected was 37 seconds. CONCLUSIONS The EE 4G Service is performing to a standard that significantly outperforms 3G networks and rivals the technical performance of home ISP services in terms of fast speeds and low latency. Improved consistency in 4G network connectivity would benefit subscribers. Epitiro will continue its research into 4G mobile broadband performance and will publish findings throughout

5 SUMMARY TABLE Test Period #1: Nov 23 rd Results Test Period #2: Dec 12 th Results Test Period #3: Dec 20 th Results Average Max Average Max Average Max Throughput Speed Cardiff Throughput Speed Bristol Throughput Speed Birmingham 12.6 Mbps 48.4 Mbps 14.7 Mbps 41.8 Mbps 19.0 Mbps 31.9 Mbps 13.2 Mbps 21.4 Mbps 18.1 Mbps 41.4 Mbps 15.8 Mbps 44.4 Mbps N/A N/A 14.8 Mbps 21.7 Mbps 20.3 Mbps 52.4 Mbps Average Min Average Min Average Min Latency Cardiff 41 ms 26 ms 33 ms 24 ms 42 ms 26 ms Latency Bristol 42 ms 27 ms 35 ms 23 ms 37 ms 23 ms Latency Birmingham N/A N/A 36 ms 22 ms 31 ms 22 ms 4G 3G 4G 3G 4G 3G Time on Bearer Cardiff Time on Bearer Bristol Time on Bearer Birmingham 52.7% 47.2% 50.7% 49.1% 65.4% 27.8% 62% 35% 51.7% 47.1% 75.8% 22.9% N/A N/A 49.6% 50.1% 77.6% 21.4% Average Min Average Min Average Min CS Fallback Return Times Cardiff CS Fallback Return Times Bristol CS Fallback Return Times Birmingham 29.9 s 18 s 29 s 14 s 44 s 21 s N/A N/A 29 s 8 s 44 s 21 s N/A N/A 35 s 7 s 45 s 37 s 5

6 INTRODUCTION Since 2000, Epitiro has measured broadband performance and regularly published studies that examine the key performance indicators (KPIs) that affect the quality of experience. Epitiro provides products and broadband performance data to MNOs, ISPs, industry bodies and telecommunication regulators for benchmarking, service assurance and customer experience management purposes. FINDINGS TO DECEMBER 20, 2012 This report is a publication of findings to December 20 th 2012 and compares the findings to those published November 23 rd and December 12 th, Figure 1 Bristol Route Map (Nov 23 rd Results) Using Epitiro's ipq technology that enables measurements of broadband services from the customer experience perspective, this report details the performance of throughput speeds for EE s 4G mobile broadband services in the cities of Cardiff, Bristol and Birmingham UK during October-December Epitiro conducted walk testing and drive testing within areas deemed to have 4G coverage according to EE. The analysis of that data forms the basis of this report. OBJECTIVES OF THE STUDY Epitiro s objectives for this on-going study are to understand the performance of EE s 4G services in context of: Figure 2 Walk Test Apparatus Outdoor coverage Indoor coverage Speeds & Latency Availability & Reliability For the purposes of this research, Epitiro focused on cellular broadband services specifically as experienced by users of the latest Android 4G smartphone devices. 6

7 METHODOLOGY SCOPE OF TESTING This study of mobile broadband services in UK cities of Cardiff, Bristol and Birmingham and focused on measuring the performance delivered by Everything Everywhere s (EE) 4G Mobile Service which launched at the end of October The measurements captured during the testing included coverage, download speed and latency, as well as areas of special investigation. APPROACH AND METHODOLOGY The measurements that form the basis of this report were conducted by walk testing and drive testing using Epitiro s ipq Performance Monitoring Solution which comprises test apps on Android smartphones that report data (real-time) to a central analysis dashboard. The approach was one of sampletaking, rather than 24x7 monitoring, in order to acquire a high-level understanding of 4G performance in the target cities. Testing was carried out between 7am and 10pm, covering the working hours and early evening of each day. The adopted drive test routes covered the central business districts and surrounding urban areas. AREAS OF SPECIAL INVESTIGATION Analysis of data after each test period prompted Areas of Special Investigation. A comparison between in-building and outdoor performance was one such initiative. Epitiro also investigated CS Fallback and return times. Epitiro also investigated how 4G data sessions behave and compares performance of 4G on both the iphone and Android smartphone models. A detailed description of the testing methodology is available from Epitiro. 7

8 FINDINGS MOBILE BROADBAND PERFORMANCE This section examines the performance of the EE 4G mobile broadband services as observed during this study, including Time on 4G Bearer, Throughput Speed and Latency. 4G AVAILABILITY IN CARDIFF, BRISTOL AND BIRMINGHAM The ipq Test App records the bearer and location as part of each series of tests, which are used to derive the route maps below. The intent has been to sample the 4G coverage to obtain an indication of service availability, rather than complete a full actual coverage map. Drive test routes are shown for Test Period 1: November 23 rd, Test Period 2: December 12 th and Test Period 3: December 20 th. Figure 3 EE 4G Drive Test Routes: Cardiff November 23 rd Results (left), December 12 th Results (right), December 20 th Results (bottom) 8

9 Figure 4 EE 4G Drive Test Routes: Bristol November 23 rd Results (left), December 12 th Results (right), December 20 th Results (bottom) Figure 5 EE 4G Drive Test Routes: Birmingham December 12th Results (left), December 20 th Results (right) 9

10 Percentage EE 4G Network Performance Analysis TIME ON 4G BEARER The charts below indicate the percentage of time the network exposed a 4G bearer during the testing conducted across Cardiff, Bristol and Birmingham. Note that as a voice call is made during the test routine, EE will establish the call on a 3G bearer. Previously an area of special investigation, the time to return to an LTE bearer following CS Fallback averaged about 30 seconds. This period of fallback during and following a test voice call could affect the Time on Bearer findings. One significant change for Test Period 3 was a focus on drive testing of city centres where we expect to experience a more concentrated deployment of LTE cells as well as a higher demand for 4G usage. Findings: In Cardiff, Time on a 4G Bearer increased to 65.5% during Test Period 3, up from 50.7% (Nov 23 rd ) and 52.7% (Dec 12th). Whilst time on 3G (27.8%) was lower than previous testing, for the first time a noticeable time on 2G (6.7%) was measured. Figure 6 EE 4G Time on Bearer: Cardiff % % 47.2% 49.1% 50.7% 1 0.1% 27.8% 6.7% 0.2% 2G 3G LTE Bearer Type November 23rd Results December 12th Results December 20th Results 10

11 Percentage Percentage EE 4G Network Performance Analysis In Bristol there was a significant change increase in time on a 4G bearer (75.8%) in the city centre drive test. Previously, an LTE bearer was available for only 47% of the time in Bristol (Dec 12 th ) vs. 62.7% (Nov 23rd). Figure 7 EE 4G Time on Bearer: Bristol % % 51.7% 47.1% 35.3% 22.9% 1.2% 1.3% 2G 3G LTE Bearer Type November 23rd Results December 12th Results December 20th Results Birmingham also showed a much higher Time on Bearer 4G (77.6%) for the city centre testing in Test Period 3. Figure 8 EE 4G Time on Bearer: Birmingham % 50.1% 49.6% 21.4% 0.3% 1. 2G 3G LTE Bearer Type December 12th Results December 20th Results Summary of Time on Bearer Testing: Time on Bearer (4G) was shown to vary with the drive test route, ranging from a low average of 47% for urban/outskirts routes to a peak average of 77.6% for city centre. All three cities showed a significantly higher Time on Bearer 4G percentage in city centres. 11

12 Percentage Percentage EE 4G Network Performance Analysis THROUGHPUT SPEED Throughput speed is a measure of the rate at which data can be delivered by the mobile broadband service to the user s device. Higher bandwidth applications, such as video and audio applications, benefit significantly from faster throughput speeds. During the testing conducted in this study, throughput speed was measured from the drive test probes to an Epitiro server hosted in Telehouse. The server was managed and monitored by Epitiro throughout the testing and is attached to bandwidth dedicated to the purpose of throughput measurement. Figure 9 Distribution of 4G Download Speeds: Cardiff 5 44% 1 35% 25% 22% 18% 19% 15% 16% 16% 17% 16% 12% 1 1 7% 5% 5% 4% 3% 2% Download Speed (Mbps) November 23rd Results December 12th Results December 20th Results Figure 10 Distribution of 4G Download Speeds: Bristol 32% 31% 28% 29% 1 14% 16% 14% 11% 6% 7% 7% 7% 5% 4% 5% 3% Download Speed (Mbps) November 23rd Results December 12th Results December 20th Results 12

13 Percentage EE 4G Network Performance Analysis Figure 11 Distribution of 4G Download Speeds: Birmingham % 32% 22% 16% 18% 14% 16% 9% 9% 11% 2% 1% 2% > 50 Download Speed (Mbps) December 12th Results December 20th Results Findings: The concentrated city centre drive testing in Test Period 3 revealed a different distribution of reported speeds for Cardiff, Bristol and Birmingham compared to the previous test periods. In Cardiff, there were slightly more speeds reported in the Mbps band (16%) and Mbps band (12%) compared to previous testing. In Bristol, fewer speeds were reported in the Mbps and Mbps bands, whilst significantly more speeds were reported in the lower Mbps band. Birmingham, by contrast, shows a significant shift in distribution of speeds from the previous testing with 51% of speeds in speed bands higher than 20 Mbps, and 2% of speeds greater than 50 Mbps. The maximum speed for Birmingham (52.4Mbps) was also the highest reported from the three test periods. Average speeds for Test Period 3 were Cardiff (19.0 Mbps), Bristol (15.8 Mbps) and Birmingham (20.3 Mbps). Summary of Throughput Speeds: Whilst the routes driven appear to affect the speeds measured, there has been a consistency in delivery of the high speeds promised with 4G. Clearly the network has been adjusted and improved since initial testing began. 13

14 Percentage Percentage EE 4G Network Performance Analysis LATENCY The latency or delay in the network is measured as the time taken to deliver a single packet of data across that network. The latency measures in this analysis are network round-trip times, and indicate the time for data to be transmitted from the probe, to the targeted server and back again. Low latency is critical for an acceptable user experience of real-time applications such as voice and gaming. It is also a key factor in the experience for web browsing where the delivery of a complete web page involves multiple requests from the user s browser. Figure 12 Latency Distribution: Cardiff % 66% 58% 31% 6% 1% 1% 1% 1% < > 200 Network RTT (ms) November 23rd Results December 12th Results December 20th Results Figure 13 Latency Distribution: Bristol % 75% 69% 27% 23% 24% 1% 1% 2% 1% < > 200 Network RTT (ms) November 23rd Results December 12th Results December 20th Results 14

15 Percentage EE 4G Network Performance Analysis Figure 14 Latency Distribution: Birmingham % 19% 9% 1% 1% 1% < > 200 Network RTT (ms) December 12th Results December 20th Results Findings: Average latency times were similar in Test Period 3 to previous testing with 99% of times less than 60ms. Cardiff city centre latency times had more in the higher 40-60ms band than in Test Period (31% vs. 6%). Bristol showed very similar results to previous testing whilst Birmingham showed 9 of latency measurements in the city centre were less than 40ms. Summary of Latency Times Latency times measured were consistently lower than the 3G times typically reported. On average, 99% of latency times are less than 60ms across the 3 cities. 15

16 Percentage EE 4G Network Performance Analysis AREAS OF SPECIAL INVESTIGATION Analysis of data has led to areas of special investigation, including investigations into CS Fallback and Return times and Call Completion Success Rates to understand how the deployment of LTE may impact the network capacity and performance of voice calls. INVESTIGATION #1 - CS FALLBACK AND RETURN Where EE have deployed their 4G network, the LTE bearer is utilised for data services, but CS fallback is employed to support voice services. The CS fallback mechanism will drop the LTE bearer, and revert to a 2G or 3G bearer whenever a voice call is made or received. This approach aims to ensure a reliable voice service is maintained across established CS infrastructure, but does result in the loss of the LTE data service during voice calls and a delay in the return to an available LTE bearer upon call completion. A delayed return to LTE reduces the user s access to the faster 4G service with data services limited to 2G or 3G services until the return to LTE is completed. Figure 15 CS Fallback Return Time Distribution % 43.9% 55.4% % % 17.9% 10.6% 10.1% 7.6% 9.6% 7.2% 6.1% 2.4% 0. < > 50 Return Time (ms) November 23rd Results December 12th Results December 20th Results Findings: The chart above shows the distribution of return times experienced following voice calls during testing. The return time is defined as the duration (in seconds) between the voice call termination and the return to an LTE bearer. Those calls where the return to LTE took longer than 60 seconds are excluded from this analysis. In the first Test Period, the average CS Fallback return time was measured as 29.9 seconds, and the minimum return time was 18.0 seconds. The distribution shows that in over 57% of voice calls, the return to LTE took over 30 seconds. In second Test Period, the observed average performance showed little variation from the first Test Period. The average CS Fallback return time was measured as 31.6 seconds, and the minimum return time was 7.0 seconds. The distribution shows that in 65% of voice calls, the return to LTE took over 30 seconds. Approximately 9.6% of calls were over 50 seconds. 16

17 The results from the third Test Period show an increase in the average CS Fallback return time, measured as 44.0 seconds. The majority of voice calls (97.6%) took longer than 30 seconds to return to an LTE bearer. INVESTIGATION #2 COMPARISON OF OTHER 3G VS. EE 3G CALL SUCCESS As EE has used existing spectrum for its 4G rollout, questions over its effect on the 3G voice capacity and Call Completion Success Ratio have been investigated. Epitiro configured two Samsung Galaxy S2s to simultaneously test a Competing 3G network and T- Mobile 3G network in a city centre area of Cardiff with a strong 4G presence. Over 4,900 voice calls were tested over a period of 3 days. Test calls were required to stay connected for 20 seconds in order to be deemed a successfully completed call. Figure 16 3G Call Completion Success Ratio: Competing Network vs. EE The Call Completion Success Ratios measured for the Competing network and EE are shown in Figure 16. During the limited testing conducted, no notable difference in call completion success was observed. These results suggest that the capacity of the EE network to carry voice calls has not been significantly impacted by the reuse of spectrum for 4G deployment. INVESTIGATION #3 COMPARISON OF IPHONE AND ANDROID 4G BEHAVIOUR Some anecdotal information was observed as part of comparative testing between Android (Samsung Galaxy S3) and iphone. Comparative speed testing between iphone v S3 LTE speeds showed no noticeable difference. iphone CS Fallback return time is typically seconds which is similar that measured on Android. When watching a video, the video pauses when a call comes in, the same as Android. The arrival of an SMS message whilst the phone is on 4G and does not force the phone onto another bearer. 17

18 EE 4G AVERAGE SUMMARY TABLE The table below lists the average measurements based on the data collected to date. Figure 17 Summary Table Test Period #1: Nov 23 rd Results Test Period #2: Dec 12 th Results Test Period #3: Dec 20 th Results Average Max Average Max Average Max Throughput Speed Cardiff Throughput Speed Bristol Throughput Speed Birmingham 12.6 Mbps 48.4 Mbps 14.7 Mbps 41.8 Mbps 19.0 Mbps 31.9 Mbps 13.2 Mbps 21.4 Mbps 18.1 Mbps 41.4 Mbps 15.8 Mbps 44.4 Mbps N/A N/A 14.8 Mbps 21.7 Mbps 20.3 Mbps 52.4 Mbps Average Min Average Min Average Min Latency Cardiff 41 ms 26 ms 33 ms 24 ms 42 ms 26 ms Latency Bristol 42 ms 27 ms 35 ms 23 ms 37 ms 23 ms Latency Birmingham N/A N/A 36 ms 22 ms 31 ms 22 ms 4G 3G 4G 3G 4G 3G Time on Bearer Cardiff Time on Bearer Bristol Time on Bearer Birmingham 52.7% 47.2% 50.7% 49.1% 65.4% 27.8% 62% 35% 51.7% 47.1% 75.8% 22.9% N/A N/A 49.6% 50.1% 77.6% 21.4% Average Min Average Min Average Min CS Fallback Return Times Cardiff CS Fallback Return Times Bristol CS Fallback Return Times Birmingham 29.9 s 18 s 29 s 14 s 44 s 21 s N/A N/A 29 s 8 s 44 s 21 s N/A N/A 35 s 7 s 45 s 37 s 18

19 CONCLUSIONS AND FUTURE WORK This report of EE 4G mobile broadband performance levels in Cardiff, Bristol and Birmingham is based on data collected from October 31st and analysed on November 23rd, December 12th and December 20th. The 4G test data collected consists of the following sample sizes; Throughput Speed - 1,101 samples Latency - 9,661 samples RF (Time one 4G Bearer) - 201,853 samples 3G Call Completion Success Ratio - 4,943 samples Routes The routes driven were altered to view the EE 4G network from urban, outskirts and city centres. Results varied with the routes driven. 4G Coverage Epitiro found EE 4G availability in the areas shown on their predictive coverage maps with no significant gaps in the routes tested. The testing did not seek to conclusively prove EE 4G coverage was available everywhere indicated on their coverage map, however. AREAS OF SPECIAL INVESTIGATION Competing 3G/EE 3G Call Success No noticeable difference in dropped calls was observed in a comparison of 3G phones from a leading competitor and EE, in a city centre location that provided 4G coverage. Other manual testing and anecdotal observations were executed in the areas of call set-up and video behaviour in the presence of a voice call. FUTURE TESTING Presently the uptake of 4G services and therefore the effect of traffic loads is not known. For this reason Epitiro will continue to monitor the performance of the EE LTE 4G network for the foreseeable future. Time on 4G Bearer The average percentage time on a 4G Bearer was 6, based on all data collected. Throughput Speed The average throughput speed from all data collected was 15.8Mbps. The fastest speed recorded was 52.4 Mbps, in Birmingham. Latency The average latency across all data collected was 39.4ms. Birmingham twice posted the fastest latency at 22ms. CS Fallback The average CS Fallback return time based on all data collected was 37 seconds. 19

20 ipq Performance Monitoring Solution The ipq Performance Monitoring Solution measures service quality directly from smartphones and instantly delivers QoE insight to your tablet or laptop - in real-time. The ipq Test App can be readily downloaded to Android or ios smartphones and scheduled to run automatically for drive testing and network monitoring, or operate by push-totest for turn-up and troubleshooting scenarios. Measurements are forwarded to a central database, ready for real-time viewing on the web-based ipq Analytics Dashboard. Using any common web browser, stakeholders across your company can immediately see 4G/3G coverage, throughput speeds and other KPIs from a single smartphone or a virtually limitless scale of smartphone probes. Whether it's your own 4G LTE network launch or analysis of competitor networks, the ipq Performance Monitoring Solution is ready to measure the QoE delivered to customers, with all the KPIs, graphs and charts you'd expect from solutions costing much, much more. ipq Performance Monitoring Solution Download the app ipq Test! See results now 20

21 Epitiro Group Ltd Epitiro House, 10/11 Raleigh Walk, Waterfront 2000, Brigantine Place Cardiff, UK CF10 4LN Tel: +44 (0)