AWTG Mobile Benchmarking Service Executive Summary January 213 www.awtg.co.uk
Content Overview and Background Equipment Methodology Sample Results
Overview and Background AWTG has carried a pilot mobile service benchmarking. The benchmarking aims at multi dimensional and multi level analysis Scope of Work Benchmarking four major operators in the UK In a busy Christmas period and busy area of approximately.6 km 2 in London through more than 12 km walk test Fully synchronized testing using state of the art test equipment and dedicated servers For the following scenarios 3G voice and various data services Recently launched LTE (4G) service Outdoor and Indoor testing Services on new Boris bus and traditional London Double Decker bus Public WiFi performance analyses against LTE network Page 3
Hardware and Software Tools AWTG used SwissQual benchmarking tool called Ranger for this project and the related post processing tools also AWTG QoE and Broadband tester has been used to capture detailed log files especially for Wi Fi benchmarking furthermore AWTG s spectrum interference analyzer tool has been used for the more detail analyses. Quality of Experience (SwissQual) SwissQual Diversity Ranger 4x Galaxy S2 2x O2 SIM 2x T Mobile SIM 2x Three SIM 2x Vodafone SIM 4x Huawei Dongles E398 Quality of Experience (AWTG Tool) Post Processing SwissQual NQDI WireShark MapInfo Cascade Pilot Page 4
Methodology Three main test cases covered on this project which include indoor, outdoor and bus tests 2.4 km of Boris bus and traditional London Double Decker bus route testing 1 12 km (approximately) of pedestrian outdoor walk testing Multi story in building testing Page 5
3G Radio Summary Report (Outdoor Walk Test) Heat Maps & Statistics OP1 OP2 OP3 OP4 55 Average RSCP 6 dbm 65 61.98 7 75 68.16 Min= 127.26 Max= 21.45 SD= 37.33 Min= 17.56 Max= 15.82 SD= 26.4 68.78 Min= 15.55 Max= 38.21 SD= 26.57 7.12 Min= 12.75 Max= 26.38 SD= 38.92 OP2 has the best RSCP in the test area Page 6
3G Radio Summary Report (Outdoor Walk Test) Heat Maps & Statistics OP1 OP2 OP3 OP4 db 2 4 6 8 1 12 8.4 Min= 26.73 Max=.88 SD= 9.32 Average Ec/Io 8.99 Min= 26.76 Max= 1.39 SD= 8.22 7.91 Min= 31.192 Max=.112 SD= 7.37 9.81 Min= 26.8 Max=.51 SD= 11.4 OP3 has the best Ec/Io and the lowest SD Page 7
FTP (Outdoor Walk Test) Heat Maps & Statistics OP4 Failed FTP Upload Test Kbps 8 7 6 5 4 3 2 1 Average Uplink Throughput NA Min= 11.1 Max= 1496.8 735 Min= 11.1 Max= 2337 545 Min= 11.5 Max= 2526 482 FTP Upload Test Success Rate Kbps 3 25 2 15 1 5 OP4 FTP Uplink Test Max=2526.3 Min=11.5 Avg=482 Successful Test= 58 Failed Test= 47 1 59 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 11 15 8% 6% 4% 2% % 73% 57% 55% NA Low success rate and throughput value for FTP UL Page 8
FTP (Outdoor Walk Test) Heat Map & Statistics OP4 FTP Downlink Test 1% 8% 6% 4% 2% % Outdoor Walk Test FTP Downlink Success Rate 29% 99% 1% 99% Kbps FTP Downlink Average Throughput Min= 31.9 Max= 1834.4 Min= 235.3 4 3,663 Max= 616.4 MiMin= 7.7 3 2,68 Max= 1152.3 Min= 179.2 1,925 2 Max= 1294.8 1 737 Acceptable success rate and throughput value for FTP DL Page 9
3G Radio Summary Report (Outdoor Walk Test) Call Statistics 1% 98% 96% 94% 92% 9% 88% 86% 84% 1% Call Statistic Completed Call Rate % 99% 96% 9% Dropped 8% 6% 4% 2% % Failed Call Statistic Dropped & Failed Call % 7.3% 3.8% 2.4% 1.4% % % % % 1% 99% 98% 97% 96% 95% 94% 93% 1% Accessibility Voice 97% 96% 1% 1% 99% 98% 97% 96% 95% 1% Retainability Voice 97.8% 1% 1% Overall radio performance for OP2 is lower in comparison with other operators Page 1
Voice MOS (Walk Test) Heat Map & Statistics OP5 MOS Heat Map 5 4 3 2 1 Average MOS 3.37 3.23 3.41 3.45 2.82 OP5 8 6 5.28 Average Setup Time for calls completed 5.8 5.35 6.41 Sec 4 2 3.53 OP5 The overall MOS value for OP5 is higher than other operators however call setup time is higher Page 11
Handover Statistics Heat Map & Statistics Integrity Successful 2G Handover Integrity Successful 3G Handover 3G OP4 Handover 1% 1% 1% 98% 1% 1% 1% 1% 99.96% 95% 95% 9% 9% 85% 8% NA 85% 8% UMTS 21 GSM 18 UMTS 9 GSM 9 Serving Technology 1% 1% 8% 6% 51.6% 48.4% 47% 48.5% 27.5% 33.1% 4% 21.3% 2% % % 4.2% % % % 9.6% 8.9% % 3G OP4 is the only operator that has some failed handovers Page 12
Indoor Test Statistics 3G OP4 MOS Web Browsing Average Throughput Average Throughput(Kbps) 3 25 2 15 1 5 1,377 999 411.6 267 342 882 2,48 123.2 2,12 1,635 1,267 127.2 732 813.6 218 264 Average Success Rate 1% 8% 6% 4% 2% % 1% 1% 1% 1% 94% 93% 79% 73% The overall web experience for OP4 users is lower compared with other operators 1% 27% 87% 76% 93% 71% 1% 93% www.bbc.co.uk www.google.com www.msn.com www.yahoo.com Page 13
Bus Test Heat Maps 3G OP4 MOS Boris Bus 3G OP4 MOS Normal Bus 3G OP4 MOS Walk db 68 7 72 74 76 3G OP4 Average RSCP Walk Normal Bus Boris Bus 7.5 73.7 73.8 Sec 8 6 4 2 Boris Bus Average Setup Time for calls completed 6.6 5.32 4.61 3.82 5 4 3 2 1 Boris Bus Average MOS 3.34 3.29 3.43 3.34 No significant degradation between Normal bus and Boris bus Page 14
LTE OP Radio Report Heat Map & Statistics 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% LTE % 3G 8% 6% 4% 2% % Page 15 12% 1% -13--128-126--124-122--12-118--116-114--112-11--18-16--14-12--1-98--96-94--92-9--88-86--84-82--8-78--76-74--72-7--68-66--64-62--6-58--56-54--52 PDF -5--48-46--44 CDF LTE OP RSRP RSRP Distribution Ave =-91.7dbm Max= -65.9dbm Min=-121.4dbm PDF CDF [dbm] There was good LTE coverage in most of the covered area
LTE OP Radio Report Heat Map & Statistics LTE OP RSRQ RSRQ distribution Ave =-5.7dbm Max= -2dbm Min=-15.3dbm 3% PDF CDF 12% 25% 1% 2% 8% PDF 15% 6% CDF 1% 4% 5% 2% LTE % 3G -22--21-2--19-18--17-16--15-14--13-12--11-1--9-8--7-6--5-4--3 % [db] There was good LTE RSRQ in most of the covered area Page 16
LTE & 3G OPs FTP DL/UL (Walk Test) Statistics Kbps 1 8 6 4 2 NA FTP Downlink Average Throughput Min= 235.3 Max= 616.4 2,68 Min= 7.7 Max= 1152.3 1,925 Min= 31.9 Max= 1834.4 3,663 Min= 1199.7 Max= 24648.1 9,423 3G OP1 3G OP2 3G OP3 3G OP4 LTE 1% 8% 6% 4% 2% % FTP Downlink Success Rate 99% 1% 99% 1% NA 3G OP1 3G OP2 3G OP3 3G OP4 LTE Kbps 1 8 6 4 2 NA FTP Uplink Average Throughput Min= 11.1 Max= 1496.8 Min= 11.1 Max= 2337 Min= 11.5 Max= 2526 735 545 482 Min= 2438.1 Max= 193.7 8,825 3G OP1 3G OP2 3G OP3 3G OP4 LTE 1% 8% 6% 4% 2% % FTP Uplink Success Rate 1% 73% 57% 55% NA 3G OP1 3G OP2 3G OP3 3G OP4 LTE FTP DL/UL Throughput for LTE OP is much higher than all 3G OPs Page 17
LTE OP Radio Report Heat Maps LTE OP RSRP LTE OP Ping Test LTE OP FTP DL LTE 3G Failed ping tests and lower FTP throughput may have been caused by poor coverage Page 18
LTE OP and 3G OP4 Radio Report Heat Maps LTE OP RSRP 3G OP4 RSCP LTE 3G 3G OP4 has better coverage compared to LTE OP Page 19
LTE OP & 3G OP4 FTP Downlink (Walk Test) Heat Maps LTE OP FTP Downlink 3G OP4 FTP Downlink FTP Downlink Throughput for LTE OP is much higher than 3G OPs Page 2
LTE OP & 3G OP4 FTP Uplink (Walk Test) Heat Maps LTE OP FTP Uplink 3G OP4 FTP Uplink FTP Uplink for LTE OP is much higher than 3G OPs Page 21
LTE OP & 3G OP4 Web Throughput (Walk Test) Heat Maps LTE OP Web Throughput 3G OP4 Web Throughput Higher web throughput for LTE OP compared to 3G OPs Page 22
LTE & 3G Web Throughput Using Handset (Walk Test) Statistics LTE OP1 3G OP2 3G OP3 3G OP4 1 83.1 8 68.8 Web Throughput LTE OP1 3G OP2 3G OP3 3G OP4 8. 7.2 Time to Load Kbps 6 4 2 15.1 17.5 13. 7.7 9.4 11.2 NA NA 4.9 4.9 5.4 4.9 5.2 5.7 BBC Google MSN Yahoo Sec 6. 4. 2.. 4.5 3.1 2.4.9.4.3 5. 4.1 4.2 3.3 3.6 3.3 1.8 BBC Google MSN Yahoo 1% 95% 9% 85% 8% Web Browsing Success Rate 93.7% 98.8% 87.2% 95.3% LTE OP1 3G OP2 3G OP3 3G OP4 1% 98% 96% 94% 92% 9% 88% Delay Test Success Rate 99.6% 99.7% Min= 4 Max= 877 SD= 59 75 Web throughput for LTE OP is much higher than all 3G Ops, also LTE OP has the lowest delay 98.9% 92.3% LTE OP1 3G OP2 3G OP3 3G OP4 ms 6 5 4 3 2 1 Average Delay Min= 9 Min= 53 Max= 4943 Max= 4973 SD= 462 SD= 561 48 462 Min= 55 Max= 4918 SD= 552 43 LTE OP1 3G OP2 3G OP3 3G OP4 Page 23
LTE and Wi Fi Comparison Test (AWTG Broadband Tester) Static Test Test Locations Mbps LTE 12 1 8 6 4 2 O2 WiFi Outdoor WiFi FTP Uplink Throughput 1.9 1. 5.3 4. Location1 Location2 Mbps LTE 12 1 8 6 4 2 O2 WiFi Outdoor WiFi FTP Downlink Throughput 11. 6.7 6.1 5.4 Location1 Location2 Mbps LTE 4 3 2 1 O2 WiFi Outdoor WiFi Internet Speed Download 36.2 22.5 19.1 13.7 ms LTE 5 4 3 2 1 O2 WiFi Outdoor WiFi Delay 47.36 32.29 43.75 36.84 LTE 3% 2% 1% O2 WiFi Outdoor WiFi Packet Loss % 23.87% 16.12% 11.4% 26.87% Location1 Location2 Location1 Location2 % Location1 Location2 Overall FTP Downlink performance with LTE is lower than WiFi while the Upload Performance is much higher Page 24
LTE and Wi Fi Comparison Test (AWTG Broadband Tester) Static Test YouTube Session Error Analysis Test Locations LTE O2 WiFi Outdoor WiFi Average Number of Errors Location1 O2 WiFi Outdoor Error WiFi Location2 O2 WiFi Outdoor Error WiFi 2 15 189 125 172 5% 7% 5% 7% 1 71 13% 16% 5 Location1 Location2 75% 73% Time to Load YouTube Throughput Location1 LTE Error Location2 LTE Error Sec 7 6 5 4 3 2 1 63.4 64 45.18 45.8 Location1 Location2 Mbps 1.2 1.8.6.4.2 1.2 1.73.72 Location1 Location2 54% 16% 9% 2% 44% 31% 22% 4% Overall YouTube experience with oudoor WiFi is better than LTE Page 25
LTE and Wi Fi Comparison Test (AWTG Broadband Tester) Static Test LTE 1 O2 WiFi Outdoor WiFi Location1 Time to Load 9. 7.3 LTE 4 O2 WiFi Outdoor WiFi Location1 Web Througput 3.17 Sec 5 4.8 5.2 5.3 5.2 Mbps 2 1.73 2.35 1.93 1.14 1.89 BBC MSN Yahoo BBC MSN Yahoo Sec 1 8 6 4 2 4.5 Location2 Time to Load 9.2 6.9 6.4 4.3 7.6 Mbps 3 2 1 1.93 Location2 Web Throughput 2.4 2.33 2.13 1.24 1.6 BBC MSN Yahoo BBC MSN Yahoo Overall performance of LTE and WiFi technologies are comparable in both locations while time to load for LTE is better than WiFi Page 26
Thank You Registered in UK No. 5793772HQ in London UK Office: +44 () 28 799 368 Fax: +44 () 28 728 961 enquiries@awtg.co.uk