AWTG Mobile Benchmarking Service Executive Summary January

Transcription

1 AWTG Mobile Benchmarking Service Executive Summary January 213

2 Content Overview and Background Equipment Methodology Sample Results

3 Overview and Background AWTG has carried a pilot mobile service benchmarking. The benchmarking aims at multi dimensional and multi level analysis Scope of Work Benchmarking four major operators in the UK In a busy Christmas period and busy area of approximately.6 km 2 in London through more than 12 km walk test Fully synchronized testing using state of the art test equipment and dedicated servers For the following scenarios 3G voice and various data services Recently launched LTE (4G) service Outdoor and Indoor testing Services on new Boris bus and traditional London Double Decker bus Public WiFi performance analyses against LTE network Page 3

4 Hardware and Software Tools AWTG used SwissQual benchmarking tool called Ranger for this project and the related post processing tools also AWTG QoE and Broadband tester has been used to capture detailed log files especially for Wi Fi benchmarking furthermore AWTG s spectrum interference analyzer tool has been used for the more detail analyses. Quality of Experience (SwissQual) SwissQual Diversity Ranger 4x Galaxy S2 2x O2 SIM 2x T Mobile SIM 2x Three SIM 2x Vodafone SIM 4x Huawei Dongles E398 Quality of Experience (AWTG Tool) Post Processing SwissQual NQDI WireShark MapInfo Cascade Pilot Page 4

5 Methodology Three main test cases covered on this project which include indoor, outdoor and bus tests 2.4 km of Boris bus and traditional London Double Decker bus route testing 1 12 km (approximately) of pedestrian outdoor walk testing Multi story in building testing Page 5

6 3G Radio Summary Report (Outdoor Walk Test) Heat Maps & Statistics OP1 OP2 OP3 OP4 55 Average RSCP 6 dbm Min= Max= SD= Min= Max= SD= Min= Max= SD= Min= Max= SD= OP2 has the best RSCP in the test area Page 6

7 3G Radio Summary Report (Outdoor Walk Test) Heat Maps & Statistics OP1 OP2 OP3 OP4 db Min= Max=.88 SD= 9.32 Average Ec/Io 8.99 Min= Max= 1.39 SD= Min= Max=.112 SD= Min= 26.8 Max=.51 SD= 11.4 OP3 has the best Ec/Io and the lowest SD Page 7

8 FTP (Outdoor Walk Test) Heat Maps & Statistics OP4 Failed FTP Upload Test Kbps Average Uplink Throughput NA Min= 11.1 Max= Min= 11.1 Max= Min= 11.5 Max= FTP Upload Test Success Rate Kbps OP4 FTP Uplink Test Max= Min=11.5 Avg=482 Successful Test= 58 Failed Test= % 6% 4% 2% % 73% 57% 55% NA Low success rate and throughput value for FTP UL Page 8

9 FTP (Outdoor Walk Test) Heat Map & Statistics OP4 FTP Downlink Test 1% 8% 6% 4% 2% % Outdoor Walk Test FTP Downlink Success Rate 29% 99% 1% 99% Kbps FTP Downlink Average Throughput Min= 31.9 Max= Min= ,663 Max= MiMin= ,68 Max= Min= ,925 2 Max= Acceptable success rate and throughput value for FTP DL Page 9

10 3G Radio Summary Report (Outdoor Walk Test) Call Statistics 1% 98% 96% 94% 92% 9% 88% 86% 84% 1% Call Statistic Completed Call Rate % 99% 96% 9% Dropped 8% 6% 4% 2% % Failed Call Statistic Dropped & Failed Call % 7.3% 3.8% 2.4% 1.4% % % % % 1% 99% 98% 97% 96% 95% 94% 93% 1% Accessibility Voice 97% 96% 1% 1% 99% 98% 97% 96% 95% 1% Retainability Voice 97.8% 1% 1% Overall radio performance for OP2 is lower in comparison with other operators Page 1

11 Voice MOS (Walk Test) Heat Map & Statistics OP5 MOS Heat Map Average MOS OP Average Setup Time for calls completed Sec OP5 The overall MOS value for OP5 is higher than other operators however call setup time is higher Page 11

12 Handover Statistics Heat Map & Statistics Integrity Successful 2G Handover Integrity Successful 3G Handover 3G OP4 Handover 1% 1% 1% 98% 1% 1% 1% 1% 99.96% 95% 95% 9% 9% 85% 8% NA 85% 8% UMTS 21 GSM 18 UMTS 9 GSM 9 Serving Technology 1% 1% 8% 6% 51.6% 48.4% 47% 48.5% 27.5% 33.1% 4% 21.3% 2% % % 4.2% % % % 9.6% 8.9% % 3G OP4 is the only operator that has some failed handovers Page 12

13 Indoor Test Statistics 3G OP4 MOS Web Browsing Average Throughput Average Throughput(Kbps) , , ,12 1,635 1, Average Success Rate 1% 8% 6% 4% 2% % 1% 1% 1% 1% 94% 93% 79% 73% The overall web experience for OP4 users is lower compared with other operators 1% 27% 87% 76% 93% 71% 1% 93% Page 13

14 Bus Test Heat Maps 3G OP4 MOS Boris Bus 3G OP4 MOS Normal Bus 3G OP4 MOS Walk db G OP4 Average RSCP Walk Normal Bus Boris Bus Sec Boris Bus Average Setup Time for calls completed Boris Bus Average MOS No significant degradation between Normal bus and Boris bus Page 14

15 LTE OP Radio Report Heat Map & Statistics 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% LTE % 3G 8% 6% 4% 2% % Page 15 12% 1% PDF CDF LTE OP RSRP RSRP Distribution Ave =-91.7dbm Max= -65.9dbm Min=-121.4dbm PDF CDF [dbm] There was good LTE coverage in most of the covered area

16 LTE OP Radio Report Heat Map & Statistics LTE OP RSRQ RSRQ distribution Ave =-5.7dbm Max= -2dbm Min=-15.3dbm 3% PDF CDF 12% 25% 1% 2% 8% PDF 15% 6% CDF 1% 4% 5% 2% LTE % 3G % [db] There was good LTE RSRQ in most of the covered area Page 16

17 LTE & 3G OPs FTP DL/UL (Walk Test) Statistics Kbps NA FTP Downlink Average Throughput Min= Max= ,68 Min= 7.7 Max= ,925 Min= 31.9 Max= ,663 Min= Max= ,423 3G OP1 3G OP2 3G OP3 3G OP4 LTE 1% 8% 6% 4% 2% % FTP Downlink Success Rate 99% 1% 99% 1% NA 3G OP1 3G OP2 3G OP3 3G OP4 LTE Kbps NA FTP Uplink Average Throughput Min= 11.1 Max= Min= 11.1 Max= 2337 Min= 11.5 Max= Min= Max= ,825 3G OP1 3G OP2 3G OP3 3G OP4 LTE 1% 8% 6% 4% 2% % FTP Uplink Success Rate 1% 73% 57% 55% NA 3G OP1 3G OP2 3G OP3 3G OP4 LTE FTP DL/UL Throughput for LTE OP is much higher than all 3G OPs Page 17

18 LTE OP Radio Report Heat Maps LTE OP RSRP LTE OP Ping Test LTE OP FTP DL LTE 3G Failed ping tests and lower FTP throughput may have been caused by poor coverage Page 18

19 LTE OP and 3G OP4 Radio Report Heat Maps LTE OP RSRP 3G OP4 RSCP LTE 3G 3G OP4 has better coverage compared to LTE OP Page 19

20 LTE OP & 3G OP4 FTP Downlink (Walk Test) Heat Maps LTE OP FTP Downlink 3G OP4 FTP Downlink FTP Downlink Throughput for LTE OP is much higher than 3G OPs Page 2

21 LTE OP & 3G OP4 FTP Uplink (Walk Test) Heat Maps LTE OP FTP Uplink 3G OP4 FTP Uplink FTP Uplink for LTE OP is much higher than 3G OPs Page 21

22 LTE OP & 3G OP4 Web Throughput (Walk Test) Heat Maps LTE OP Web Throughput 3G OP4 Web Throughput Higher web throughput for LTE OP compared to 3G OPs Page 22

23 LTE & 3G Web Throughput Using Handset (Walk Test) Statistics LTE OP1 3G OP2 3G OP3 3G OP Web Throughput LTE OP1 3G OP2 3G OP3 3G OP Time to Load Kbps NA NA BBC Google MSN Yahoo Sec BBC Google MSN Yahoo 1% 95% 9% 85% 8% Web Browsing Success Rate 93.7% 98.8% 87.2% 95.3% LTE OP1 3G OP2 3G OP3 3G OP4 1% 98% 96% 94% 92% 9% 88% Delay Test Success Rate 99.6% 99.7% Min= 4 Max= 877 SD= Web throughput for LTE OP is much higher than all 3G Ops, also LTE OP has the lowest delay 98.9% 92.3% LTE OP1 3G OP2 3G OP3 3G OP4 ms Average Delay Min= 9 Min= 53 Max= 4943 Max= 4973 SD= 462 SD= Min= 55 Max= 4918 SD= LTE OP1 3G OP2 3G OP3 3G OP4 Page 23

24 LTE and Wi Fi Comparison Test (AWTG Broadband Tester) Static Test Test Locations Mbps LTE O2 WiFi Outdoor WiFi FTP Uplink Throughput Location1 Location2 Mbps LTE O2 WiFi Outdoor WiFi FTP Downlink Throughput Location1 Location2 Mbps LTE O2 WiFi Outdoor WiFi Internet Speed Download ms LTE O2 WiFi Outdoor WiFi Delay LTE 3% 2% 1% O2 WiFi Outdoor WiFi Packet Loss % 23.87% 16.12% 11.4% 26.87% Location1 Location2 Location1 Location2 % Location1 Location2 Overall FTP Downlink performance with LTE is lower than WiFi while the Upload Performance is much higher Page 24

25 LTE and Wi Fi Comparison Test (AWTG Broadband Tester) Static Test YouTube Session Error Analysis Test Locations LTE O2 WiFi Outdoor WiFi Average Number of Errors Location1 O2 WiFi Outdoor Error WiFi Location2 O2 WiFi Outdoor Error WiFi % 7% 5% 7% % 16% 5 Location1 Location2 75% 73% Time to Load YouTube Throughput Location1 LTE Error Location2 LTE Error Sec Location1 Location2 Mbps Location1 Location2 54% 16% 9% 2% 44% 31% 22% 4% Overall YouTube experience with oudoor WiFi is better than LTE Page 25

26 LTE and Wi Fi Comparison Test (AWTG Broadband Tester) Static Test LTE 1 O2 WiFi Outdoor WiFi Location1 Time to Load LTE 4 O2 WiFi Outdoor WiFi Location1 Web Througput 3.17 Sec Mbps BBC MSN Yahoo BBC MSN Yahoo Sec Location2 Time to Load Mbps Location2 Web Throughput BBC MSN Yahoo BBC MSN Yahoo Overall performance of LTE and WiFi technologies are comparable in both locations while time to load for LTE is better than WiFi Page 26

27 Thank You Registered in UK No HQ in London UK Office: +44 () Fax: +44 ()