Certification of Study Results for Telefónica O2. 11 th August Nationwide Mobile Service Benchmarking Study

Transcription

1 Certification of Study Results for Telefónica 11 th August 2014 Nationwide Mobile Service Benchmarking Study

2 Table of Contents Overview... 1 Certification Statement... 2 Results Summary... 3 Voice Service Reliability... 3 Data Service Speed... 4 Web Browsing Speed... 4 File Download Speed... 5 File Upload Speed... 6 File Download Burst Speed... 7 File Upload Burst Speed... 8 Data Service Reliability... 9 Web Browsing Reliability File Download Reliability File Upload Reliability File Download Burst Reliability File Upload Burst Reliability Methodology Overview About Spirent i Proprietary & Confidential

3 Overview Spirent Communications, an independent company with expertise in measuring and analyzing the user experience of mobile services, conducted a mobile service benchmarking study on the Vodafone, 3, and EE mobile networks between 6 th May and 17 th July 2014 on behalf of Telefónica UK Limited. The study methodology was defined by Telefónica UK. The British Approvals Board for Telecommunications (BABT) independently witnessed and audited the measurement collection process on behalf of Telefónica UK Limited (). For the study, mobile service benchmarking tests were conducted using Apple iphone 5c devices purchased over the counter directly from Apple retail outlets in the U.K. The devices were configured with the latest commercial version of Apple ios software as of May All tests in the study were performed using Spirent s User Experience Analytics systems, which are used by many operators and mobile device manufacturers worldwide. Spirent collected user experience metrics including voice call completion success, file transfer speed and web browsing speed across 20 major cities in England, Scotland and Wales representing the top 20 urban areas by population (as identified by Telefónica UK Limited). Measurements were performed in the following cities: Birmingham, Bradford, Brighton, Bristol, Cardiff, Coventry, Edinburgh, Glasgow, Hull, Leeds, Leicester, Liverpool, London, Manchester, Newcastle, Nottingham, Plymouth, Sheffield, Sunderland and Wakefield. During the study, Spirent captured over 20,000 minutes of measurements spread across the 20 cities. Minimum collection times/sample sizes per city were defined by Telefónica UK. Collection time was equally split between measuring the user experience of voice services (call completion success tests) and data services (file transfer and web browsing speed tests). Over 170,000 samples were collected for data service measurements and more than 47,000 samples were collected for voice service measurements across the four networks over the duration of the study. The study results and full details of the methodology were provided to Telefónica UK in the following document: Q2 July 2014 iphone5 Results.xls issued 28 th July 2014 This document provides a certification and summary of the study results and methodology. 1 Proprietary & Confidential

4 Certification Statement Spirent Communications conducted a mobile service benchmarking study on the Vodafone, 3, and EE mobile networks between 6 th May and 17 th July 2014 on behalf of Telefónica UK Limited. Spirent confirms the following: All testing in the study was performed consistent with the methodology defined by Telefónica UK Limited. The data and conclusions presented in the study report accurately reflect the results achieved from the testing. The principles of the testing and the test methodologies in the study are commensurate with commonly accepted industry practice. The study is part of a quarterly independent service benchmarking program commissioned by Telefónica and awarded via competitive tender. Use of Spirent s name or logo associated with any public claim related based on this certification note or related study must be agreed in writing with Spirent. Any public claim made by Telefónica UK or associated company will be the responsibility of Telefónica UK. 2 Proprietary & Confidential

5 Results Summary The study results and full details of the methodology were provided to Telefónica UK Limited in the following document: Q2 July 2014 iphone5 Results.xls issued 28 th July 2014 The following sections provided a summary of the study results and relevant conclusions supported by the data. Voice Service Reliability exhibited the second highest measured overall call completion success rate (98.85%). Across all operators, the highest call completion success rate was 99.01% and the lowest overall call completion rate was 98.78%. Note: The differences between the overall call completion rates of all 4 operators are not statistically meaningful as the 95% confidence intervals are overlapping. Call Completion Success Rate by Operator 100% 98.8% 98.8% 98.8% 99.0% Higher 95% 90% Lower Figure 1: Overall Call Completion Success Rate by Network. Error bars indicate 95% confidence interval. 3 Proprietary & Confidential

6 Time (s) SPIRENT Data Service Speed Web Browsing Speed exhibited the fastest average overall web browsing speed (2.7 seconds). The slowest average overall browsing speed measured across the 4 networks was 3.4 seconds. Web Page Load Time by Operator Slower Figure 2: Web Page Load Time by Operator Faster 4 Proprietary & Confidential

7 Mbps SPIRENT File Download Speed exhibited the 2 nd fastest measured average file download speeds (11.3 Mbps). The fastest measured average file download speed across all operators was 11.6 Mbps and the slowest was 6.4 Mbps. File Download Speed by Operator 15 Faster Figure 3: File Download Speed by Operator Slower 5 Proprietary & Confidential

8 Mbps SPIRENT File Upload Speed exhibited the fastest measured average file upload speeds (5.5 Mbps). The slowest average file upload speed measured across the 4 networks was 3.0 Mbps. File Upload Speed by Operator 10 Faster Figure 4: File Upload Speed by Operator Slower 6 Proprietary & Confidential

9 Mbps SPIRENT File Download Burst Speed exhibited the 2 nd fastest measured average file download speeds over the 15 second burst test (13.2 Mbps). The fastest measured average file download speed across all operators was 13.6 Mbps, the slowest 8.2 Mbps. File Download Burst Speed by Operator 20 Faster Figure 5: File Download Burst Speed by Operator Slower 7 Proprietary & Confidential

10 Mbps SPIRENT File Upload Burst Speed exhibited 2 nd fastest measured average file upload speeds over the 15 second burst test (9.7 Mbps). The fastest measured average file download speed across all operators was 11.9 Mbps and the slowest was 5.6 Mbps. File Upload Burst Speed by Operator 15 Faster Figure 6: File Upload Burst Speed by Operator Slower 8 Proprietary & Confidential

11 Data Service Reliability Based on aggregating the samples for web browsing and fixes file size download and upload tests, exhibited the 2 nd highest measured overall reliability (98.2%). The highest measured reliability across all operators was 98.6%, the lowest 96.8%. Overall Data Service Reliability by Operator 100% 98.2% 98.6% Higher 96.8% 96.8% 95% 90% Figure 7: Overall Data Service Reliability by Operator Lower 9 Proprietary & Confidential

12 Web Browsing Reliability exhibited the 2 nd highest measured overall browsing success rate (97.90%). The highest measured overall browsing success rate across all operators was 98.60% and the lowest was 96.4%. The difference between s results and all other networks was statistically meaningful i.e. the relative ranking of s result is expected to be repeatable with a 95% confidence. Web Browsing Reliability by Operator 100% 97.90% 98.60% Higher 96.40% 96.40% 95% 90% Lower Figure 8: Web Browsing Reliability by Operator. The error bars indicate 95% confidence interval Proprietary & Confidential

13 File Download Reliability exhibited the 2 nd highest measured overall HTTP File Download success rate (98.60%). The highest measured success rate across all operators was 99.50% and the lowest was 97.40%. The difference between s results and all other networks was statistically meaningful i.e. the relative ranking of s result is expected to be repeatable with a 95% confidence. File Download Reliability by Operator 100% 98.60% 97.70% 99.50% 97.40% Higher 95% 90% Lower Figure 9: File Download Reliability by Operator. Error bars indicate 95% confidence interval Proprietary & Confidential

14 File Upload Reliability exhibited the highest measured overall HTTP File Upload success rate (99.20%). The lowest measured success rate across all operators was 97.60%. The difference between s results and the other networks was statistically meaningful i.e. the relative ranking of s results to these networks is expected to be repeatable with a 95% confidence. File Upload Reliability by Operator 100% 99.20% 97.60% 98.10% 98.30% Higher 95% 90% Lower Figure 10: File Upload Reliability by Operator. The error bars indicate 95% confidence interval Proprietary & Confidential

15 File Download Burst Reliability exhibited the 2 nd highest measured overall HTTP File Download Burst Test success rate (97.90%). The highest measured overall browsing success rate across all operators was 99.00% and the lowest was 96.40%. The difference between s results and all other networks was not statistically meaningful. File Download Burst Reliability by Operator 100% 95% 97.90% 97.30% 99.00% 96.40% Higher 90% Lower Figure 11: File Download Burst Reliability by Operator. Error bars indicate 95% confidence interval Proprietary & Confidential

16 File Upload Burst Reliability exhibited the highest measured overall HTTP File Upload Burst Test success rate (99.40%). The lowest measured success rate across all operators was 98.80%. The difference between s results and all other networks was not statistically meaningful. File Upload Burst Reliability by Operator 100% 99.40% 98.90% 99.30% 98.80% Higher 95% 90% Lower Figure 12: File Upload Burst Reliability by Operator. The error bars indicate 95% confidence interval Proprietary & Confidential

17 Methodology Overview Vodafone and subscriptions were provided by Telefónica UK Limited. Subscriptions for other networks were purchased by Spirent directly from the operators. Subscriptions were provisioned and results constantly reviewed to ensure no data capping or throttling of speeds occurred. Spirent used never locked (SIM Free) new iphone 5c devices purchased over the counter directly from Apple retail outlets in the U.K. All SIMs purchased by Spirent as part of standard service plans from the operators, where an operator offered various quality of service tariffs the one offering the highest data speeds were used. All devices were left in default 4G preferred mode. Measurements were captured between the hours of 8 am to 11 pm (with samples distributed across the test period) on Monday - Friday. No testing was performed during UK public holidays. All testing (call sequencing, placing of data samples, recording of results) was automated using Spirent s User Experience Analytics systems: Nomad (for Voice Services) Datum (for Data Services) These systems require no modification of devices or firmware beyond the installation of the Datum software app on the device. Data testing was split between pedestrian (walk testing), static (stationary indoor and outdoor locations) and in-vehicle mobility scenarios. Voice testing was conducted for pedestrian and invehicle scenarios (testing inside a moving vehicle). All routes and locations were independently selected by Spirent Static locations were based on a cross-section of public places where it is likely that many mobile subscribers would want to use their mobile devices and where one or more networks could not be reasonably foreseen to have an advantage. Pedestrian routes were walk tests, within city centre boundaries. In-vehicle mobility routes included testing on major traffic routes within city boundaries. Data tests at each test location were carried out concurrently on each network. The minimum number of data and call samples for each city was defined by Telefónica UK Limited. Spirent tested data performance of file downloads (3 MB file size and 15 second burst test), file uploads (1 MB file size and 15 second burst test) and browsing across 5 widely used websites separated by a mix of 6 and 45 second wait periods. Approximately 60,000 file transfer samples and 118,000 web browsing samples were captured across the four networks over the duration of the study. Five live websites selected by Telefónica UK (BBC, Guardian, Ebay, Facebook and Yahoo) were tested Proprietary & Confidential

18 For fixed size file transfers timeout periods of 100 seconds (file transfers) and 30 seconds (web impressions) were used, samples exceeding these timeouts were classified as failures. For burst testing samples that didn t achieve an average throughput speed of 250 kbps for downloads or 80 kbps for uploads were recorded as failures and excluded from the network speed calculations Voice call performance measured the ability of mobile subscribers to receive and place calls to a UK landline number on the networks and retain them for total duration of 40 seconds from the time when the call was initiated. A wait time of 25 seconds minimum between completion of previous and initiation of next call was used to ensure sufficient time for devices to re-acquire the network. Over 23,000 mobile originated calls and 23,000 mobile terminated calls were placed across the four networks over the duration of the study. Mobile originated or terminated call placement was synchronized across all networks, the initiation of calls to/from the devices occurred simultaneously for each network. A quality audit of data was performed by Spirent field and office staff, typically within one working day of data being captured. The data collection methodology was audited by BABT with a lead assessor physically witnessing the collection of data on June 4 th 2014 in London. The statistical analysis performed on the results was defined by Telefónica UK Limited Proprietary & Confidential

19 About Spirent Spirent is the leading provider of services and systems for measuring and analyzing the user experience of mobile devices and services. Our driving vision is to: Define the methods and invent the technology for measuring and analyzing the user experience of mobile devices and services Provide the user experience analytics needed to make business-critical decisions about launching, marketing and improving new devices and services Become the industry s global trusted-partner for assuring and improving the user experience of devices and services We offer the following user experience analytics solutions: Fit4Launch Programs, Analytics Systems, and Insights. Fit4Launch Programs Fit4Launch is the leading program for mobile service providers and device manufacturers to identify user experience issues for new mobile devices before they go to market. Fit4Launch combines live network and lab-based test cases to capture real-world user experience and accelerate root cause analysis. Fit4Launch enables all new devices to be ranked and compared based on user experience categories including calling, speech, file transfer, web browsing, video and battery life. Low performing devices can be identified prior to launch, enabling service providers and device makers to work together to improve the user experience of mobile services. Analytics Systems For service providers and device makers who want to measure and analyze the user experience of devices and services, Spirent offers the same systems used to deliver our Fit4Launch programs. With our suite of user experience analytics systems, service providers and device makers can measure and analyze user experience on any device, operating system or network around the world. Our suite of User Experience Analytics systems includes: Nomad HD for Voice Services (VoLTE, HD Voice and more) Datum for Data Services and Multi-Service Usage Chromatic for Video Services Quantum for Battery Life 17 Proprietary & Confidential

20 Insights Insights provides the world s leading operators and device manufacturers with custom user experience studies to enable decisions about launching, marketing and improving new devices and services. Spirent tailors Insights projects to meet the client s specific objectives and executes testing independently to ensure maximum objectivity. Applications of Insights studies include: Launching new devices and services including LTE, VoLTE, HD-Voice, RCS and more Marketing new devices and services enabling marketing claims based on factual data Improving user experience identifying weak areas and opportunities for improvement For more information about Spirent s user experience analytics suite, please go to: Proprietary & Confidential

21 1325 Borregas Avenue Sunnyvale, CA USA Americas SPIRENT Europe and the Middle East +44 (0) Asia and the Pacific Spirent. All Rights Reserved. All of the company names and/or brand names and/or product names referred to in this document, in particular, the name Spirent and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in accordance with relevant national laws. All other registered trademarks or trademarks are the property of their respective owners. The information contained in this document is subject to change without notice and does not represent a commitment on the part of Spirent. The information in this document is believed to be accurate and reliable; however, Spirent assumes no responsibility or liability for any errors or inaccuracies that may appear in the document. Rev. A 06/14