SAMKNOWS ANALYSIS OF ROGERS BROADBAND PERFORMANCE IN MAY 2016

Transcription

1 ELECTRONIC COPY SAMKNOWS ANALYSIS OF ROGERS BROADBAND PERFORMANCE IN MAY 2016 Delivered by to: Gordon Reilly Rogers Dated: September 21 st, 2016 ELECTRONIC COPY

2 [THIS PAGE LEFT INTENTIONALLY BLANK] Confidential SamKnows 2

3 Contents A EXECUTIVE SUMMARY 5 B TESTING METHODOLOGY 6 B.1 Recruitment 6 B.2 Sample Composition 8 B.3 Test Configuration and Schedule 10 B.4 Data Analysis 11 C KEY PERFORMANCE INDICATORS 12 C.1 Download Throughput 12 C.2 Packet Loss 18 C.3 US providers vs Rogers comparison 19 Confidential SamKnows 3

4 About This Document This document is a summary report, which has been commissioned by Rogers and completed by SamKnows. For the purposes of this study 304 households across Rogers s customer base were given a specially configured hardware device (SamKnows Whitebox), which runs a series of purpose-built tests to measure Internet performance. The analysis in this report is carried out on data collected over the date period of 1 st May to 31 st May Any comments on the analysis in this document should be directed to Roxanne Robinson (roxanne@samknows.com). Confidential SamKnows 4

5 A Executive Summary This report presents the findings of a project commissioned by Rogers to study broadband performance across their network. This study was conducted during 1st May to 31st May 2016 using a sample of 304 fixed measurement devices ( Whiteboxes ) deployed to Rogers customers. SamKnows has been engaged with other regulators and broadband providers around the world and the methodology used has been developed to meet the needs of both regulators and operators to enable them to better understand and characterize their networks. Tests were performed in a similar fashion to the FCC s Measuring Broadband America project, gathering statistics on upload and download speeds. This report focuses on both the upload and the download tests specifically. These were run every other hour each day for a duration of 30 seconds. SamKnows created the panel using standard recruitment methodology. Potential volunteers were contacted via an campaign run by Rogers. Recipients of this were then directed to a SamKnows operated website where they could register for the project. Following specific stages SamKnows then selected the final panel of participants who could receive a Whitebox. The study found: - All Rogers current in-market packages exceed 100% of advertised download and upload speeds based on 24/7 results. - Speedboost pushes download speeds far beyond the advertised speed for the majority of packages based on 24/7 results, with lower speed plans receiving the strongest benefit. - Rogers delivers 115% of advertised download speeds during 24-hour periods, and 114% during peak hours. This performance is ahead of the respective averages of 110% and 105% for US cable operators. Confidential SamKnows 5

6 B Testing Methodology B.1 Recruitment For the purposes of this study, SamKnows sought to recruit 304 panellists using the following method: 1) Potential volunteers were contacted via an campaign run by Rogers. This was targeted at all their customers on current in-market service packages, which were representative of the overall subscriber base. Following this SamKnows assumed full control over volunteer selection. 2) Recipients of this were directed to a SamKnows-operated website to register for the project. This involved completing an online form, requesting information about which Rogers package they were subscribed to, and where they resided. 3) SamKnows reviewed the demographics of this initial panel to ensure a representative sample and to identify any deficiencies or shortfalls that would prevent SamKnows achieving the goals of the project. 4) A sub-set of the volunteers were then selected based on the statistical requirements of the panel. Selected volunteers were then asked to complete an acknowledgment of User Terms and Conditions that outlined the permissions to be granted by the volunteer in key areas such as privacy. 5) Of those volunteers that completed the User Terms and Conditions, SamKnows selected the final panel of participants, each of whom received a Whitebox for self-installation. SamKnows arranged for dispatch of the Whitebox, and SamKnows provided full support during the Whitebox installation phase. 6) Panellists were asked to install the Whitebox per the provided instructions and leave it connected for the duration of the study. Confidential SamKnows 6

7 NETWORK SCHEMATIC The installation of the SamKnows Whitebox is as follows: Option A: Option B: To home computers To home computers SamKnows Whitebox SamKnows Whitebox Rogers CABLE Modem Rogers CABLE Modem Confidential SamKnows 7

8 B.2 Sample Composition 304 Rogers customers were provided with a Whitebox, which they were instructed to leave powered on and connected to their Internet Modem/Gateway. 86 panellists were excluded for the following reasons: 1) The Whitebox hadn t run tests for at least 5 days in the month (due to it not being turned on, or the panellist was actively using the broadband connection continuously) 2) The panellist s service package could not be reliably determined 3) The results were highly erratic and/or intermittent 4) On a legacy Rogers speed package. 5) As this report focuses on new products, most of the excluded panellists belong to older products not suitable for the purpose of this study. These exclusions were made to ensure the integrity of the panel and ensure each speed package was statistically representative. As a result the report is based on 218 panellists. Confidential SamKnows 8

9 The table below depicts the breakdown of the sample: ISP Package (Download x Upload) Sample Size Rogers 30Mbps x 5 Mbps* 15 Rogers 60Mbps x 10Mbps 51 Rogers 100Mbps x 10Mbps 88 Rogers 250Mbps x 20Mbps 46 TOTAL 218 Table 1: Sample size split by Download Throughput service package SamKnows typically recommends a minimum of 40 samples per service package being reported upon, as this is what is considered statistically significant based on research carried out by regulators around the world. This is also the minimum sample size used by the FCC in their Measuring Broadband America program. The 30Mbps plan does not have a sufficient representative sample size to be statistically significant. However, it has been included in this report at the request of the client. Confidential SamKnows 9

10 B.3 Test Configuration and Schedule All measurements for download and upload were conducted on a series of off-net test nodes. The same metrics, test parameters and testing schedule as the FCC s Measuring Broadband America project were used. A detailed description of the methodology underpinning each metric can be found at Test-Suite-Whitepaper-D02.pdf Four servers were used in total and they were hosted at a data centre in Toronto. A summary of the speed related parameters and measurement frequency for each metric is detailed below: Test Schedule (Fixed) for Rogers Test Target(s) Frequency Duration Download speed 1 off-net test node Every other hour, 24/7 Fixed 30 seconds*** Upload speed 1 off-net test node Every other hour, 24x7 Fixed 30 seconds*** Latency/Loss 1 off-net test node Every hour, 24x7 Video Streaming 1 off-net test node Every hour, 24x7 Web Browsing speed 1 off-net test node Every hour, 24x7 DNS response time 1 off-net test node Every hour, 24x7 Jitter 1 off-net test node Every hour, 24x7 *** The speed tests run for a fixed duration of 30 seconds, but output the cumulative average transfer speed every 5 seconds (i.e. the first average is for seconds 0-5, the second is for 0-10, etc.). This allows us to measure performance over multiple durations, whilst running only one test. Below is also an outline of the number of tests carried out for each of the major metrics along with the volume of data used for these tests: Test Number of Tests Volume of Data Consumed Download speed 100, TB Upload speed 100, TB Latency/Loss 193, GB Web Browsing speed 588, TB Confidential SamKnows 10

11 B.4 Data Analysis A large raw dataset was collected over the period 1 st May to 31 st May The following process describes how the data was analysed to produce the charts and tables in this report: 1. Failed measurements and significant outliers were identified and removed from the raw dataset. A significant outlier is one where there has been a clear error. Failed measurements occur when a test fails (this could be due to the user being online when the test was trying to run or something similar). The FCC define the removal of any problem units as: removed measurements for any unit that exhibited greater than or equal to 10% failures in a particular one hour period (the purpose is to remove periods where units were unable to reach the Internet). 2. Panellists that changed service package intra-month were identified and data was removed for the service package that panellists spent the shortest duration on. This is in line with SamKnows standard global methodology. 3. Per-Whitebox averages were computed for the period of the day for each metric. 4. Per-service-package averages were computed from the per-whitebox averages calculated in step 3. Please note that the analysis is not weighted by Rogers s customer distribution. However, a sample plan was built to mitigate this based on the customer base. Confidential SamKnows 11

12 C Key Performance Indicators C.1 Download Throughput Sustained download throughput is a measure of the speed a broadband connection is able to deliver for sustained periods of time. This metric is the one most commonly used by regulators when reporting on speed. All of Rogers s packages performed above their advertised rates by a minimum of 8%, with packages up to 60Mbps averaging 108% of the baseline, and the 100Mbps package outperforming advertised speed by 22%. Sustained download speed as a percentage of advertised speed Throughput (%) 140% 120% 100% 80% 60% 40% 20% 0% 108% 108% 122% 117% 30x5 60x10 100x10 250x20 Package Figure a.1: Sustained Download Speed as a percentage of Advertised Speed, by package Confidential SamKnows 12

13 Burst download throughput is a measure of the short-term speed that can be achieved on broadband connections. This is commonly much faster than the sustained rate for cable service providers thanks to technologies such as SpeedBoost. The chart below shows the effects of SpeedBoost technology employed in Rogers s packages, all of which show varying degrees of improvement over sustained throughput. The 60Mbps package benefitted the most, increasing advertised download speeds from 108% to 153% - a 45% point difference with the 100Mbps package also showing a good boost from 122% to 153%. As expected from the SpeedBoost s use of spare bandwidth, the 250Mbps plan benefitted the least from burst speed, increasing by a comparatively limited 10% points from 117%. Burst download speed as a percentage of advertised speed Throughput (%) 180% 160% 140% 120% 100% 80% 60% 40% 20% 0% 153% 153% 139% 127% 30x5 60x10 100x10 250x20 Package Figure a.2: Burst Download Speed as a percentage of Advertised Speed, by package Confidential SamKnows 13

14 Peak periods of activity had nearly negligible effects on the majority of Rogers s packages, with the 30, 60, and 100Mbps plans showing, at most, a single percentage point decrease in download throughput compared to 24-hour measurements. The 250Mbps is the most affected during peak periods, losing 5% points, but remaining well above the advertised download speed. Peak vs 24hr sustained download speed as a percentage of advertised speed Throughput (%) 140% 120% 100% 80% 60% 40% 20% 0% 108% 108% 108% 107% 122% 121% 117% 112% 30x5 60x10 100x10 250x20 Package 24hr Peak Figure a.3: Peak vs Off-peak Sustained Download Speed as a percentage of Advertised Speed, by package Confidential SamKnows 14

15 Upload Throughput Sustained upload throughput is a measure of the speed a broadband connection is able to send for sustained periods of time. Upload throughput was above the advertised speed for all of Rogers s packages, with plans up to 60Mbps all delivering 7% above advertised speeds, and the 100Mbps outperforming by 13%. The 250Mbps plan had the lowest performance, but also delivered an upload throughput that was 5% above the baseline. Sustained upload speed as a percentage of advertised speed 120% 107% 107% 113% 105% 100% Throughput (%) 80% 60% 40% 20% 0% 30x5 60x10 100x10 250x20 Package Figure b.1: Sustained Upload Speed as a percentage of Advertised Speed, by package Confidential SamKnows 15

16 Burst upload throughput is a measure of the burst speeds attained during an upload measurement. Some service providers enable Boost technologies only in the downstream direction whilst others enable it in both directions (this is less common). Results from burst upload tests show limited improvements compared to sustained throughput. Gains for the 30 and 250Mbps were minimal, with a single percentage point increase for each. Burst upload speed as a percentage of advertised speed Throughput (%) 140% 120% 100% 80% 60% 40% 20% 0% 108% 110% 115% 106% 30x5 60x10 100x10 250x20 Package Figure b.2: Burst Upload Speed as a percentage of Advertised Speed, by package Peak hours had a limited effect on upload speeds, with the majority of packages exceeding advertised rates and seeing minor declines of upward to 1.5% points compared to the 24-hour baseline. Peak vs 24hr sustained upload speed as a percentage of advertised speed Throughput (%) 120% 100% 80% 60% 40% 20% 0% 107% 107% 107% 106% 113% 112% 105% 101% 30x5 60x10 100x10 250x20 Package 24hr Peak Figure b.3: Peak vs Off-peak Sustained Upload Speed as a percentage of Advertised Speed, by package Confidential SamKnows 16

17 Latency Latency is a measure of the time it takes for a single packet of data to reach its destination from its point of origin and back. It is also known as round-trip latency. Measurements for latency were comparable across packages, with a difference of 3ms between the 30Mbps plan s minimum of 14.6ms, and the 250Mbps plan s 18.8ms maximum. Overall, latency was good across all of Rogers s packages, delivering latencies that would have no impact on real-world user applications. Latency Latency (ms) x5 60x10 100x10 250x20 Package Figure c: Round-trip latency, in milliseconds, by package Confidential SamKnows 17

18 C.2 Packet Loss Packet loss is a measure of the percentage of data packages sent from the home to the destination that are lost during the transmission of data. Rogers s plans experienced a range of packet loss rates. It was low for the 30, 100, and 250Mbps plans, never reaching the 0.1% figure, but higher for the remaining plans. The 60Mbps plan stands out particularly, with a packet loss of nearly 0.22%, but nevertheless low enough to have no negative impact on the user s experience. Packet Loss 0.25% 0.218% Packet Loss (%) 0.20% 0.15% 0.10% 0.05% 0.051% 0.076% 0.096% 0.00% 30x5 60x10 100x10 250x20 Package Figure d: Packet loss, by package Confidential SamKnows 18

19 C.3 US providers vs Rogers comparison The chart below shows a comparison between the sustained download performance of Rogers to US ISPs. The US data is sourced from the FCC s Measuring Broadband America 2015 report. The measurement methodology used by the FCC and Rogers is directly comparable, although some small differences exist (such as the measurement servers in use). US broadband providers achieved an average 111% of advertised speed during the 24-hour measurement period, with Rogers attaining 115% during May 2016, thus exceeding average US operator performance by 4% points. Rogers s performance during peak hours was also more reliable, maintaining an average download throughput at 114% compared to advertised speeds a mere 1.7% point drop compared to the US operator average decrease of 6.7% Sustained download speed as a percentage of advertised speed 250% 218% 203% 200% Throughput (%) 150% 100% 50% 83% 81% 115% 111% 92% 87% 90% 81% 81% 77% 91% 81% 116% 112% 103% 98% 114% 110% 100% 97% 116% 110% 103% 98% 115% 114% 99% 95% 115% 110% 132% 107% 24hr Peak 0% AT&T - DSL AT&T - U-Verse CenturyLink Frontier DSL Verizon DSL Windstream Cablevision Charter Comcast Cox Mediacom TWC Rogers Frontier Fiber Verizon Fiber Hughes ViaSat/Exede DSL Cable Fiber Satellite ISP Figure e: Sustained Download Speed as a percentage of Advertised Speed comparison between Rogers and US providers, by ISP Confidential SamKnows 19

20 Rogers is currently delivering lower latencies than US cable operators, both as an average (17.44ms vs 20.62ms), and at the individual speed tier level. Whilst the difference is not as stark for the Mbps tier, Rogers s 30, and 60Mbps plans deliver latencies that are well below the US cable operator average. Speed Tier Rogers US Cable 30Mbps 14.82ms 25.25ms 50-60Mbps 16.22ms 22.36ms Mbps 18.44ms 18.87ms Mbps 18.77ms 22.36ms Table 2: Latency comparison between Rogers and US cable operators, in milliseconds, by speed tier Packet loss rates, however, see Rogers underperforming the US cable operator set both at average (0.118% vs 0.07%) as well as at the individual speed tier level. Whilst differences across most speed tiers are very small thousandths of percentage points for the Mbps tier, for example the difference is more notable in the 50-60Mbps tier, where Rogers s 60Mbps plan underperforms US cable operator average by nearly 0.12%. Speed Tier Rogers US Cable 30Mbps 0.111% 0.10% 50-60Mbps 0.218% 0.10% Mbps 0.076% 0.07% Mbps 0.096% 0.05% Table 3: Packet Loss comparison between Rogers and US cable operators, by speed tier Confidential SamKnows 20