Mobile Broadband Report:

Mobile Broadband Report: Quality of Delivery for Mobile Apps Q2 2015

2

Table of Contents Executive summary Introduction What influences mobile app performance? Mobile Broadband Throughput variation Throughput variation and TCP What do the throughput variation and packet loss results mean? Packet Loss Packet loss events Packet loss & goodput Network type Conclusion The Kwicr solution & methodology 3

4

Executive Summary Mobile app and content owners rely on mobile broadband, the amalgamation of various cellular and Wi-Fi infrastructure, to reach their customers. Numerous studies show a strong correlation between quality and user engagement. As the quality improves so does user engagement and with it the ability to monetize. We describe this as Quality of Delivery (QoD), or the speed at which a mobile app can reliably and consistently send and receive data. Unfortunately, everyone is attempting to deliver mobile apps using the same technology and approach as fixed broadband websites. Mobile broadband infrastructure is fundamentally different than fixed broadband infrastructure. Throughput and latency are the traditional measurements of network quality, but completely fail to describe mobile broadband network quality. The underlying transport layer technology of the Internet, TCP/IP, was not designed for mobile broadband. App owners are generally aware that issues like contention create challenges unique to mobile. However, large-scale hard data has not been available until now. Using the unique analytic capabilities of Kwicr s Mobile Delivery Network, we analyzed traffic from millions of users in 150 countries over a three-month period to compile data on mobile broadband performance (see methodology on page 21). This report documents two new and important findings. First, bandwidth is extremely variable across all mobile broadband access technologies and geographies. This variability occurs both within a session and across all sessions and renders the traditional throughput metric useless. For the first time, we are able to quantify this variability. Second, this variability is intertwined with high packet loss, and we were able to describe and characterize both packet loss events as well as packet loss rates for regions and countries. This report quantifies how mobile QoD is primarily impacted by packet loss and throughput variation and to a much lesser extent by throughput and latency. Global Highlights Brazil has 55% penetration of 3G/4G mobile broadband with 23% of all subscribers using smartphones. Overall, we measured cellular throughput in Brazil at 1.1 Mbps. While cellular packet loss was generally lower than Wi-Fi packet loss, Germany s packet loss rate of 10.2% was the highest of all of the countries we measured and higher than most countries Wi-Fi packet loss. French Wi-Fi networks averaged 3.83% packet loss and 155.2 loss events per application session. India is the second largest mobile market with more than 941 million subscriptions. 75% of subscribers are covered by 3G or 4G and 11% of all users currently use 3G/4G. Cellular packet loss was measured at 2.97%, which was lower than the Asian average of 5%. As you will see from the data, Kwicr is able to demonstrate for the first time that the coefficient of variation is key to understanding how network performance affects quality of delivery on mobile broadband. Sean Welch, Kwicr CEO 5

6

Introduction Mobile app users have little patience for spinning wheels, buffering, or other performance problems. Frustrated users abandon applications after just a few seconds many never to return. User abandonment represents a huge problem for app owners, but also an opportunity. If they can improve the performance of their apps, companies will reduce abandonment and increase ROI. Unfortunately, many companies don t have an accurate understanding of what causes performance problems on mobile broadband. While many use throughput as a proxy for app performance, the truth is more complicated. This report shines a light on the real causes of mobile app performance problems. It s more than just throughput. 7

What influences mobile app performance A mobile app s performance is dependent upon more than just throughput. The app type, device type, and content delivery network can all strongly influence performance. App type The app type dictates how the network is utilized. For instance, a streaming audio app typically has long-lived (over 30 minutes) sessions that consume a low volume of bandwidth (128-256 Kbps). In contrast, apps that stream video have varying session lengths depending upon the type of video content (i.e. longform versus shortform) being served to the app. Compared to streaming apps, social media apps are characterized by much higher volumes of upload traffic, which place different demands on the network than streaming apps. And compared to streaming and social apps, e-commerce apps utilize far less bandwidth, but have the potential to more negatively impact revenue under conditions of high packet loss. Mobile video is impacted by both packet loss and throughput variation. Both types of events can negatively impact TCP performance. This results in stalls and lower quality streaming, even if there is ample bandwidth to service the customer. Streaming audio is also impacted by packet loss and throughput variation. When either event occurs, less bandwidth becomes available, which can result in stalls or silence while the app rebuffers. E-commerce apps are impacted less by throughput variation since there is very little bandwidth utilized by the typical retail mobile app. Packet loss can definitely slow down the load times for the static objects that may be used by the mobile app, since TCP will reduce its throughput in the face of packet loss, even when bandwidth is plentiful. Device type The most obvious way in which the device type impacts app performance is that older devices utilize slower processors and have less available memory. Additionally, older devices may only support more crowded 2.4GHz Wi-Fi bands, or not be enabled with the fastest cellular technologies such as LTE and LTE Advanced. On the plus side, older devices often have lower resolution displays, which means that their thirst for data consumption in streaming video and social apps is less than their newer, higher resolution counterparts. Content Delivery Network A Content Delivery Network (CDN) should not be overlooked as an option for improving app performance, particularly for apps with cacheable content such as streaming video, streaming audio, and static web page content. With a CDN, app content is brought closer to the edge of the network, lowering latency, resulting in better TCP performance. TCP assumes less congestion and delivers better throughput when the app content is closer to the mobile app user. CDNs are good for serving not only streaming apps but also static objects that may get rendered by the HTML engine within the app. However, CDNs are not effective for dynamic content, such as uploading of photos and files. Mobile Broadband Network Conventional wisdom states that throughput and latency are the most important measures of how well a network performs. In reality, these metrics are not nearly as important as throughput variation and packet loss for understanding the QoD experienced by app users Even in a high throughput/ low latency network environment, small amounts of throughput variation and packet loss can result in lower quality video or audio delivery, video or audio stalls, and sluggish performance from social and e-commerce applications. Figure 1 video audio retail social live streaming 8

ios IP CDN ORIGIN SERVER Android CLOUD MOBILE BACKEND mobile broadband content Figure 2 Figure 2. The mobile app delivery architecture begins at the mobile backend and ends at the device. Depending on the device and the user s location, the app connects to the Internet through mobile broadband or Wi-Fi. Mobile Broadband Mobile devices are the visible, customer-facing part of the application infrastructure. Behind the app itself, though, there are many interconnected components involved in delivering the app experience, including the mobile network infrastructure, CDNs, the Internet, and the app s custom mobile backend. The mobile device connects to the Internet through mobile broadband, i.e. Wi-Fi, 2G, 3G, or 4G networks. Wi-Fi networks include private residential and enterprise Wi-Fi, public hotspots at shopping, dining and travel hubs, and outdoor public Wi-Fi. The Wi-Fi infrastructure connects to the Internet through a number of fixed broadband technologies ranging from DSL to HFC (cable) to FTTX (Fiber to the home / neighborhood) to metro Ethernet. If the device utilizes a cellular connection, it may connect at real world speeds up to 256 Kbps (2G), 3 Mbps (3G) and 5 Mbps+ (4G). These cellular networks consist of both the base stations required to communicate wirelessly with the mobile device, a backhaul network to deliver traffic to and from the mobile core, and a mobile core network that manages mobile device IP addressing and Internet connectivity. If required, the mobile application often makes use of one or more CDNs to accelerate delivery of streaming and static content for the reasons described earlier. And behind the CDNs, the mobile backend is used for user authentication, billing, and general orchestration of any cloud-based traffic to and from the app s users. Did you know? CDNs are a valuable part of the mobile application delivery infrastructure. Without a CDN, the delivery of content across the Internet will often be slowed by the fact that TCP will provide less bandwidth for the content due to a longer roundtrip time between the endpoint and the content. 9

Throughput Variation At Kwicr, we believe that mean throughput (measured in megabits per second) fails to characterize either network or app performance. In addition to mean throughput, the variation and change in throughput during an app session is required to provide useful insight into real world app performance. the throughput for every HTTP transaction, and perform three different calculations: mean throughput, standard deviation, and coefficient of variation. The reason why we perform three different calculcations for throughput is because the single throughput metric is an insufficient measure of network performance. Together, the Kwicr SDK in the app and the Kwicr cloud measure Brazil France Germany Hong Kong India Korea Russia 1.14 2.74 3.05 2.95 2.04 1.94 2.64 1.94 2.72 2.11 2.38 2.10 2.25 3.39 Cellular Wi-Fi What We Measure Throughput for the session, calculated as the mean of the throughputs for all the HTTP transactions within the app session. This includes throughput for multiple HTTP transactions that are occurring at the same time Standard deviation of the session, calculated from the throughputs of all the HTTP transactions of an app session Coefficient of Variation, which is the standard deviation divided by the throughput Singapore 3.55 4.33 US 1.97 3.17 Worldwide North America Central &South America EMEA 1.3 2.2 2.0 2.2 2.4 3.1 3.2 3.2 Cellular Wi-Fi Figure 3. Mean throughput is a starting point and allows us to compare different geographies and technologies. However, the reality is that many users will have a poor experience in spite of high throughput. To fully understand bandwidth, we must understand its variability and how that impacts the TCP protocol. Asia 2.3 2.8 Oceana 2.3 3.4 0 1 2 3 4 5 Figure 3 megabits per second 10

Brazil 1.40 1.45 Why standard deviation and CV matter France Germany Hong Kong India Korea Russia Singapore US Worldwide North America Central & South America EMEA Asia 1.67 1.64 1.52 1.45 1.54 1.68 1.81 1.89 1.28 1.34 1.24 1.33 1.72 1.81 1.48 1.54 1.5 1.6 1.5 1.6 1.5 1.5 1.5 1.6 1.6 1.6 Cellular Wi-Fi Cellular Wi-Fi The standard deviation describes the distribution of throughputs of the HTTP transactions in a given app session. Instead of comparing standard deviations amongst different populations (i.e. different app sessions), statisticians prefer using the Coefficient of Variation (CV), which normalizes the standard deviation against the throughput. This allows app sessions with different throughputs to be compared to each other. Across a variety of different mean throughputs, the CV indicates in a normalized fashion how much throughput will vary. Kwicr collects throughput data and calculates standard deviation and CV information for Wi-Fi networks and for 2G, 3G and 4G cellular networks of mobile service providers in more than 150 countries and territories. Figure 4 1.6 Oceana 1.8 0.0 0.4 0.8 1.2 1.6 2.0 Figure 4. When it comes to CV, anything above 1 is very volatile. The user experiences throughput variation as video stalls and lower quality for video streaming apps. For webpage downloads, the user experiences lower download speeds. The Bottom Line Throughput variability expressed by the standard deviation and CV matter because in the real world, throughput in mobile networks varies considerably. The number of subscribers, the burstiness of their activity, and the changing distance from the mobile endpoint to the base station or Wi-Fi access point all are significant contributors to changes in throughput that can and do occur during a single app session. 11

reaches peak bandwidth 1 2 slow start 3 4 6 7 5 Even slower, and not taking advantage of available bandwidth Figure 5 time Throughput Variation and TCP As stated earlier, TCP performs exceptionally well with wired networks, because packets are lost only when the network is congested. TCP is designed to work with wired networks and equitably split bandwidth across the applications that are utilizing the fixed sized links on these networks. With TCP, packet loss is expected to occur in response to congested traffic conditions. Unlike wired networks, mobile broadband is characterized by quickly varying available bandwidth, which is impacted by factors such as endpoint motion, weather, topographic feature interference (i.e. mountains and buildings), the active or passive coordination between multiple access points or base stations, and contention for airwaves. Due to the more rapidly changing bandwidth characteristics of wireless networks, TCP often underutilizes the available bandwidth of these networks. In the example above, for instance, TCP accurately discovers the available bandwidth on a wireless link (1), but then the available throughput on the link decreases (2). TCP compensates for this decrease in bandwidth by initiating a slow start to discover the new bandwidth of the link (3). However, as the network slows down even more, TCP overruns the available link bandwidth (4). Then TCP detects a lost packet due to the overrun, correctly concludes it is overrunning the network, and reacts by invoking slow start again (5) in an attempt to discover the link s actual throughput. However, during this process, TCP misses the fact that the link bandwidth begins to increase (6). Since TCP is still attempting to discover a previous speed of the link, significant available bandwidth goes underutilized (7) as TCP s calculated speed for the link is considerably slower than the actual speed at which traffic can be sent. This example illustrates what happens in the real world in mobile broadband networks. On the wireless last mile, endpoints are fighting with each other for airtime to receive and transmit wireless data. When more endpoints arrive and begin to utilize a base station or Wi-Fi access point, there is simply less bandwidth available for the endpoints that were already using that wireless resource. This dynamic movement of endpoints onto and off of wireless infrastructure results in the very high CV (greater than 1) seen across all Wi-Fi and cellular networks worldwide. These CV measurements confirm the hypothesis that wireless bandwidth is a dynamically moving target even within a single app session and that the use of throughput as a performance metric fails to characterize the real behavior of mobile broadband. 12

What do the throughput variation and packet loss results mean? Kwicr s benchmarking study is the first time that standard deviation and coefficient of variation have ever been measured in a worldwide survey of network performance. Our data indicates that the throughput available to a mobile app for download and upload varies greatly during a single app session. This variability occurs across all networks we have measured, both Wi-Fi and cellular, across all regions measured, and in all countries measured. Even for countries with high relative throughput, low relative latency, and low relative packet loss, there was high variation in throughput during a given app session. As a result, apps may perform poorly when utilizing the mobile broadband network even if the traditional metrics of network quality indicate that the apps should perform well. Standard deviation is a useful tool for understanding the variability in bandwidth. It measures the distribution of samples within a population. The first standard deviation covers a total of 68.2% of the samples. For instance, we could use radar to measure the speed of cars going past a particular spot during rush hour traffic between the hours of 4:30-6:30 p.m. We might find that the average speed is 45 mph, and the standard deviation is 20 mph. That would mean that 68.2% of the vehicles would be traveling between 25 mph and 65 mph during that time. Clearly, the person traveling at 25 mph was nowhere close to the average traffic speed of 45 mph! However, we have to be careful when dealing with standard deviations from multiple populations. Let s extend our traffic example to the traffic to compare the speeds of commercial airliners during the same period. For the airplanes, the mean speed might be 450 mph, and the standard deviation is still 20mph, meaning that 68.2% of the planes (the population covered by the standard deviation) would be traveling between 430 mph and 470 mph. Clearly, the 20 mph standard deviation for each time period fails to describe the far greater variability in speeds for the cars. 100 90 80 70 60 50 40 30 20 10 0 cellular CV for North America CV > 1 for 75% of sessions CV < 1 for 25% of sessions 0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5 coefficient of variation For this reason, in order to understand the variation in bandwidth, we should focus on the CV of the throughput in the region or country of interest. In North America, the mean CV for cellular traffic is 1.5, which means that the standard deviation was larger than the mean throughput across the population of apps run on North America. When the standard deviation is larger than the mean (i.e. the CV is greater than 1), it means there are many outliers in the population (the throughput measurements in this case), which indicates an extremely large variation in the throughput. If we put together a cumulative distribution function of the CVs for North American cellular data, then we can see that more than 75% of the sessions measured had CVs of greater than 1! This indicates a VERY wide variation in the throughput seen in a vast majority of individual app sessions. The best way to compare these two populations of vehicles speeds is with the coefficient of variation (CV), which is the standard deviation divided by the mean. Comparing the CVs shows a CV of.44 for the cars and a CV of.04 for planes, which now properly reflects the far greater variability in car speeds compared to airplane speeds. 13

throughput CDF LTE 100% 80% 60% 40% 20% 0% 0 1 2 3 4 5 6 megabits per second Figure 6 USA Singapore Let s compare: USA vs. Singapore The two biggest sources of data for LTE network performance within Kwicr s database are from mobile application sessions originating from the US and Singapore. When we compare the performance of LTE in these two networks using the traditional metric of throughput, it appears that Singapore has a better LTE network, since the measured throughput for Singapore LTE networks has a mean of 5Mbps and the mean for US LTE networks is 2.7Mbps. A closer look at the cumulative distribution function for LTE throughput provides additional insight into the LTE throughput that we have measured. Since the Singapore CDF is significantly lower than the US curve, it means there are more sessions experiencing higher throughput in Singapore. For instance, 56% of Singapore LTE sessions are 2Mbps or faster, and only 44% of US LTE sessions are 2Mbps or faster. coefficient of variation CDF LTE 30% 25% 20% 15% 10% 5% 0% 0% 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Figure 7 However, when we examine the CV for the two LTE networks, we gain more insight into the user experience. As explained earlier, a lower CV is better, since it means there is less variation and volatility in the throughput experienced during application session. A lower CV results in higher resolution viewing and fewer stalls for mobile video and faster page loads for social media and e-commerce. The mean CV for the US and Singapore are fairly close 1.7 for Singapore and 1.5 for the US. Again, for a more detailed understanding, we can look at the percentage of application sessions with a CV of less than 1 in both countries LTE networks. For Singapore, only 20% of app sessions have CVs below 1. In the US, 26% of app sessions have a CV of less than 1. Although the CV for LTE in both countries is quite high, it is noticeably better in the US, with a significantly higher number of low CV application sessions, as seen from the CV CDF below. 14

Brazil 3.53 3.26 Packet loss France Germany Hong Kong India Korea Russia Singapore US 0.89 1.99 0.74 1.02 1.52 0.69 1.42 3.83 2.92 2.97 3.83 3.60 3.57 5.98 7.49 10.20 8.07 8.24 Cellular Wi-Fi Packet loss is a key metric measured by Kwicr. For mobile networks, packet loss usually results from the inability of a wireless receiver to successfully receive a transmission from a wireless transmitter. For example, a dropped packet transmitted from a Wi-Fi access point towards a mobile endpoint causes packet loss. Since packet loss can occur during the wireless transmission process, Kwicr only records the volume of packet loss, not the underlying reason for the packet loss. Packet loss can result from a number of factors, including but not limited to environmental conditions, interference from other wireless devices, and network congestion. Worldwide North America Central & South America EMEA 2.0 3.5 3.7 3.1 3.6 3.9 3.7 4.9 Cellular Wi-Fi During the session, packet loss percentage is calculated as a percentage of the total number of packets sent and received. For example, a session might consist of 100,000 packets, with 85,000 downstream towards the endpoint, and 15,000 upstream towards the mobile backend. If 2,000 packets are lost, then the session would be said to have a 2% packet loss rate. Asia Oceana 3.3 5.0 Figure 8. While cellular packet loss 0.9 was generally lower than Wi-Fi packet 4.7 loss, Germany s packet loss rate of 10.2% was the highest of all of the countries we measured and higher 0 2 4 6 8 10 than 12most countries Wi-Fi packet loss. packet loss % Figure 8 The Bottom Line Asia was the region that had both the highest packet loss percentage and second highest throughput for cellular networks. This is clear evidence that throughput is an inadequate measure and proxy for cellular network performance. The packet loss percentage is very significant, since it has such a negative impact on TCP goodput. Central & South America and EMEA are near or at the bottom in Wi-Fi throughput at 2.2 Mbps and 2.8 Mbps, respectively. However, EMEA enjoys the second lowest packet loss percentage worldwide at 3.7%, while Central and South America suffer from the highest packet loss worldwide at 4.9%. So even though their throughputs are nearly identical, a significantly better user experience is delivered in EMEA on Wi-Fi compared to Central and South America. 15

Packet Loss Events As described earlier, the percentage of packet loss is the fraction of all packets sent or received that do not successfully arrive at their destination. A packet loss event, however, begins with the loss of a single packet, and then ends when a packet is successfully sent or received. There is no correlation between packet loss events and the percentage of packet loss. Consider the scenarios below, for instance, in which network A has two packet loss events, and loses 15 out of 30 packets for the session, for a 50% packet loss rate. Network B has five packet loss events, but loses only 5 out of 30 packets during these events, for a 16.6% packet loss rate. Even though Network B has more loss events, it has far fewer lost or dropped packets. A 15 lost packets out of 30: 50% packet loss, 2 loss events = lost packet B 5 lost packets out of 30: 16.6% packet loss, 4 loss events Figure 9 loss events: Wi-Fi Loss Events per Session Sessions Percentage 0 1-10 11-100 101-1,000 1,001-10,000 87933 47654 40838 21136 2148 44% 24% 20% 11% 1% Figure 10 loss events: cellular Loss Events per Session Sessions Percentage 0 1-10 11-100 101-1,000 1,001-10,000 103024 32294 14832 5854 735 65% 21% 10% 4% 0% Figure 10. It s clear that a substantial number of sessions experience significant impairment. The charts at left describe the number of loss events seen worldwide for Wi-Fi and cellular, respectively. Each pie chart represents a histogram distribution of the number of loss events seen for each wireless technology. For instance, for apps run on Wi-Fi, 87,933 app sessions had no packet loss events, and 47,654 app sessions had 1-10 loss events. Remember that each loss event represents one or more simultaneous lost packets, so this chart does not represent the percentage of lost packets. Lost packets can result in buffering or lower quality streaming for both video and audio streaming apps. For web sites and social media, lost packets can result in slower page load times. And for e-commerce, lost packets may cause some transactions to fail altogether. Depending upon the volume of packet loss, these impacts on apps may range from mild (i.e. slightly lower resolution video) to catastrophic (i.e. cannot complete an e-commerce transaction). 16

Brazil 28.7 136.6 France 137.2 155.2 Cellular Germany 45.3 96.0 Wi-Fi Hong Kong 84.8 205.0 India 37.9 218.4 Korea 19.7 83.7 Russia 45.7 110.7 Singapore 15.7 35.4 US 20.7 88.4 Worldwide 30.7 100.3 Cellular North America 17.5 77.6 Wi-Fi Central & South America 32.6 167.8 EMEA 37.2 87.2 Asia 53.9 168.4 Oceana 19.3 79.5 0 50 100 150 200 250 packet loss events Figure 11 Understanding the impact of packet loss requires a measurement of how often loss occurs. Loss events show how different regions and technologies compare in the frequency of this problem. 17

1 2 available bandwidth packet loss event 4 packet loss event slow start 3 slow start 5 Packet loss events result in network underutilization due to congestion control time Figure 12 Packet loss and goodput Automatically selecting the optimal network speed was a key design goal when TCP was designed four decades ago. TCP has a very good implementation for meeting this goal if you assume the network only loses packets when it is at capacity, i.e. congested. For wired networks with very reliable packet delivery, TCP is very good at adjusting the rate of transmission to dynamically match the available bandwidth in the wired network. However, with today s mobile broadband networks, whether they be Wi-Fi or licensed spectrum cellular, packet loss happens on a regular basis even when networks are operating below capacity. As a result, each packet loss event, even if it is just a single packet, causes TCP to significantly underutilize the available bandwidth on a wireless link. This happens because TCP reduces the bandwidth of the link as soon as it detects a lost packet, which for the wired network would have been a reliable indicator of congestion. Multiple packet loss events, even if they constitute a very small percentage of the overall packet count for a session, can cause a network to operate at speeds considerably below the capacity of the wireless link. In the scenario depicted in the chart above, the available bandwidth of the wireless link is fixed for the duration of the analysis. TCP is in the process of discovering the available bandwidth on the wireless link at (0), and discovers the available bandwidth on the link at (1). At (2), a packet loss event occurs, and TCP begins its slow start process (3) in order to determine the link bandwidth. Notice at this point, TCP goodput is significantly below the throughput available on the link. At (4), another packet loss event occurs, which causes TCP to begin another slow start (5), which will cause TCP to significantly underutilizing the available bandwidth on the link. Goodput is a measure of the actual throughput that is possible at the TCP layer, as opposed to the mere throughput possible at the lower wireless link and transmission layers. With TCP, goodput is usually significantly impacted by a reduction in lower layer throughput in the presence of even small amounts of packet loss. 18

Network type: Wi-Fi, 2G, 3G, and 4G networks Every Wi-Fi network, whether it is a single access point at a private residence or thousands of Wi-Fi access points deployed in an airport, utilizes free, unlicensed spectrum in the 2.4 GHz or 5 GHz frequencies. In addition to the wired broadband network capacity, the distance between the endpoint and the access point, and the number of individual users actively utilizing the Wi-Fi access point can all impact Wi-Fi performance. Depending upon the type of Wi-Fi being utilized, real world throughput can range from 2 Mbps to upwards of 100 Mbps. 2G and 3G networks are usually built using either the CDMA or GSM architecture. For each technology family, real world speeds range from 35-171 Kbps for 2G technologies such as CDMA 1xRTT and GPRS. 3G networks real world speeds range from 384 Kbps-3 Mbps. LTE networks are based on the GSM architecture and have real world throughputs of 5-10 Mbps. The definition of what constitutes 3G and 4G varies depending upon whether the question is asked of a standards body, a mobile network engineer, or a mobile service provider s marketing organization. Part of the confusion stems from the fact that some 3G technologies can deliver better throughput than 4G technologies under certain conditions. Instead of attempting to put an end to the debate, for the purposes of this study, Kwicr categorizes the different network technology types as 2G, 3G, and 4G for this study as well as subsequent studies according to the information in the table below. 19

average packet loss by network Wi-Fi 3.66 2G 3.24 3G 4.06 4G 2.79 Figure 13 0.0 0.9 1.8 2.7 3.6 4.5 packet loss % average loss events by network Wi-Fi 100.29 2G 30.20 3G 23.35 4G 34.67 Figure 14 0 22 44 66 88 110 packet loss events average throughput by network Wi-Fi 3.12 2G 0.75 3G 1.35 4G 3.06 Figure 15 0.0 0.7 1.4 2.1 2.8 3.5 megabits per second 20

coefficient of variation by network Wi-Fi 2G 3G 4G 1.58 1.43 1.52 1.51 0.0 0.4 0.8 1.2 1.6 2.0 Figure 16 latency by network Wi-Fi 172.8 The Bottom Line We have analyzed several performance metrics comparing the worldwide aggregate performance of Wi-Fi, 2G, 3G, and 4G wireless broadband networks. The results for packet loss percentage, throughput, and latency appear to match generally accepted understanding of less accurate packet scheduling and delivery for Wi-Fi, and increasing amounts of throughput and lower latency in the progression from 2G to 3G to 4G. However, two of the results are non-intuitive, namely higher packet loss event count for 4G networks and almost identical CV values for 2G, 3G, and 4G networks. The packet loss events for 4G may be higher due to the fact that most 4G networks today are deployed in higher band frequencies than 2G and 3G. More extensive analysis could be done on the data Kwicr has gathered on a per country and per service provider basis that could provide supporting arguments that would either confirm or disprove this hypothesis. 2G 1169.5 3G 405.9 High Variation on 4G 4G 180.5 0 240 480 720 960 1200 milliseconds Figure 17 It s noteable that the CV for 2G, 3G and 4G networks is practically the same. While 4G networks have undisputably higher throughput, they are still subject to the same variation in bandwidth (when normalized for throughput) as 2G and 3G networks. What this means is that the variation in bandwidth is not a problem that has been solved with 4G networks, and since the throughput of 4G networks is actually higher, there is even more absolute variability in throughput in 4G networks than 2G or 3G networks. As more and more people subscribe to 4G LTE networks, they will continue to experience problems with mobile data delivery. Even though CV on 4G is about the same as the CV on 3G, the absolute variability in bandwdith is higher for 4G LTE networks than it is for 2G and 3G networks. 21

Conclusion This Mobile Broadband Report is the first global survey of mobile broadband network performance to examine packet loss and throughput variation, in addition to traditional metrics like throughput and latency. Packet loss and throughput variation are important, because they directly affect Quality of Delivery, which is the speed at which mobile applications can reliably and consistantly send and receive data. QoD, in turn, has a direct impact on an app owner s ability to monetize, because without good QoD, users will abandon an app or cut short sessions. Our findings reflect expected differences in network performance, such as higher throughput for 4G networks. But we also saw unexpected results, such as very high variability in throughput for low latency, high throughput networks in the developed world. We believe this study will reshape the debate about how to deliver the best possible QoD for mobile apps. Going forward, any dialogue about QoD will necessarily consider the impact of packet loss and Coefficient of Variation, as well as the rich variation in these metrics across a population of app users in different countries, using different network technologies and mobile service providers at different times of day. Understanding how app QoD can vary under these different environments will empower app owners to improve the user experience and maximize revenue. 22

The Kwicr Solution Kwicr s Mobile Delivery Network (MDN) provides mobile app owners and developers with the acceleration, analytics and control they need to dramatically increase the performance of any mobile app, at any location and on any network. The Kwicr MDN consists of a software SDK that can be easily compiled into mobile apps. It works in conjunction with the Kwicr cloud, which sits next to the mobile back end for the app. Recall high throughput, low latency mobile broadband are necessary, but insufficient requirements for high Quality of Delivery (QoD). Kwicr s MDN can improve QoD under conditions of both packet loss and dynamically changing bandwidth, for both content that is downloaded to the mobile app as well as uploaded content such as photos and videos. Kwicr s MDN is designed to be complimentary to content that is accelerated by Content Delivery Networks, since the Kwicr MDN provide additional acceleration beyond the first level of acceleration offered by CDNs. In addition to these meeting acceleration requirements, Kwicr s MDN also offers customers: Analytics to gain insight into effectiveness Simple integration into both Android and ios apps Non-destructive acceleration (Kwicr does not change the data being accelerated. This applies to encrypted information as well, so even encrypted app content an be accelerated.) Non-collection of personally identifiable information (Kwicr never collects information that allows us to identify your individual customers.) Fair acceleration (All subscribers from the same app accessing a congested resource at the mobile backend will get fair acceleration of their traffic) There are two ingredients that imbue Kwicr s MDN with a unique ability to improve QoD fairly across multiple devices accessing the same set of resources. The first approach for improving QoD is a low overhead forward error correction (FEC) mechanism that protects against packet loss that would otherwise cripple TCP s ability to maximize mobile link utilization. Kwicr s FEC can recover lost packets, thereby preventing TCP from slowing down as a result of a packet loss. Unlike approaches that merely modify and optimize TCP, Kwicr s MDN technology is able deliver improved QoD for uploads, offer fair acceleration to multiple Kwicr-enabled mobile apps accessing the same bottlenecked-resource, and dynamically adjust to the changing bandwidth of the mobile broadband last mile. Methodology Kwicr measures network performance bidirectionally between an app running on an ios or Android mobile endpoint and the mobile backend associated with the app. By enabling their apps with the Kwicr SDK, both Kwicr and the app publishers are able to monitor the network performance from the endpoint s perspective. The network performance at the mobile backend is measured by the Kwicr Cloud, which is deployed in 11 PoPs across 5 continents and is within close physical and network proximity to the mobile backends supporting the Kwicr-enabled app. The Kwicr instrumentation architecture leverages the users of these commercially available apps as a worldwide-distributed team of testers who measure the real-world performance on Wi-Fi and licensed spectrum cellular networks. Measurements are performed on a per-app session basis and include throughput, standard deviation of throughput, coefficient of variation of throughput, packet loss percentage and packet loss events for each session. This benchmarking study encompasses measurements taken during a three month period from April 1, 2015 through June 30, 2015. More than 350,000 app sessions across 10 instrumented, publicly available commercial apps were measured on a variety of mobile broadband networks including Wi-Fi, 2G (CDMA 1xRTT, GPRS EDGE), 3G (CDMA EVDO rev 0/A/B, EHRPD, HSPA), and 4G (LTE and LTE Advanced). The second way in which Kwicr improves QoD is via a very accurate throughput estimation and flow control technique. By regularly and accurately measuring the true speed available on a mobile broadband link, the Kwicr MDN can maximize utilization of the mobile broadband last mile.