ENHANCING FACEBOOK APP EXPERIENCE

Transcription

1 ENHANCING FACEBOOK APP EXPERIENCE AN ANALYSIS OF NETWORK AND APPLICATION RESEARCH, TESTING AND OPTIMIZATION IN THAILAND In collaboration with:

2 contents Page 3. Executive summary Page 5. App experience is everything Page 6. Measuring quality engagement Page 8. Applying the methodology to the DTAC network Page 9. Putting the test into practice Page 12. Results Page 14. Conclusion Page 16. Glossary ENHANCING FACEBOOK APP EXPERIENCE CONTENTS 2

3 Executive summary In today s data-hungry world, a positive app experience on smartphones is central to user satisfaction. Greater smartphone use has led to a growth in data revenue, in many operators exceeding voice revenue. This makes developing a quality app experience important for developers, operators and users alike; yet measuring this experience within data-centric applications is no easy task and, if it is difficult to measure, how can it be improved? Networks have traditionally been tuned to improve voice services and large video/ data downloads using established key performance indicators (KPIs). As networks have matured, they have adapted to primarily deliver data services; but, the current network enhancement approach does not address mobile application performance, and data and network signaling needs are different. In particular, quantifying the success or failure of a user s app experience is much more complex than with traditional voice services which either successfully connects and maintains a call, or does not. DTAC, one of the leading network operators in Thailand, recognized the need to address and optimize user app experience. With the aim of becoming the number one digital brand in Thailand by 2020, DTAC approached Ericsson and Thailand s most popular social media platform, Facebook, to focus on how the Facebook user experience could be optimized on mobile devices. DTAC in focus As of Q4 2015, the company has 25 million mobile network subscribers in Thailand 15 million subscribers use smartphones and are active data users 2G, 3G and 4G services are available nationwide (as of Q4 2015) Facebook is the mostaccessed social media platform on the network ENHANCING FACEBOOK APP EXPERIENCE EXECUTIVE SUMMARY 3

4 Using a methodology developed by Ericsson and Facebook, typical Facebook interactions were triggered from Facebook test accounts and measured on different mobile test devices in central Bangkok; an area chosen for its very high traffic load, due to a high number of data subscribers. The partners correlated user-centric KPIs from the Facebook application with network performance KPIs, in a way that made it possible to understand the perceived user experience in various network conditions. After adjustments to the DTAC network, results showed improvements to the Facebook app experience during busy hour. Within that scope, time-to-content improved up to 54 percent and time-to-share up to 63 percent respectively. This means that Facebook users were not only able to get content from their friends faster, but could also share their experiences within a quicker time period. DTAC network adjustments After adjustments to the DTAC network Facebook users were able to get their friend s content and share experiences faster Many of the changes have systemic implications that can improve performance across the entire DTAC network, and for all Facebook mobile users. As a direct result of the optimization, the experience of using Facebook services for DTAC s users was significantly improved. This model can be replicated to monitor, analyze, measure and improve the app experience across any mobile network. Consequently, this will help operators cost-effectively target end-to-end network improvements for the greatest impact on user satisfaction, loyalty and retention. ENHANCING FACEBOOK APP EXPERIENCE EXECUTIVE SUMMARY 4

5 App experience is everything A user s experience with mobile apps, together with their ability to make voice calls, is the most important aspect to consider when determining the performance of a mobile network. There is a huge difference between how an operator should optimize a voice-only mobile network, designed fifteen years ago, compared to now, where there are many different types of data and services. A voice call is one service measuring and planning what targets to have is fairly simple but in today s data networks, with many services and needs, there are different and more complex requirements on the network. In today s data networks there are different and more complex requirements on the network 15 YEARS AGO TODAY To address this, Ericsson, Facebook and DTAC joined forces to optimize app experience, particularly on Facebook the most popular social network for DTAC subscribers. The first kick-off meeting was held in February 2015 in Bangkok, and the objectives of the project were aligned. In May, Ericsson started to measure the Facebook app experience on the DTAC network. This was carried out by using a test agent developed by Facebook, which was then deployed on different devices that triggered typical Facebook interactions. Hundreds of thousands of measurement samples were collected by the test agent based on the key metrics used to define app experience. Ericsson and DTAC also collected network metrics in the areas where the test agents were deployed. Using this data, engineers correlated both the network data and the end-to-end metrics to identify the key bottlenecks that impacted the Facebook app experience. Based on these findings, actions were then taken to remove these bottlenecks and this resulted in substantial improvements for the Facebook app experience. This highlights that it is possible to increase app coverage and improve app experience using existing network resources. ENHANCING FACEBOOK APP EXPERIENCE APP EXPERIENCE IS EVERYTHING 5

6 Measuring quality engagement The usability of an app can have a very direct effect on user satisfaction. If something goes wrong, subscribers are likely to blame the network operator even if there isn t a network issue. This, in turn, influences everything from brand reputation and customer loyalty, to the ability to generate sufficient return on investment. For both the content provider and the mobile operator, it is crucial to have an experience that is responsive and fast, and to have services that work well throughout the chain. Research shows that time-to-content is a crucial metric and a has direct correlation with satisfaction and loyalty to mobile operators. If it takes too long, it could tarnish the user s perception of the network coverage, causing stress, negative opinions and even complaints against the operator. Added to this, most apps (including WhatsApp, Instagram and YouTube) are encrypted; this means that information required to assess the quality of user experience is going directly to the app provider, and not necessarily to the operator. Apps, smart devices and mobile networks as well as the companies and developers that make them together form a single ecosystem that drives most of the growth. Within this ecosystem, apps and smart devices are continuously evolving and being upgraded, and this places high demands on the operator s network coverage, capacity and load. Heart rate increases 38% with mobile content delays Source: Ericsson Mobility Report MWC edition (February 2016) Data in the network originates from a wide spectrum of fast-evolving apps, both local and global, and this requires operators to make continuously updated decisions about service levels. At the other end, app developers face a similar uncertainty, as they must design their applications with a myriad of devices, platforms and network types as well as performance levels in mind. This can challenge application developers like Facebook, given that they often provide a visually-appealing experience with real-time content and large-scale photos. Such instances demand an end-to-end network view, as traditional network statistics alone may not reflect the actual user experience. It s tricky to understand if a user is happy or not, as millions of other processes are happening within the network simultaneously. Expanding app coverage is also central to helping billions of people get connected for the first time. ENHANCING FACEBOOK APP EXPERIENCE MEASURING QUALITY ENGAGEMENT 6

7 Yet only so much can be achieved under laboratory conditions. Measuring app experience effectively requires seeing how apps work on real devices, in real networks, and in a variety of real-time conditions. With this in mind, Facebook, DTAC and Ericsson agreed on a joint project to discover the real user app experience across the mobile subscriber base, as well as any associated network limitations, wherever and whenever these occurred. Internet.org Internet.org is a Facebook-led initiative with the goal of bringing internet access and the benefits of connectivity to the two-thirds of the world that do not have it. Both independently and as members of Internet.org, Facebook and Ericsson have made it a priority to help operators tune and optimize their networks. This covers improvements in app experience and more efficient data delivery, as well as providing scaled-down apps that work well in networkconstrained and price-sensitive environments. Examples of their common efforts include founding Internet.org with other partners in 2013, a related Innovation Lab launched in 2015, as well as a series of hackathons aimed at improving app performance in network conditions found in many emerging markets. ENHANCING FACEBOOK APP EXPERIENCE MEASURING QUALITY ENGAGEMENT 7

8 Applying the methodology to the DTAC network To better understand and ultimately optimize the Facebook experience in DTAC s live commercial network, a series of objectives, methodologies, measurements and subsequent improvements were introduced. Within the scope of the project, Ericsson and Facebook also aimed improve the quality of the user s app experience by quantifying network optimizations made. For this framework, the primary practical activities involved: Placing high-end and low-end devices across different geographic areas and throughout different times of the day to judge positive and negative user experiences Applying the methodology to monitor Facebook user KPIs, and then correlating this with network statistics to identify bottlenecks Analyzing and improving the Facebook experience in central Bangkok Starting in July 2015, Ericsson applied a series of metrics that considered the end-to-end network experience impacting app experience, or, in other words, the key measurable aspects that could go right or wrong in the network when using an app. The methodology framework Define KPIs, test cases and test tools Execute test cases Execute improvement plan Analyze improvements Preparations for measurements Analysis and definition of improvement plan Repeat test cases Prepare final report ENHANCING FACEBOOK APP EXPERIENCE APPLYING THE METHODOLOGY TO THE DTAC NETWORK 8

9 Putting the test into practice The methodology applied by Ericsson and Facebook 1. Defining Facebook service KPIs, including time-to-content, time-to-share and Facebook app experience 2. Preparing the measurement set-up, including test tools, network traces etc. 3. Executing the test case 4. Analyzing and defining an improvement plan, including analysis of end-to-end bottlenecks and proposed areas to optimize 5. Carrying out the improvement plan in a small area of the operator s network 6. Repeating test cases 7. Analyzing improvements and implementing them where possible Specifically for this project, the Android-based test agent, which had the same networking stack as the commercial Facebook app, was used to simulate app traffic. It triggered typical Facebook use cases (test cases) and reported user-centric KPIs for a better understanding of the Facebook user experience. The test agent also generated additional data that could be correlated with DTAC network statistics, in order to determine the link between user experience and network performance. Standard use cases included feed download, single and multi-picture download and single picture upload all actions that are performed on the commercial app hundreds of millions of times a day. The test agent was then loaded onto different types of smartphones high-end models and low-end ones. Both were powerful enough to eliminate device limitations as factors in the testing, while still enabling clear identification of any device-driven differences in user experience. Facebook test agent Agent Facebook app Autonomously emulates Facebook user activity 24/7 Real Facebook app QoE reporting functionality Facebook networking stack ENHANCING FACEBOOK APP EXPERIENCE PUTTING THE TEST INTO PRACTICE 9

10 PREPARATION Using the developed test agent, Ericsson and DTAC prepared the network by tracing in the radio access network (RAN) nodes, along with network probes for analysis of user plane behavior, among other actions. In addition, for each network measurement, the test agent data was tagged with a number of parameters including time, device and which cell it used. With two dozen high and low-end handsets placed in different strategic environments (such as loaded sites, less-loaded sites and attached to macro sites), Ericsson was then able to access this information and post-process it with data from the mobile network to spot patterns and instances. For example, within the eight locations, if the service was poor in one location but the others were OK, then it was noted that the problem was not with the Facebook server or the core network. Potential issues could be included or excluded from the study based on observing correlations. Problem is not with the Facebook server or the core network A number of different locations were chosen throughout the central Bangkok area, including sites close to the base station. Mobility tests involving physically driving the devices around Bangkok were also conducted to cover a larger geographical area. Each location was carefully considered with DTAC to maximize the effectiveness of the study on both 3G and 4G networks. ENHANCING FACEBOOK APP EXPERIENCE PUTTING THE TEST INTO PRACTICE 10

11 PERFORMANCE When the test agent showed a poor user experience, it was then possible to analyze the bad sample and identify where in the network the issues seemed to occur (such as in the RAN, core or servers) and then dig deeper based on that information. With this in place, Ericsson and Facebook were able to correlate user-centric KPIs reported by the test agent with network KPIs. This allowed them to develop an outside-in troubleshooting approach working from actual user experience, not just network data. The approach can be summarized as follows: Understand the user experience compare Facebook user performance data from different devices, geographical locations and time of day in order to understand positive and negative user experiences Drill down on negative samples analyze probe data to understand where in the network problems occur and analyze agent data to see which types of transactions go wrong; for example, transport layer security setup or object transfers Focus on the network elements detail statistics and configurations of network nodes and elements to pinpoint bottlenecks and issues Identify key bottlenecks define which key bottlenecks and improvement areas have the highest impact on user experience. Overall, the key focuses were the total time taken to download a feed, image or multiple images, and the time taken to upload an image. The analysis phase took a couple of months, but by November 2015 the team had a clear picture of what the issues were and what actions needed to be taken. After executing the network optimization activities, the same tests were carried out again to compare the before and after results. This highlighted that the project had improved user app experience significantly. ENHANCING FACEBOOK APP EXPERIENCE PUTTING THE TEST INTO PRACTICE 11

12 Results Results from the tests illustrated the impact of device selection and traffic patterns, as well as the correlation between Facebook experience and mobile network traffic load. System-wide bottlenecks and other observations in the RAN and Facebook servers were also identified percent improvement in app experience during busy hour 54 percent improvement in time-to-content 63 percent improvement in time-to-share 60 percent improvement in Facebook upload and download time There were some locations where app experiences were consistently poor because of network capacity, coverage and configuration constraints. For instance, if the time-to-content exceeded three seconds during busy hour, it was often because the RAN was working close to maximum capacity. ENHANCING FACEBOOK APP EXPERIENCE RESULTS 12

13 This was particularly the case on the 3G network and on the low-end handsets. By tuning the network more efficiently and maximizing resource on the uplink and downlink, user performance was improved across the radio network not just for Facebook users. After the adjustments were carried out, there were dramatic enhancements in performance on the Facebook test agent. For the worse 10 percent of test cases, time to content was lowered from 32 to 11.4 seconds seconds Seconds seconds 5 0 Average Worse 10 percent In order to have the greatest possible impact on user experience, many improvements were focused on fixing the worst performing scenarios. For instance, for the worst 10 percent of test cases, time-to-content was lowered from 32 seconds to 11.4 seconds. It s vital that the user has the same experience on their mobile as they would on their desktop via Wi-Fi, so improving the network operations helped deliver that goal. In fact, the success of the study meant that more Facebook users could be served from the same mobile network node, without having to build additional infrastructure and hardware. ENHANCING FACEBOOK APP EXPERIENCE RESULTS 13

14 Conclusion The Bangkok region has some of the most active mobile internet users and data subscribers in Southeast Asia. As such, the network is expected to deliver quality user experience particularly with the high penetration of smartphones. However, to date, there has not been an adequate methodology including KPIs, tools and correlation methods for measuring quality user app experience on data-centric applications; plus, networks in fast-growing emerging markets are often particularly challenged when it comes to data performance. Ericsson, Facebook and DTAC came together to analyze the real user experience and its relation to network capabilities. Facebook and Ericsson applied a unique, innovative methodology that helped to identify key issues and bottlenecks that affect user metrics; such as time-to-content within the Facebook app. The bottlenecks in the network were identified by correlating user-centric KPIs with network KPIs. This kind of intelligent correlation and end-to-end understanding of the network then made it possible to understand the user experience in various network conditions. Innovative methodology MOBILE NETWORK METRICS IMPROVED APP COVERAGE APP METRICS By addressing these issues, the user experience of the Facebook app in the DTAC network was improved substantially, with quality app experience during busy hour rising by 13 to 34 percent. Many of the changes have systemic implications that could improve performance across the entire DTAC network, and also for all Facebook s mobile users. ENHANCING FACEBOOK APP EXPERIENCE CONCLUSION 14

15 As part of Internet.org s mission to bring affordable internet access to the two-thirds of the world not yet connected, Facebook and Ericsson, in conjunction with DTAC, applied a methodology to correlate user experience with network performance in order to do this. DTAC now has the ability to focus network improvements on the areas that impact its users most, and this cost-effective approach to enhancing network performance positions the company as a data leader within its market. Facebook was able to better understand the many network variables that can impact user adoption, which is particularly critical as Facebook and Ericsson work to connect hundreds of millions, if not billions, of unconnected people to the internet. Finally, Ericsson has enhanced a model that can now be applied to any mobile network. This will help operators cost-effectively target network improvements in the areas that will have the most impact on user satisfaction, loyalty and retention. With up to a 60 percent improvement on both upload and download time, Facebook users on the DTAC network can now benefit from an optimized user experience. Up to a 60 percent improvement on both upload and download time ENHANCING FACEBOOK APP EXPERIENCE CONCLUSION 15

16 GLOSSARY App coverage App experience Bottleneck End-to-end RAN Time-to-content Time-to-share The geographical area in which an app works as the network operator intends The perception a user has when using an application on their mobile device A situation that causes delay in a process or system Including all stages of a process Radio access networks Time taken for content to load on a user s device Time taken for users to share content with other users ENHANCING FACEBOOK APP EXPERIENCE GLOSSARY 16

17 DTAC Established in 1989, DTAC is the second-largest mobile operator in Thailand. The company offers mobile services to its customers via contract and pre-paid tariff plans, with over 25 million subscribers registered as of Q Currently, dtac offers customers a comprehensive range of 2G, 3G and 4G services on the 850 MHz, 1800 MHz, and 2100 MHz bands. Since 2001, DTAC has been partly owned by the leading Norwegian telecommunications company, Telecom Group. ENHANCING FACEBOOK APP EXPERIENCE GLOSSARY 17

18 Facebook Facebook is one of the world s most popular social media platforms, with over 1.7 billion active monthly users and counting. It s widely used to connect with friends and family, share experiences and to discover what s happening around the world. Launched in 2004 by Mark Zuckerberg, Facebook was originally designed as a way for college students to connect with each other online. Today, it is a publicly-listed company with more than 14,000 employees. Facebook is available across the globe to anyone aged 13 years or older. Facebook Inc. has also acquired a number of other companies, including Instagram, Oculus and WhatsApp. ENHANCING FACEBOOK APP EXPERIENCE GLOSSARY 18

19 Ericsson Ericsson is the driving force behind the Networked Society a world leader in communications technology and services. Its long-term relationships with every major telecom operator in the world allow people, business and society to fulfill their potential and create a more sustainable future. The company s services, software and infrastructure especially in mobility, broadband and the cloud enable the telecom industry and other sectors to do better business, increase efficiency, improve the user experience and capture new opportunities. With approximately 115,000 professionals and customers in 180 countries, Ericsson combines global scale with technology and services leadership. Its support networks connect more than 2.5 billion subscribers; 40 percent of the world s mobile traffic is carried over Ericsson networks, and investments in research and development ensure that its solutions and customers stay in front. ENHANCING FACEBOOK APP EXPERIENCE GLOSSARY 19

20 Internet.org Internet.org is a Facebook-led initiative with the goal of bringing the internet to the two-thirds of the world s population that do not have it. This effort will require two key innovations: Bringing down the underlying costs of delivering data Using less data by building more efficient apps If the industry can achieve a tenfold improvement in each of these areas delivering data and building more efficient apps then it becomes economically reasonable to offer free basic services to those who cannot afford them, and to begin sustainable delivery on the promise of connectivity as a human right. To make this a reality, Internet.org and the rest of the industry, need to work together to drive efficiency gains across platforms, devices and operating systems. Creating more efficient technologies will speed up the rollout of more sophisticated technologies, which provide higher-quality experiences to more people in developing countries, while also enabling the industry to continue growing and investing in infrastructure development. Working together toward this common goal, the aim is to achieve shared learnings and advances that move the industry and society forward. Ericsson SE Stockholm, Sweden Telephone / FGC Ericsson AB 2016