Measuring Web Browsing Quality of Experience: Requirements for Gaining Meaningful Insight

Transcription

1 Measuring Web Browsing Quality of Experience: Requirements for Gaining Meaningful Insight An Industry Whitepaper Contents Executive Summary... 1 Introduction to Web Browsing QoE... 2 Designing a Solution... 3 The Key Measurement: Page Load Wait Time. 3 The Cost of Slow Page Loads... 4 Contributing Factors... 4 Web Page Considerations... 6 Understanding a Web Page... 6 Choosing Representative Web Pages... 7 Measurements and Metrics... 8 Identifying a Page Load... 8 Obtaining Accurate Load Times... 8 Isolating QoE Factors... 9 Making the Metric... 9 A Proposed Solution... 9 Conclusions Summary of Solution Requirements Additional Resources Invitation to Provide Feedback Executive Summary The ability to monitor web browsing QoE is essential to determining when and where network conditions are contributing to an impaired customer experience. Understanding the relationship between web browsing QoE and network factors helps communications service providers (CSPs) to identify problems, to understand causes and contributing factors, and to evaluate potential solutions. For example, web browsing QoE reporting can help a mobile operator decide whether to increase a cell s transmit power to improve signal strength, or decrease it to reduce handovers from overlapping cells. Additionally, decisions related to CDN or cache placement, peering agreements, and transit contracts cannot be made blindly, and confidence increases greatly with in-depth QoE assessments of various subscriber activities including web browsing. There have been recent attempts to approximate web browsing QoE by collecting and correlating data associated user behaviors. However, these approaches fall short for a number of reasons, and have only been considered as proxies for a real metric because the most important measurement of all page load wait time has been difficult for anyone, let alone CSPs, to obtain across the network and for each individual subscriber. In this paper, we explore the challenges of monitoring web browsing QoE and propose requirements for those who wish to produce solutions in this space. We show that by meeting a relatively small list of specific requirements, a real-world solution can deliver meaningful insight into web browsing quality of experience. Version 1.1

2 Introduction to Web Browsing QoE Since the earliest days of its widespread adoption, Internet web browsing has been the activity most prominently and consistently linked to subscriber quality of experience (QoE). 1 It was the appearance of the first graphical web browsers that brought the Internet out of academia and into the public consciousness, and ushered in today s consumer Internet. Graphical web browsing made the Internet an indispensible part of our daily lives, and subscribers have always been extremely sensitive to poor page load times because, unlike money, products and many services, time is universally viewed as something that cannot be replaced. 2 The ability to monitor web browsing QoE is therefore essential to determining when and where network conditions are contributing to an impaired customer experience. Understanding the relationship between web browsing QoE and network factors helps communications service providers (CSPs) to identify problems, to understand causes and contributing factors, and to evaluate potential solutions. For example, web browsing QoE reporting can help a mobile operator decide whether to increase a cell s transmit power to improve signal strength, or decrease it to reduce handovers from overlapping cells. 3 Additionally, decisions related to CDN or cache placement, peering agreements, and transit contracts cannot be made blindly, and confidence increases greatly with in-depth QoE assessments of various subscriber activities including web browsing. The ideal outcome to the exercise of measuring a subscriber s web browsing quality of experience is a single, meaningful score, in the same vein as mean opinion scores (MOS) for voice-over-ip or video 4 streaming. However, this seemingly simple target betrays enormous complexity. Even just a little bit of examination runs into a daunting list of questions: What factors do subscribers implicitly consider when grading a web browsing experience? For which of those factors is the network responsible? How can those factors be measured? Can they be measured in real-time? How are those factors turned into a final, single metric/score? Web pages are not the same: some are small and simple, while others are enormous and complex. How can a single metric account for this reality? When is a web page done loading (e.g., when it s readable? When all GETs are completed?)? Web pages change frequently; can the measurement mechanism keep up? Which web pages should be considered? All of them? Just the most popular? What about the most important? We re barely a minute into the investigation, and it s already obvious that this clear goal to measure the customer experience for web browsing is not going to be straightforward. In this paper, we will explore in detail the questions above, and will propose requirements for those who wish to produce solutions in this space. We also hope that by doing so, CSPs will be able to make informed evaluations and decisions regarding the solutions in which they choose to invest. 1 One example of an early study from the late 1990s asks the question Tired of having to make coffee while waiting for a homepage to download? before making recommendations about how content providers can improve load times. 2 A 2012 IEEE study examines the human psychology of wait times related to subscriber QoE and the challenges associated with accurately measuring browsing QoE. 3 AT&T has released a study looking at web QoE challenges and metrics for mobile operators. 4 You can learn more about the subject of Video QoE in particular in Measuring Internet Video QoE from the Viewer s Perspective 2

3 Designing a Solution There have been recent attempts to approximate web browsing QoE by collecting and correlating data associated with user behaviors. For instance, measurements such as number of user clicks for a particular page, or how often subscribers abandon a page load, have been used to infer web browsing QoE as part of a general study (not a solution) specific to 3G networks with a reported accuracy of about 80%. 5 However, these approaches fall short for a number of reasons, and have only been considered as proxies for a real metric because the most important measurement of all page load wait time has been difficult for anyone, let alone CSPs, to obtain across the network and for each individual subscriber. 6 The Key Measurement: Page Load Wait Time As difficult as it may be to obtain, the time a subscriber must wait for a page to load is the central data point upon which all others depend in forming an objective judgement of web browsing QoE. A 2012 IEEE study notes that: when it comes to web browsing, it has been widely recognized that in contrast to the domains of audio and video quality, where psychoacoustic and psycho-visual phenomena are dominant, end-user waiting time is the key determinant of QoE: the longer users have to wait for the web page to arrive (or transactions to complete), the more dissatisfied they tend become with the service. 7 The goal of measuring web browsing QoE is therefore simplified by the fact that the only metric that matters to subscribers, and the key metric that opens the door to insight for CSPs, is how long it takes for a page to completely load. The same study notes that subscribers do not perceive web browsing as a sequence of single isolated page retrieval events (e.g., separate page loads or the more technical separate HTTP GET requests) but as an immersive flow experience, so the popular phrase surfing the web comes as no surprise. This flow state is characterized by positive emotions (enjoyment) and focused attention and as a result, heightened human performance and engagement. The notion of flow implies that the quality of the web browsing experience is determined by the timings of multiple page-view events that occur over a certain time frame during which the user interacts with a website and forms a quality judgment. This has a dual influence on the relationship between waiting times and QoE: on the one hand, flow experiences cause users to lose their sense of time, resulting in distorted time perception. On the other hand, a sudden instance of overly long waiting time(s) abruptly ends the pleasant flow state and thus tends to be perceived particularly negatively. 5 See section 1 of the AT&T study. 6 For instance, reporting systems that are built into network transport equipment do not provide the resolution required to measure page load time since only flow data up to Layer-3 can be typically obtained (see Section 1 of the AT&T study for more information). 7 See section 3.1 of the 2012 IEEE study examining the impact of web browsing wait times. 3

4 As such, accurately determining web browsing QoE requires a solution that can measure time intervals between all of the subscriber/server events that occur during page loads, and then call out wait times that clearly interrupt the flow of a particular web browsing session. Applications such as the Web Performance Group s Navigation Time 8 and Google s Page Speed 9 allow web developers to test designs against latency thresholds, but what exists for a CSP with millions of subscribers? The Cost of Slow Page Loads The most obvious impact of a slow web browsing experience is that it frustrates subscribers. As a direct result of poor web browsing QoE, dissatisfied customers are likely to call technical support and complain, and ultimately they may churn to a difference provider. The support cost to CSPs grows with the severity of the QoE degradation and the number of affected subscribers. Armed with accurate data about quality issues, CSPs can make confident CEM decisions while avoiding the cost of remediating unsatisfied subscribers. Content providers also have a cost to bear. Google and Amazon have both independently released research results showing that latency on their websites has a direct impact on sales. Google, upon introducing an added 0.5 seconds on page loads to deliver 30 results instead of 10, found revenue dropped 20% as a result 10. Similarly, Amazon conducted an experiment that showed with every 100ms increase in latency, sales dropped by 1%. It s clear that subscribers are affected by latency, and that it doesn t have to be particularly high to have a heavy impact on subscriber QoE. In other words, long wait times are bad for everyone there are no winners here. Contributing Factors There are many factors that can affect page load performance, several of which are shown in Table 1. Table 1 - Factors affecting page load performance Considerations Device Characteristics Web Page Characteristics Network Characteristics Factors Client device Access device Operating system type and version Browser type and version Browser plug-ins Web page size (i.e., bytes) Web page content types (e.g., text, images, audio, video) Web page complexity (e.g., number of separate HTTP GETs required, content styles) Web server load Access network congestion Peering and transit configurations CDN and cache placement 8 Specifications can be viewed here. 9 The developers page is located here. 10 Interested readers can find the original research in User Preference and Search Engine Latency 4

5 Signal strength (mobile) Session handovers 11 Device Characteristics Specific device types and their installed operating systems and browser configurations can vary widely in terms of performance across the access network, as shown by Figure 1. Figure 1 - Access network round trip time for 14 different mobile devices Web Page Characteristics To complicate matters, Figure 2 shows how web pages have evolved from serving relatively static objects, such as hypertext and images, to hosting rich media applications and third-party content such as advertising. In such cases, various parts of the page must be fetched from multiple domains and servers. As a simple example, subscribers would expect to load almost instantly, but the satisfaction tolerance increases for more complex pages with longer load times such as 11 According to AT&T, inter-radio-access-technology (IRAT) handovers are the largest contributor to poor browsing experience in mobile networks (section 5.1 of this AT&T study). 5

6 Figure 2 - Average number of objects fetched to load a web page ( ) 12 Despite the variability in terms of complexity, each web page has an optimum load time that can be measured objectively and used as a baseline to compare against deviations. Web Page Considerations It turns out that while it is easy to ask the question, how long did it take that web page to load? it is actually very challenging to provide a meaningful answer. To do so, a solution must address a number of nuances. Understanding a Web Page As noted earlier and illustrated in Figure 2, modern web pages are very complex. The subscriber s initial click in a search engine or entry in a browser s URL field sets off a flurry of component retrievals and network interactions that build the page. All web pages have a structural anatomy of object retrieval events associated with rendering the page in a browser. Figure 3 shows a partial view of the anatomy of object retrieval events that occur when a browser is building a complex web page requested by a subscriber. Only when all of these objects have been delivered is the web page truly complete. 12 This diagram is provided by WebSiteOptimization.com. 6

7 Figure 3 - Partial list of objects fetched and load times for a complex web page We can conclude that to ascertain when a web page has loaded, a solution must understand the structure of the web page. Complicating matters is that, of course, every web page is different. Therefore, a web browsing QoE solution must include an automated machine learning component by which it can learn how to read and understand a web page, as prerequisites for determining when that particular page has loaded. For each and every page to be monitored, a distinct anatomy profile must and stored. Furthermore, since pages can change without warning, the anatomy for each page must be updated frequently. By monitoring the many requests to each page, the solution can augment the profile with a set of statistics that indicate typical load times; then, the solution can determine when page loads are taking longer than expected by comparing individual page load events against a database profile of normal page load times. However, millions of pages are accessed by subscribers every day, and building anatomy profiles and subsequently assessing QoE for each of the long tail pages is not economically viable. This reality raises the question: which web pages get included in the QoE measurements? Choosing Representative Web Pages Broadly, there are two classes of web pages for which a CSP should carefully monitor the subscriber quality of experience: the most popular web pages, and the most important web pages. The full list of web pages that are monitored for QoE will then include an automatically determined list of the most popular web pages and those specified by the CSP. This approach provides the needed balance without sacrificing relevance or requiring an uneconomical infrastructure. 7

8 Popular Web Pages While on any given network subscribers will access enormous numbers of web pages, a relatively small number of pages account for the majority of visits. From this observation, we can determine that the QoE solution really only needs to be able to understand a relatively few (albeit, still at least one hundred) web pages in order to provide a meaningful web browsing QoE metric for the network as a whole. That is, even though some web pages wouldn t be considered in the assessment, enough would be included to provide an overall view. However, while there is considerable overlap between regions, the most popular pages vary by country, language, etc. This reality imposes another requirement on the hypothetical solution: the ability to learn the most popular web pages per region. In this manner, the solution could be deployed anywhere and would automatically determine which pages should receive the investment of being understood and examined for QoE. To maintain relevance as tastes and seasons change, the popularity list should be updated regularly (certainly no less than every week, and ideally on a daily basis). Important Web Pages Of course, popularity isn t the only determining factor; some pages are important for other reasons. For instance, CSPs might want to pay particular attention to their own portals, or to partnered content sites, or to new high-profile sites. To allow for this type of targeted attention, the solution should permit CSPs to manually define a list of pages that must be monitored. Measurements and Metrics The key component of a functional solution is an entity (e.g., an inline device with deep packet inspection capabilities) that can interpret real-time network data to classify Layer-7 HTTP flows and, most critically, includes the ability to measure time between events. 13 By comparing the timing of those events against those included in a particular web page s profile, the solution can ultimately determine when QoE is suffering. Identifying a Page Load The first step in determining page load times is to first determine a page load. On the surface, this might sound silly, but recall Figure 2 and Figure 3 pages today are incredibly complex, and each of those 100+ objects can generate its own HTTP GET. Any solution that purports to measure page loads must therefore be able to determine which GETs correspond to the first request of an actual page load and which can be temporarily ignored. Obtaining Accurate Load Times Assuming the solution can separate what is a new page load from all the requests that are parts of other pages, the next step is actually starting a clock and measuring a page load time. The data is available: tucked away in the headers of Layer-7 HTTP browsing information is a start and end indicator for each and every page load 14. CSPs must have the ability to measure the time 13 Another approach would be to inject JavaScript to measure and report page load times, but this approach is less than ideal for at least a few reasons. For instance: it is intrusive, it wouldn t work on all pages, and it could be blocked by browser plug-ins. 8

9 interval between the moment the page load event is initiated by the subscriber and the moment the page finishes loading (as determined by the page s anatomy profile). As a requirement, the solution s traffic classification engine must be able to associate every single one of the page load sub-events in the request/response interplay between subscriber and server with the overall event of loading the page. For scalability and optimum insight, it must also be able to classify flows by subscriber and not merely by IP or via aggregate clickstream analysis. Isolating QoE Factors A major motivating factor for CSPs to monitor web browsing QoE is to quickly identify, isolate, and remedy quality issues on the network to minimize the negative subscriber impact. To do so, they must be able to get more information than just the QoE for a web page load. Specifically, CSPs need to know when the problem is with something they can control, rather than the page itself. Furthermore, they want to know if there are common elements to poor (or great) QoE scores. For instance: Do certain device, OS, and browser combinations routinely show up in the list of bad experiences? Did the media optimization solution a CSP just deployed deliver a bump in QoE? Is there a variation of QoE across different logical or geographic segments of the network? Does web browsing QoE vary based upon the peering or transit link from which a page originated? Do certain subscribers have a consistently poor experience (a potential indicator of a problem with the home network)? It is essential, then, that in addition to the actual web page QoE score the solution also provides pertinent attributes associated with the traffic, and that these attributes can be explored in an ad hoc manner by the CSP. 15 Such comparisons reveal the final answers to the pressing question of, why are these subscribers having a poor web browsing experience? revealing insightful correlations and causes that lead directly to confident CEM actions that remedy a discovered issue. Making the Metric For each monitored web page, the solution aspires to produce a single number (e.g., like a MOS score) that captures the quality of experience from the end subscriber s perspective. Whether the score is out of five, 10, 100, or something else, the final critical piece is to make sure the score actually and accurately captures the end subscriber s perspective. To do so, the solution s output must be calibrated against actual subscriber experiences. A Proposed Solution Figure 4 shows how all of the key components come together to provide CSPs with daily insight into web browsing QoE as actually perceived by subscribers. 14 Such indicators are not available from encrypted browsing sessions. 15 Because you don t always know what you re looking for before you need to look for it. 9

10 Figure 4 - Key components of a web browsing QoE monitoring solution 10

11 Conclusions The ability to monitor web browsing QoE is essential to determining when and where network conditions are contributing to an impaired customer experience. Understanding the relationship between web browsing QoE and network factors helps CSPs to identify problems, to understand causes and contributing factors, and to evaluate potential solutions. Many studies have shown that the critical measurement that must be considered when building a web browsing QoE metric is page load wait time, but industry efforts have largely ignored this measurement because it has thus far been difficult to obtain in production environments. However, we have shown that by meeting a relatively small list of specific requirements (described in Table 2), a real-world solution can deliver meaningful insight into web browsing quality of experience. Summary of Solution Requirements The following table summarizes the minimum requirements for determining meaningful web browsing quality of experience from the subscriber perspective: Table 2 - Summary of solution requirements Requirement Ability to measure page load time, from initial request to load completion Machine learning to build and regularly refresh web page anatomy profiles Mechanism to automatically build and refresh a list of the Top N web pages Mechanism to allow a CSP to specify particular web pages Ability to determine when an HTTP GET Explanation Multiple studies have indicated that page load time is the single largest factor in determining a subscriber s perceived quality of experience for web browsing activities. Everything else is a proxy or a secondary concern. Web pages can be incredibly complex, consisting of 100+ unique objects; web pages change regularly; every web page is different. To build a statistical model of the usual page load for a particular page (and to allow subsequent comparisons), the solution must be able to understand the precise structure of each page being monitored. Pages are not uniformly popular, and it is not economical to build and store anatomy profiles for all (or even most) pages. Consequently, building and refreshing a list of the most popular pages lets a CSP concentrate efforts on the pages most likely to be included in a browsing session. However, popularity varies by region, language, etc., so the list must be produced automatically based on observed subscriber behavior. There are important pages that are important to a CSP but aren t necessarily the most popular ones with subscribers (e.g., the CSP s own portal, partner sites, emerging sites, etc.), so the solution should allow for CSPs to specify additional pages. The vast majority of GET requests do not 11

12 corresponds to a new page load Ability to provide a comprehensive set of attributes associated with each monitored page load A meaningful web browsing QoE metric correspond to the beginning of a page load; to only monitor page loads, then, the solution must be able to distinguish between actual page requests and everything else. In order to be useful to CSPs who want to identify, diagnose, and resolve quality issues, the QoE metric must include pertinent details that enable examination and insight. The score must correspond to actual subscriber opinions, and so must be calibrated against real-world experiences. Additional Resources In addition to the resources cited in the footnotes throughout this document, please consider reading the Sandvine technology showcase Web Browsing Quality of Experience Score, available at Invitation to Provide Feedback Thank you for taking the time to read this whitepaper. We hope that you found it useful, and that it contributed to a greater understanding of some of the challenges with and nuances of measuring web browsing quality of experience from the subscriber s perspective. If you have any feedback at all, then please get in touch with us at whitepapers@sandvine.com. 12

13 Headquarters Sandvine Incorporated ULC Waterloo, Ontario Canada Phone: European Offices Sandvine Limited Basingstoke, UK Phone: sales@sandvine.co.uk Copyright 2015 Sandvine Incorporated ULC. Sandvine and the Sandvine logo are registered trademarks of Sandvine Incorporated ULC. All rights reserved.