Web Browsing Quality of Experience Score A Sandvine Technology Showcase Contents Executive Summary... 1 Introduction to Web QoE... 2 Sandvine s Web Browsing QoE Metric... 3 Maintaining a Web Page Library... 3 Popular Web Pages... 3 Other Important Pages... 3 Building Web Page Profiles... 3 Making Measurement in Real-Time... 4 Detecting a Page Load... 4 Measuring Page Load Times... 5 Extracting Attributes... 5 Producing a Meaningful QoE Metric... 7 The Complete Solution... 8 Conclusions... 9 Related Resources... 9 Invitation to Provide Feedback... 10 Executive Summary The ability to monitor the subscriber quality of experience for web browsing is essential to determining when and where network conditions are contributing to an impaired customer experience. Understanding the relationship between web browsing QoE and network factors helps communications service providers (CSPs) to identify problems, to understand causes and contributing factors, and to evaluate potential solutions. Sandvine s web browsing QoE solution allows CSPs to monitor, in detail, the web browsing experience of their subscribers. The solution builds a comprehensive library of web page anatomy profiles, each associated with a web page that is either popular with subscribers or otherwise designated as important by the operator. By combining these web anatomy profiles, with realtime measurements made as web browsing traffic flows through the network, the solution determines the time it takes to load each page. This page load time measurement is turned into a web browsing QoE score (on a scale up to five) that corresponds to subscribers real-world experiences. To maximize utility for the CSP, each web browsing QoE score is accompanied by more than 20 associated attributes that can be used to diagnose issues and to identify quality-related trends.
Introduction to Web QoE The ability to monitor the subscriber quality of experience for web browsing is essential to determining when and where network conditions are contributing to an impaired customer experience. Understanding the relationship between web browsing QoE and network factors helps communications service providers (CSPs) to identify problems, to understand causes and contributing factors, and to evaluate potential solutions. For example, web browsing QoE reporting can help a mobile operator decide whether to increase a cell s transmit power to improve signal strength, or decrease it to reduce handovers from overlapping cells. Additionally, decisions related to CDN or cache placement, peering agreements, and transit contracts cannot be made blindly, and confidence increases greatly with in-depth QoE assessments of various subscriber activities, including web browsing. There have been recent attempts to approximate web browsing QoE by collecting and correlating data associated with user behaviors; however, these approaches fall short for a number of reasons, and have only been considered as proxies for a real metric because the most important measurement of all page load wait time has been difficult for anyone, let alone CSPs, to obtain across the network and for each individual subscriber. The whitepaper Measuring Web Browsing Quality of Experience: Requirements for Gaining Meaningful Insight 1 explored the challenges associated with measuring the subscriber experience for web browsing and identified (and explained) a number of solution requirements, including: Ability to measure page load time, from initial request to load completion Machine learning to build and regularly refresh web page anatomy profiles Mechanism to automatically build and refresh a list of the Top N web pages Mechanism to allow a CSP to specify particular web pages Ability to determine when an HTTP GET corresponds to a new page load Ability to provide a comprehensive set of attributes associated with each monitored page load A meaningful web browsing QoE metric This paper presents Sandvine s web browsing quality of experience measurement solution, and explains how the solution overcomes technical challenges to meet all of the requirements. 1 Available at www.sandvine.com Page 2
Sandvine s Web Browsing QoE Metric The sections below focus on particular aspects of the overall solution. Maintaining a Web Page Library Broadly, there are two classes of web pages for which a CSP should carefully monitor the subscriber quality of experience: the most popular pages, and the most important pages. Combining these two lists provides needed balance without sacrificing relevance or requiring an uneconomical infrastructure. Popular Web Pages On any given network subscribers will access a long list of web pages, but a relatively small number of pages account for the majority of visits. Additionally, while there are some pages that will be globally popular, a significant proportion of the most popular pages from region to region will differ. To ensure accurate representation of the locally popular pages, the Sandvine solution observes subscriber traffic and empirically determines the most popular pages in the network (i.e., a Top N list). 2 This list is updated every day, so that even relatively quick variations in browsing activity are accounted for. Other Important Pages Popularity isn t the only factor that determines what pages should be monitored for QoE; some pages are important for other reasons. Sandvine s solution allows CSPs to manually define a list of pages that must be monitored. In this manner, CSPs can pay particular attention to their own portals, or to partnered content sites, or to new high-profile sites. Building Web Page Profiles Modern web pages are very complex (see Figure 1): the subscriber s initial click in a search engine or entry in a browser s URL field sets off a flurry of component retrievals and network interactions that build the page. To provide a meaningful QoE metric, a solution has to accurately measure the time to load a page; but to do that, it must understand the structure of the page. 3 Plus, every page is different, and page structures change frequently and without warning. To overcome these challenges, the Sandvine solution maintains a library of distinct web page profiles: at regular, configurable intervals 4, Sandvine s Service Delivery Engine (SDE) goes through the list of pages to be monitored, fetching and building each one with an embedded web browser; each page is parsed to create the anatomy profile of discrete component objects. These profiles are stored on the SDE. 2 By default, the Sandvine solution monitors the top 300 pages, but this number is configurable All web pages have a structural anatomy of object retrieval events associated with rendering the page in a browser. 4 The default rebuild of every pages happens once a day, overnight, but could be configured to run more frequently. 3 Page 3
Figure 1 - Partial list of objects fetched and load times for a complex web page Making Measurement in Real-Time Sandvine s PCEF/TDF device, the Policy Traffic Switch (PTS), intersects the network s data plane and makes detailed measurements in real-time. 5 These measurements are subsequently used to determine the page load time. Detecting a Page Load The first step in determining page load times is to first determine a page load event. This task is made complicated by the fact that a very small percentage of the observed page requests actually correspond to a new page being loaded (i.e., most are for elements of a page, rather than a page itself - recall Figure 1). To detect new page loads, the Policy Traffic Switch relies on a combination of heuristics determined through research and experimentation 6, including: GET requests Browser user-agents Resource lengths (very long URLs are unlikely to correspond to a top-level web page) Referrer fan-out File extensions (we can ignore file extensions that are unlikely to be a top-level page) Status codes HTML content type Content length 5 More information about this topic in particular is available in Internet Traffic Classification; specifically, see the section Traffic Measurements and Metrics 6 Since the web browsing QoE measurements are contained in a Policy Pack, it is easy to make updates, additions, and changes to these heuristics in the future as research reveals new techniques Page 4
Extensive testing indicates that, when used in combination, these factors are reliable indicators of a page load event. Measuring Page Load Times To measure page load time, the PTS relies on the start and end indicators in the Layer-7 HTTP headers. By monitoring these indicators and understanding how they correspond to the page s anatomy profile, the PTS can measure the time interval between the moment the page load event is initiated by the subscriber and the moment the page finishes loading. In short, the PTS measures the start and end for every fetch during the building of a web page, and these values are associated with the distinct objects contained in the anatomy profile. In this manner, the solution is able to determine when the page as a whole has finished being delivered and built. Figure 2 shows the significant difference in page load time between the three most frequently loaded web pages on a Canadian fixed access network. The most popular site (the yellowish line) averages about 4 seconds to load, while the other two load much more quickly. Additionally, the most popular site experienced performance degradation at 9pm precisely when the network is approaching peak utilization. The second most popular site (the charcoal line) exhibits variation in average load time during the evening, ranging from one second to as much as 5.5 seconds. Figure 2 - Network Demographics report: Web Browsing Page Load Time by Web Page Extracting Attributes A major motivating factor for CSPs to monitor web browsing QoE is to quickly identify, isolate, and remedy quality issues on the network to minimize the negative subscriber impact. To do so, they must be able to get more information than just the QoE for a web page load. Page 5
Specifically, CSPs need to know when the problem is with something they can control, rather than the page itself. Furthermore, they want to know if there are common elements to poor (or great) QoE scores. For instance: Do certain device, OS, and browser combinations routinely show up in the list of bad experiences? Did the media optimization solution a CSP just deployed deliver a bump in QoE? Is there a variation of QoE across different logical or geographic segments of the network? Does web browsing QoE vary based upon the peering or transit link from which a page originated? Do certain subscribers have a consistently poor experience (a potential indicator of a problem with the home network)? To give CSPs the information they need to arrive at meaningful, actionable conclusions, for each page load time provided by the PTS, the PTS also provides an assortment of associated attributes, including: 7 Timestamp of the URL request Identity of the PTS that logged the event Identity of the PTS cluster that logged the event Location of the subscriber Subscriber IP address Subscriber Name Website URL (host + resource, with query parameters stripped from the resource) Referrer User Agent header Client Device Client Device Type Client Device OS Transport Protocol Session Protocol Application ID Internet IP CDN AS Path Access Device Code Internet Round-Trip Time (RTT) of the first request flow Additionally, the PTS includes a value to indicate if the web page timed out while waiting for the load to finish. 7 An explanation of how the PTS obtains attributes and information from Internet traffic is included in the Sandvine technology showcase Internet Traffic Classification; specifically, see the section Traffic Attributes Page 6
Figure 3 - Active web browsing flows on a network, with associated attributes and QoE scores listed under "Classifiers" (screenshot from the PowerView screen of Control Center 8) Producing a Meaningful QoE Metric For each monitored web page, the PTS calculates a mean opinion score (MOS) out of five based on the ITU G.1030 specification. 9 To ensure the scores that emerge accurately capture the subscribers perception of their experiences, real-world testing was used as a calibration. 10 Figure 4 - Web Browsing QoE by Web Page (screenshot from Sandvine s Network Demographics interface) 8 Control Center is Sandvine s policy and operations management graphical user interface You can read about the ITU G.1030 in extreme detail starting here: https://www.itu.int/rec/t-rec-g.1030/en 10 The method is quite simple, really: you just ask real people to score their experience for each page and compare against the calculated MOS, and then adjust the MOS calculation as required. 9 Page 7
The Complete Solution A simplified view of the Sandvine solution deployment is shown below, in Figure 5. Figure 5 - Components of Sandvine s Web Browsing QoE solution Figure 6 shows how CSPs can view quality metrics in real-time by using Sandvine s Control Center graphical user interface. In this example, the network s active web browsing flows are shown, differentiated by quality. Figure 6 - Distribution of the quality of active web browsing sessions on a network (screenshot from the PowerView screen of Control Center) Page 8
Conclusions The ability to monitor the subscriber quality of experience for web browsing is essential to determining when and where network conditions are contributing to an impaired customer experience. Sandvine s web browsing QoE solution allows CSPs to monitor, in detail, the web browsing experience of their subscribers. The solution builds a comprehensive library of web page anatomy profiles, each associated with a web page that is either popular with subscribers or otherwise designated as important by the operator. Using these anatomy profiles, and combining with real-time measurements made as web browsing traffic flows through the network, the solution determines the time it takes to load each page. This page load time measurement is turned into a web browsing QoE score (on a scale up to five) that corresponds to subscribers real-world experiences. To maximize utility for the CSP, each web browsing QoE score is accompanied by more than 20 associated attributes that can be used to diagnose issues and to identify quality-related trends. Table 1 - Relating Sandvine's solution to key solution requirements Requirement Ability to measure page load time, from initial request to load completion Machine learning to build and regularly refresh web page anatomy profiles Mechanism to automatically build and refresh a list of the Top N web pages Mechanism to allow a CSP to specify particular web pages Ability to determine when an HTTP GET corresponds to a new page load Ability to provide a comprehensive set of attributes associated with each monitored page load A meaningful web browsing QoE metric Explanation of Sandvine Solution The Policy Traffic Switch (PTS) sits inline and inspects web browsing traffic. The PTS is able to detect the time at which a page load is initiated and the time at which it is completed, and from those points can determine the page load time. On a daily basis, using an integrated browser and a web page parser, the Sandvine solution builds complete, detailed anatomy profiles of each page to be monitored. The PTS monitors actual subscriber browsing to identify the most popular web pages, refreshing the list on a daily basis. The solution allows CSPs to define a list of pages to be monitored in addition to the empirically determined Top N list. Through a combination of heuristics and observations, the PTS can determine which GETs correspond to new page loads, and which can be ignored. For each web browsing QoE observation provided, the PTS also provides an accompanying set of more than 20 associated attributes. Sandvine s web browsing QoE metric is calculated using the ITU G.1030 specification, and is calibrated against real-world subscriber experiences. Related Resources In addition to the footnoted items throughout this document, the following are relevant: Page 9
(whitepaper) Measuring Web Browsing Quality of Experience: Requirements for Gaining Meaningful Insight (whitepaper) Measuring Internet Video Quality of Experience from the Viewer s Perspective (technology showcase) Video Quality of Experience Score Invitation to Provide Feedback Thank you for taking the time to read this technology showcase. We hope that you found it useful, and that it helped you understand our web browsing quality of experience solution. If you have any feedback or have questions that have gone unanswered, then please send a note to whitepapers@sandvine.com Page 10
Headquarters Sandvine Incorporated ULC Waterloo, Ontario Canada Phone: +1 519 880 2600 Email: sales@sandvine.com European Offices Sandvine Limited Basingstoke, UK Phone: +44 0 1256 698021 Email: sales@sandvine.co.uk Copyright 2015 Sandvine Incorporated ULC. Sandvine and the Sandvine logo are registered trademarks of Sandvine Incorporated ULC. All rights reserved.