Single Source of Truth

Transcription

1 Whitepaper Single Source of Truth A Guide for RUM (Real User Measurement) in the Workplace By Cliff Crocker, VP Product Management, SOASTA, Inc.

2 Table of Contents Executive Summary... 3 Aligning Goals:... 4 The tools used by line of business and the tools used by Operations... 5 The back end and front end of measuring page load time... 5 The Evolution of Web Performance Measurement to 2005 Simple Web and browser landscape to 2009 Browser diversity and the mobile Web to Present The dam bursts... 6 Gathering real user data with Boomerang... 6 LogNormal and user experience... 7 Real User Measurement: Actionable Intelligence for Both Operations and Line of Business... 7 RUM as a business tool... 7 Implementing RUM... 7 mpulse RUM from SOASTA... 8 RUM case study on Walmart data... 9 Conclusion SOASTA, Inc. SINGLE SOURCE OF TRUTH WHITEPAPER 2

3 Executive Summary There is a strong correlation between page load time and human behavior. If users cannot engage quickly with sites and apps, their user experience suffers and they will take their business elsewhere. While the focus on the back end (server) is on stability and scalability, 80 to 90 percent of page load time is spent on the front end (browser). page load time and user engagement, and shows where to optimize performance for more positive outcomes. The key is user experience. A gratifying user experience helps convert prospects into customers and turn casual visitors into repeat visitors. Organizations that depend on user engagement through their web sites keep an eye on user experience and constantly tune pages to improve it. By whatever measure revenue, customers, cart abandonment rates, bounce rate, length of session the largest properties on the Web have repeatedly demonstrated a correlation between user engagement and site responsiveness, also known as page load time. Fast pages mean a better user experience, which leads to better conversion. In the quest to optimize user experience by decreasing page load time and increasing customer conversion, line of business managers turn to the Operations team to examine their data and quantify the impact of performance. Operations has long focused on optimizing back-end, serverside performance, and this remains critical, but 80 to 90 percent of Web performance problems are in the front-end, client-side environment. These client-side problems in page load time are numerous and compelling, and it takes a more holistic approach to measure them than commonly used synthetic toolsets alone provide. Synthetic monitoring of web site performance using bots or agents has given way to real user measurement (RUM) based on data collected directly from the client. This paper provides an overview of the business and technical aspects of measuring performance. It will provide an understanding of how RUM can offer a single source of truth performance improvements in the bottom line. "Reduce staff and monetary investments in synthetic monitoring and redirect them to higher value activities, such as real-user monitoring." Gartner "Use Synthetic Monitoring for Availability and Real User Montioring for Performance", Jonah Kowall, 10 December 2012 SOASTA, Inc. SINGLE SOURCE OF TRUTH WHITEPAPER 3

4 Figure 1. Single page visits vs. page load time Single Page Visits Load Time PAGE LOAD TIME IS SO CLOSELY LINKED TO USER EXPERIENCE AND USER ENGAGEMENT THAT A DELAY OF EVEN ONE SECOND CAN BE THE TIPPING POINT THAT SENDS A USER TO A COMPETING SITE. Aligning Goals: How Line of Business and Operations Measure Web Performance app downloads, but low app usage. These are symptoms of user experience and user engagement problems. In the world of Web and mobile commerce, these business issues, often discovered by line of business managers in Marketing or Sales, have technical roots that lead to Operations, then into the area of Web performance. The tools used by line of business... Most businesses use several kinds of tools to look at key performance indicators (KPI) like conversion rate, bounce rate, exit rate, cart size, downloads and number of page views: Marketing analytics tools like Omniture, Google Analytics and WebTrends track reach and longterm customer engagement. Customer experience management tools including CA Customer Experience Manager and Tealeaf (IBM) analyze the what they did and how they did it to identify ways of improving the user experience. When these tools show metrics that are low or uneven across the site, the line of business asks why, then tries something new. Often, however, it s not a business issue, but a problem with the responsiveness of Web pages and apps. It s a well-documented fact that user engagement e.g., conversion, bounce rate, length of session correlates very closely to response time. 1,2,3,4 Page load time is so closely linked to user experience and user engagement that a delay of even one second can be the tipping point that sends a user to a competing site. In October 2012, LogNormal (now a SOASTA company) published a study using a cross-section sampling of sites containing approximately 1.5 million visits and 8 million page views. This study showed a very strong correlation between page load time and bounce rate (single page visits to the site). As shown by the slope of the line in Figure 1, seconds count. Unable to solve the user experience problem with its own tools, the line of business reviews the problem with Operations to see whether Web performance is at issue. 1 AOL: Visitors in the top-ten percentile of site speed viewed from 30 to 250 percent more pages than visitors in the bottomten percentile. The Secret Weapons of the AOL Optimization Team, by Dave Artz, June Shopzilla: Speeding up average page load time from 6 seconds to 1.2 seconds resulted in 7-12 percent higher conversion and 25 percent more page views. Shopzilla Site Redesign We get what we measure, Phil Dixon, June Bing and Google: Starting at 500 milliseconds, server delays led to lower satisfaction, lower revenue per user and longer time to click. Performance-Related Changes and their User Impact, Eric Schurman and Jake Brutlag, June Amazon: Every 100ms delay costs 1 percent of sales. Make Data Useful, by Greg Linden, November From The Performance Golden Rule, Steve Souders, February SOASTA, Inc. SINGLE SOURCE OF TRUTH WHITEPAPER 4

5 backend frontend Figure 2. Waterfall chart, back end vs. front end for typical site (x-axis: time) ms 249 ms 209 ms 333 ms 56 ms 197 ms 85 ms 479 ms 386 ms 678 ms 546 ms 503 ms 651 ms 75 ms 203 ms 181 ms See 5 OPERATIONS AND LINE OF BUSINESS NEED TOOLS THAT HELP THEM ALIGN THEIR GOALS BY REACHING ALL THE WAY THROUGH TO THE USER S BROWSER OR APP TO IDENTIFY PERFORMANCE PROBLEMS THAT IMPACT USER EXPERIENCE.... and the tools used by Operations Operations relies on its own set of tools to keep an eye on site health: Synthetic monitoring tools such as Keynote, Compuware Gomez, and HP BAC test for uptime and availability, but work outside of the actual user experience. Application performance monitoring tools like CA Introscope and Compuware dynatrace isolate server-side application bottlenecks, but their primary focus is not on the client-side experience. Often, Operations dashboards show that pages are responding normally to requests for content and that page load time is acceptable, by their measurement. However, these results come from synthetic tools that do not provide an accurate view of performance from the client side, which has grown in importance over time. The back end and front end of measuring page load time Back-end tools measure server-side response in Time to First Byte (TTFB), the critical amount of of time includes the application calls, database lookups, packet movement over the network and HTML assembly that can affect user engagement. Figure 2 depicts how much more time front-end, client-side processing executing JavaScript and cascading style sheets (CSS); laying out and rendering the page; downloading ads, images, widgets, Flash objects and other resources on the page can take than back-end response time. With 80 to 90 percent of page load time spent on the front end 6, Operations and line of business need a holistic monitoring solution that aligns their common reporting goals and displays the entire performance picture so that they can optimize user experience in web sites and apps. History shows that this is not as easy as it sounds. 5 For an illustration of the Navigation Timing API processing model, see 6 From The Performance Golden Rule, Steve Souders, February SOASTA, Inc. SINGLE SOURCE OF TRUTH WHITEPAPER 5

6 The Evolution of Web Performance Measurement Even as the focus of testing has moved to the front end, evolving variables have made the client a moving target to 2005 Simple Web and browser landscape In the early days, measuring Web performance against web sites was relatively simple since the only browser that mattered was Internet Explorer 6. Web teams assumed IE6 as if it were a standard, coding and monitoring their pages accordingly. Also, in this era before content providers and third-party JavaScript, the Web was mostly a vehicle for showing and linking simple pages, so almost all Web content went straight from behind the Companies monitored their sites synthetically, using bots (agents) to simulate the activity tool for identifying user experience problems rooted in Web performance. USING A BOT TO MEASURE REAL USER EXPERIENCE IS LIKE ASKING A ROBOT TO DRIVE ACROSS THE COUNTRY AND REPORT WHETHER IT WAS EXCITING OR BORING. IN FACT, IT S LIKE ASKING THE ROBOT TO DO IT HUNDREDS OF TIMES ON DIFFERENT ROADS IN DIFFERENT VEHICLES to 2009 Browser diversity and the mobile Web As the Web grew into an application platform, Web teams began working with dynamic content and technologies like Ajax and JavaScript. These demanded more of the browser, leading to alternative products like Firefox and Chrome, and prompting Microsoft to invest more heavily in IE. Suddenly, the diversity of browsers complicated performance measurement: it was no longer safe problems could go undetected if the organization measured only against IE. Furthermore, the lines between the mobile Web and the desktop Web began to blur with the launch of the iphone. This meant important new points of user engagement, setting off a scramble in synthetic monitoring to handle many more combinations and scenarios in order to still be relevant in measuring performance to Present The dam bursts Finally, it became clear that synthetically monitoring performance in multiple browsers on multiple devices with multiple types of content and applications would not scale. Tablets, Android, the mobile Web and disparate mobile platforms conspired to make user experience much more volatile and sensitive as users moved around. It was time to look for new ways of measuring page load time from inside the browser, as close as possible to the point of user engagement. Gathering real user data with Boomerang the experience of real users in business terms, such as perceived response time, user reactions, conversions, purchases and length of session. Bots cannot generate reliable information at that level through estimates and synthetic modeling; it s necessary to collect organic data from the browser as the user navigates the page. Using a bot to measure real user experience is like asking a robot to drive across the country and report whether the experience was exciting or boring. In fact, considering the possible the robot to make the drive hundreds of times on different roads in different vehicles. engineers began experimenting with JavaScript embedded on Web pages to measure page load time. These projects went beyond back-end TTFB to measure the front- and back-end time from start to end of the user s request. In 2010, Yahoo! published its Boomerang tool, an open source JavaScript library for collecting organic performance data through the browser and sending it back to a server in beacons. applications to access timing information related to navigation and elements. If the user s browser page load time. SOASTA, Inc. SINGLE SOURCE OF TRUTH WHITEPAPER 6

7 LogNormal and user experience Boomerang set the stage for capturing accurate information from the client device with a few lines of JavaScript in a Web page and sending it back to a server for analysis. But the resulting avalanche of data overwhelmed most Operations teams struggling to use it for performance optimization. This led to products like LogNormal for analyzing and visualizing organic, real-time data on page load time. LogNormal offers Operations and line of business immediate insight into user experience straight from the browser, in metrics that align their common goals and allow them to tune web real-time, back-end analysis, Web performance measurement has entered the era of real user measurement. LogNormal is now part of SOASTA and the product is called mpulse. Real User Measurement: Actionable Intelligence for Both Operations and Line of Business Because RUM collects metrics from the browser or mobile application, performance engineers are turning to it for a more holistic view of user experience. browser, location, page, device and connection speed. While synthetic monitoring may measure measure from inside the browser or application, catching every interaction the user has with the site. RUM IS A BUSINESS TOOL THAT QUANTIFIES THE IMPACT OF HIGH PAGE LOAD TIME AND SHOWS LINE OF BUSINESS MANAGERS WHERE THEY NEED TO EXPEND EFFORT TO AFFECT KPIS LIKE CONVERSION AND BOUNCE RATE. RUM as a business tool In three important ways, RUM picks up where synthetic monitoring tools leave off: No bots Instead of using bots to approximate what the user is seeing, RUM presents data directly from the user s session. No guess work With RUM, there is no need to guess and map out the likely path users will follow through the site for synthetic modeling. RUM measures the parts of the site that really matter to users. No ambiguous data RUM provides a single source of truth about page load time and helps RUM is more than an engineering tool or a Web performance tool. It is a business tool that to prioritize their effort on which pages, demographics, locations, browsers, devices to affect KPIs like conversion and bounce rate. Implementing RUM RUM works in three steps: Web developers add a RUM JavaScript snippet to their pages, or embed a RUM library within their native application. As users visit the page to shop and consume content, small request objects (beacons) containing The receiving server aggregates and processes the data, displaying metrics through a portal or writing to a data warehouse. The typical Boomerang JavaScript implementation uses a low-overhead, asynchronous, nonblocking snippet of JavaScript. This ensures that measurement of page load time is as non-invasive as possible, minimizing the observer effect that would otherwise skew results. 7 RUM is currently at work measuring user engagement on customer-facing web sites and mobile sites and media companies. 7 For more details, see The non-blocking script loader pattern, by Philip Tellis, December SOASTA, Inc. SINGLE SOURCE OF TRUTH WHITEPAPER 7

8 Figure 3. mpulse in action mpulse RUM from SOASTA mpulse from SOASTA is a RUM product for Web and mobile applications based on the Boomerang JavaScript library. mpulse measures page load time through any browser that supports JavaScript, and leverages the Navigation Timing API. mpulse supplements the client-side component of RUM with the aggregation and reporting tools required to analyze large volumes of data in real time. It is unique among RUM tools in covering all three platforms of user engagement: Web, mobile Web and native app. See mpulse in action at SOASTA, Inc. SINGLE SOURCE OF TRUTH WHITEPAPER 8

9 RUM case study on Walmart data 8 With the goal of improving page load time, the Web performance optimization team at Walmart implemented RUM on its site. They took baseline measurements, created targets for page performance and then measured the impact of page optimization on several business metrics. In general, starting at as little as 1-2 seconds of page load time, conversion rate declines sharply (Figure 3).!"#$%&'"()*+,-)",-"*.-/)&0,.-"&%1)"%0"%/)&%2)"0,1)"+.%("34)",-*&)%0)0"5&.4"6"1."7"0)*.-(0" '' " Conversion Rate Vs. Load Time!"8/)&%++"%/)&%2)"0,1)"+.%("34)",0"+.9)&"5.&"1$)"*.-/)&1)("'.':+%3.-";<=>>"#)*.-(0?"1$%-"1$)"-.-@ Population (%) Conversion Rate (%) *.-/)&1)("'.':+%3.-";A=B<"#)*.-(0?" Figure 4. Conversion rate vs. page load time >15 Load Time (Seconds) At the department or category level, there is a correlation between long load times and low conversion (Figure 4). Figure 5. Department vs. conversion T_Page (Seconds) Department Vs. Conversion T_Page - Not Converted (Seconds) T_Page - Converted (Seconds) Department Even at the page level, long load times still correlate to low conversion (Figure 5). Page Type Vs. Conversion T_Page - Converted (Seconds) T_Page - Not Converted (Seconds) Figure 6. Page type vs. conversion T_Page (Seconds) Page Type In short, customers with the best possible load time had the best possible conversion. Based on this RUM data, Walmart found that for every second of improvement, conversions increased by up to 2 percent. They also found that, for every 100 milliseconds of improvement, they grew incremental revenue by up to 1 percent. The study illustrates the potential of RUM in identifying the impact of Web performance on human behavior so that developers can optimize pages and apps accordingly. 8 Slides and content cited at Web Performance Today, Joshua Bixby, February SOASTA, Inc. SINGLE SOURCE OF TRUTH WHITEPAPER 9

10 Visualize Business Impact Use a multi-faceted approach to understanding the user demographic. Act on What Matters Respond fast and validate resolutions in real time. Fuel DevOps Utilize a single pane of glass for every role in the organization. Conclusion When users perceive that it takes too long to get the content they want, they go to a competitor s site or mobile app. Success in user engagement on the Web and in mobile depends on site responsiveness and page load time. As organizations try to optimize their sites for higher conversion, they can use back-end tools to measure page load time from the network perspective and synthetic monitoring tools to approximate the user experience, but they often overlook the true client-side impact of performance. mpulse from SOASTA is based on real user measurement (RUM), an effective, non-invasive way of measuring perceived load time from the user, at the exact point of engagement. mpulse clearly measures the impact of performance on human behavior with data that both Operations and line of business can use to tune web sites and mobile apps for better user experience. About Cliff Crocker Cliff Crocker, vice president of product management at SOASTA, is a ten-year veteran of Web and mobile performance, and has experience successfully managing systems architecture, capacity planning, load testing and Web/mobile performance monitoring. Prior to joining SOASTA, Cliff spent two years as senior engineering manager at Walmart Labs, where he led the performance, reliability and site analytics initiatives for all its web site properties. He previously spent four years at Keynote Systems providing technical quality analysis for senior management at leading mobile and Web companies. Cliff is an active participant in the Web performance community, evangelizing the importance of performance and its impact on the bottom line. Cliff holds bachelor degrees in Business Administration, Computer Information Systems and Economics from Colorado State University. Attribution Much of the content of this paper was presented and discussed in a public web presentation by SOASTA that included Cliff Crocker and other RUM pioneers: Buddy Brewer Former CEO of LogNormal, now SOASTA VP of Engineering Philip Tellis Former CIO of LogNormal, now SOASTA Chief Architect, and Aaron Kulick Founder of the SF WebPerf group and current guerrilla engineer on the Big Fast Data team at WalmartLabs To replay the event, visit: About SOASTA, Inc. SOASTA is the leader in cloud testing. Its web and mobile test automation solutions, CloudTest and TouchTest, enables developers, QA professionals and IT operations teams to test with unprecedented speed, scale and precision. The innovative product set streamlines test creation, automates provisioning and execution, and distills analytics to deliver actionable intelligence faster. With that their applications will perform as customers are many of today s most successful brands including American Girl, Chegg, Gilt Groupe, Hallmark, Intuit, held and headquartered in Mountain View, Calif. For more information about SOASTA, please visit Headquarters Castro St, Fourth Floor Mountain View, CA FAX: EMEA +44 (0) Thames Court 1 Victoria Street Windsor, Berkshire SL4 1YB To learn more visit: soasta.com or us at info@soasta.com 2013 SOASTA. All rights reserved. SOASTA, the SOASTA logo, and SOASTA CloudTest are trademarks of SOASTA. All other product or company names may be trademarks and/ or registered trademarks of their respective owners. Connect with us: