Essential Ingredients for Optimizing End User Experience Monitoring

Transcription

1 Essential Ingredients for Optimizing End User Experience Monitoring An ENTERPRISE MANAGEMENT ASSOCIATES (EMA ) White Paper Prepared for Neustar IT MANAGEMENT RESEARCH,

2 Table of Contents Executive Summary...1 The Shift from Availability to Experience...1 Essential Elements for User Experience Monitoring...2 To Emulate or Not to Emulate?...3 Having a Second (and Third?) Home...4 The CDN Shadow...5 Neustar Webmetrics Optimizing the User Experience...6 EMA Perspective...7 About Neustar...7

3 Executive Summary In today s Web-centric world, User Experience Management (UEM) services have fast become the best practice for assuring site and Web-based application performance, most especially for those assets which are intended to be accessed from or across public networks such as the Internet. While there are a number of UEM providers available to IT organizations, it is important to understand that not all UEM services use the same strategies and techniques for providing their services. Understanding specifics such as browser fidelity, the influence of Content Delivery Networks (CDNs), and multi-homing may be more important than you realize. This paper reviews these topics and provides information to allow you to make the right decision for your organization. The Shift from Availability to Experience User Experience Management (UEM) has become an essential element of IT service and application assurance. For organizations that depend on network-based business processes and Web storefronts, it is not sufficient to sit back and hope that customers and prospective customers are having a good experience they need to be aggressively and proactively assuring good experience. At stake are customer satisfaction, customer retention, competitiveness, top line revenues, and in some cases even financial penalties if Service Level Agreements (SLAs) are in place. Early approaches to UEM were primarily concerned with internal testing and basic availability. But as the sophistication and complexity of hosted sites and services have increased, so have the Today s UEM technology must help operators move sophistication and complexity of UEM products and services. beyond is it up? to how is Today s UEM technology must help operators move beyond is it up? to how is it performing? and what are my customers it performing? and what are experiencing? And that means shifting from reacting to reported my customers experiencing? issues, in the form of angry customers calling the help desk, to proactive what if monitoring, trending, and alerting so that potential experience-impacting issues and problems can be uncovered and addressed on a preventative basis. Of course, when problems do happen before they can be prevented (and they always do) then the diagnostic information collected by UEM systems needs to be sufficiently rich and relevant to accelerate the troubleshooting and restoration process. Per Enterprise Management Associates (EMA) research into UEM and its close relative, Quality of Experience (QoE), response time leads availability as the primary metric for understanding and characterizing user experience, and the vast majority of organizations 84% of business constituencies and 77% of IT professionals see QoE/UEM as becoming increasingly important in the future. This paper will focus on several of the key requirements of UEM solutions and how to evaluate the alternative approaches available in the marketplace. As part of the development of this paper, we conducted in-depth interviews with IT professionals from two organizations actively using UEM services and discussed what was important to them in terms of both technology and practices. The first is a site manager for a Web-based commercial organization serving both business and consumers who are mostly based in North America. The second is a service assurance manager for a suite of SaaS (Software as a Service) products which are being offered worldwide.

4 Essential Elements for User Experience Monitoring The use of monitoring to understand how end users are experiencing Websites and Internet applications has been a developing and maturing practice for years. Techniques for measuring end-user experience include: Server-side direct measurement: In this approach, agents are placed on the server to take measurements of transactions or sessions, either passively or including tag injection. Passive network session monitoring: This method involves sniffing packet streams and tracking sessions and transactions as they pass the point of instrumentation. Client-side direct instrumentation: With this technique, agents are placed on end-user systems and measurements are taken of actual session or transaction experiences. External synthetic/active testing: This approach involves regular, systematic testing and monitoring of typical user sessions or transactions that have been scripted or recorded, either from within a private network or across the public Internet. EMA believes that each of these approaches can deliver unique and valuable insights for assuring services, and recommends a balanced mix in order to fully ensure optimal operations. This paper will focus on the last of these types external synthetic/active testing which provides an essential viewpoint for anyone managing Web-based content or applications which are expected to be reached across the Internet. The most effective model that the industry has seen for delivering this type of testing is by means of a third-party service, which establishes measurement and monitoring platforms across a broad geography and then offers testing services to those wishing to assure quality performance. Essential elements for a user experience monitoring service include the following: Global infrastructure with multiple sampling points distributed geographically in a way that can broadly portray typical access locations and paths to content. Continuous, periodic testing to ensure problems are found proactively. Flexibility to accommodate the dynamic nature of served content and the latest technologies underpinning rich Internet applications. Data collection and storage to support problem determination, trending, and analysis. Beyond these general capability requirements, it is also highly important to determine the accuracy and reliability of testing results. In doing so, one must understand the technological and architectural choices that third-party testing services have made in implementing their offerings. In particular, the use of browser emulation must be well understood, the techniques whereby sampling stations connect to the Internet will play a role, and the influence of supplemental accelerators such as content delivery networks must be assessed. This paper will focus on these three important concepts, which support testing balance and completeness.

5 To Emulate or Not to Emulate? An essential element of any testing approach or strategy is the mechanism which will be used to conduct the actual tests. In the case of UEM solutions, one important aspect of the mechanism is the decision regarding whether or not to use live/real browsers or emulated browsers. Each approach has its advantages and disadvantages. At its essence, the difference between them is the degree to which the testing platform is truly replicating an end user or customer s client conditions native browser testers are a true replication, whereas emulated browsers make measured assumptions that will result in approximations. On the plus side, emulated browsers are simpler to maintain and easier to scale over large numbers of test sites. As a result, they can provide broader coverage of possible variations of browser configurations as well as browser technologies. This generally makes them easier for UEM service providers to maintain, and thus less expensive to operate, so services based on emulated browsers tend to be less expensive. They are well suited to basic monitoring of availability and performance for the many sites and applications that do not use complex content structures or application components. For example the monitoring requirements of most business critical applications such as Web 2.0, SaaS, e-commerce, video streaming, and social media can over-extend emulated browser systems. True browser approaches offer all of the same capabilities as emulated browsing while covering the cases emulators do not handle well. True browser approaches offer all of the same capabilities as emulated browsing while covering the cases emulators do not handle well. A primary use-case for the true browser test platform is when there are specific browser configurations or technologies which are an essential part of an internet application or site and cannot be adequately tested using emulation. We have one app that only works on IE, so we want to thoroughly test a specific version and configuration, said the service assurance manager, as a result, true browser testing was an essential requirement. True browser testing is also important when the latest Rich Internet Application (RIA) technologies are in use. Examples of RIA frameworks include Ajax, Adobe Flash/Flex, and Microsoft Silverlight. Due to the lack of consistent treatment of RIAs across various browser platforms, emulated approaches cannot be used. Not only do these technologies stress UEM approaches, but building and maintaining testing scripts which properly exercise them can also be difficult if your UEM provider has not designed their testing architecture specifically to deal with them. Bottom line if you wish to include RIAs as part of your UEM testing approach and you wish to fully and accurately replicate user experiences, you will need to engage a UEM provider that supports true browser testing. Anything else could lead to incomplete monitoring data and, as a result, sub-optimal decision making.

6 Having a Second (and Third?) Home Another essential aspect of how end-user experience monitoring must be optimized requires an understanding of the techniques by which those end users are connected to the Internet. Most consumer users will have single point connections to the Internet, while most companies of any size connect to more than one ISP as a matter of ensuring business continuity and managing operational risk. A common method for seamlessly connecting to more than one ISP, as shown in Figure 1, is known as multi-homing. Similarly, connecting to a single ISP is known as single homing. Besides providing more reliable access to the Internet, multi-homing also enables traffic optimization, so the paths to any particular Internet address can be dynamically assessed and the most efficient path chosen based on the current conditions. See Box 1 for additional details on BGP and multi-homing. Figure 1. Multi-homed vs. Single-home Internet access configuration Box 1: Basics of BGP and Multi-homing BGP (Border Gateway Protocol) is the most common technology for exchanging routing information between peers and partners attached to the Internet. It can be used to enforce routing policies and to load balance connections via approaches such as multi-homing. The Autonomous System (AS), a group of elements sharing the same routing policies, and usually under the same ownership, is the fundamental operating unit of BGP. AS entities are assigned registered numbers, or ASNs. When BGP is used to set up a peering with more than one Internet provider, the configuration is said to be multi-homed. When there is no BGP being used (for example, when static routes are defined), or when there is only one Internet provider involved, then the connection is termed as single homed.

7 So how is user experience monitoring different between a single homed and multi-homed configuration? In a single homed approach, monitoring will be consistent in that it will always be bound to a single ISP. It is possible to test multiple ISPs individually but to truly capture all end users experiences would require that you test across all ISPs. With over 3000 ISPs in the United States alone, this is simply not a practical or cost-effective approach. As a result, many UEM providers concentrate on the biggest ISPs, which often results in an unrealistically good picture of average user experience. Also, in the real world, experience by the majority of business-based end users will involve access routes that are selected dynamically and can change regularly between ISPs. In order to ensure that end-user experience measurements are the best representation of actual experience, the randomness of dynamic multi-homed access must be included. The only way to do this effectively is to use a monitoring strategy where measurements are taken from access points that are multi-homed themselves, and where BGP can be used to force the sampling across multiple Internet routes. Understanding when individual ISPs are experiencing performance degradations is useful, but only the largest of site owners can have any hope of being able to take any direct action on that information. I want to know when one or more ISPs are having peering issues which might impact delivery, said the service assurance manager. But what I m more interested in is whether or not my end users are seeing acceptable levels of performance that is how my SLAs are defined. The CDN Shadow In order to optimize the delivery of Web-based content, many site owners engage the services of content delivery networks, or CDNs. What CDNs do is dynamically and intelligently store Web-based content on a global geographical basis so that content, when requested, can be served from a relatively nearby source, thus minimizing latency penalties. In order to achieve this advantage, most site and application owners rely on third-party CDN service providers. There are many such providers in business today. Since CDNs certainly play a role in the effectiveness and efficiency of delivering content across the Internet, they also play a significant role in end-user experience. In fact, many Websites would not be possible without them. We live and die by the CDNs, said the site manager. We ve made architectural decisions about content that wouldn t work without them. Even those who are not so heavily dependent on CDNs, such as SaaS providers and those with thinner content footprints, will often use CDNs for a portion of their materials or offerings. So how do you make sure that your CDN is playing the role you expect it to play? The CDN providers will provide measurements to represent their performance, but they typically rely on measurements that are taken close to their caching servers often from the same data center where the servers are located, and often alongside major telecomm service provider backbones. That s a good best case measure, but does not represent a realistic case of how customers are reaching and experiencing your site.

8 In order to get a more accurate representation, you need to make You need to make sure your sure your UEM measurements are taken from places that represent typical access configurations, where paths will potentially from places that represent UEM measurements are taken require several additional hops, each influencing the end experience. Furthermore, and related to the discussion about multi- typical access configurations. homing above, it is important to take measurements in a manner that accommodates the dynamic nature of how content delivery paths are determined. Again, this means not relying on UEM measurements which are taken too close to large service provider Internet backbones (such as by placing measurement points within big carrier data centers) and/or those using static routes or single-homed connections. Neustar Webmetrics Optimizing the User Experience Neustar offers the Webmetrics Monitoring solution for user experience management as a full service offering for round-the-clock, round-the-globe monitoring of Website and application availability and performance. Webmetrics includes Site Monitoring, Application Monitoring, Web Services Monitoring, Stream Monitoring, and Network Service Monitoring capabilities. The Neustar Webmetrics solution addresses the three key topics discussed in this paper: 1. True Browser Testing Neustar Webmetrics offers true browser testing for Microsoft Internet Explorer in addition to emulated testing. This allows a balanced approach which can deliver highly detailed and accurate testing of RIA-based sites and applications while also providing broad coverage for less complex sites and applications. 2. Multi-homing Webmetrics has over 100 points of presence in major cities throughout the world. These sites employ BGP multi-homing in order to provide a more realistic assessment of user experience than other UEM providers who rely on direct connection to a single telecommunications provider. 3. CDN Management The Webmetrics approach offers an important and unique method for assessing effectiveness of CDNs relative to the user experience. By locating their test points in locations that are not owned by major telecomm providers, they include additional hops and paths that will be more representative of the majority of actual end users. And how do Neustar s customers view the Webmetrics Monitoring solution? We initially looked to Webmetrics because they provided the best capability at the lowest cost, said the site manager, but eventually we were really won over by their responsiveness to our evolving needs. The service availability manager had additional reasons for selecting Webmetrics, sharing that we believe they have the best network of agents across the globe and they have the most objective measurements due to their testing approach, which is very important to us for analyzing and reporting on SLA compliance.

9 EMA Perspective Enterprise Management Associates maintains that the path to long-term strategic value for information technologies is through the evolution towards service-oriented, proactive, operational methods across the full lifecycle. An essential step along this path for any organization offering a Web-based or Internet-accessible site or portal for their products and services is the deployment of user experience monitoring and management technologies and practices. And with the underlying technologies for Web-based browsers and applications moving as quickly as they are, EMA further recommends that IT teams look to advanced UEM solutions with true browser fidelity, a global infrastructure with multihomed Internet access, and comprehensive data collection and reporting to give them the competitive edge in assuring quality experiences. Not all UEM solutions are created equal. In evaluating UEM solutions, it is imperative that the underlying technologies and service architecture be examined to ensure that the testing and monitoring will be accurate, comprehensive, and actionable, while meeting the specific needs of your organization. It is important to realize that not all UEM solutions are created equal. In particular, if you need true native browser testing or you are seeking the most realistic and reality-based approach for assessing experience in and around CDNs, cloud computing, SaaS, and multi-homed Internet access, then it would be worth a close look at the Neustar Webmetrics solution. About Neustar Neustar (NYSE: NSR) solves complex communications challenges by providing market-leading and innovative solutions and directory services that enable trusted communication across networks, applications and enterprises around the world. Neustar Webmetrics services provide Website testing and monitoring for companies that want to ensure online performance, competitive advantage and a positive end-user experience. Webmetrics services test, monitor and measure the performance of Websites, Web transactions, Web services and streaming media to ensure 24/7 uptime and performance integrity. Visit Neustar online at and Webmetrics online at

10 About Enterprise Management Associates, Inc. Founded in 1996, Enterprise Management Associates (EMA) is a leading industry analyst firm that specializes in going beyond the surface to provide deep insight across the full spectrum of IT management technologies. EMA analysts leverage a unique combination of practical experience, insight into industry best practices, and in-depth knowledge of current and planned vendor solutions to help its clients achieve their goals. Learn more about EMA research, analysis, and consulting services for enterprise IT professionals and IT vendors at or follow EMA on Twitter. This report in whole or in part may not be duplicated, reproduced, stored in a retrieval system or retransmitted without prior written permission of Enterprise Management Associates, Inc. All opinions and estimates herein constitute our judgement as of this date and are subject to change without notice. Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. EMA and Enterprise Management Associates are trademarks of Enterprise Management Associates, Inc. in the United States and other countries. EMA, ENTERPRISE MANAGEMENT ASSOCIATES, and the mobius symbol are registered trademarks or common-law trademarks of Enterprise Management Associates, Inc. Corporate Headquarters: 5777 Central Avenue, Suite 105 Boulder, CO Phone: Fax: