Ready, Set, Go! or Slow?

Transcription

1 Leveraging the Cloud for On-Demand Performance Testing COLLABORATIVE WHITEPAPER SERIES

2 New user behaviors that businesses are introducing to their web sites are driving more volume than ever before. Product promotions and shifting business models can trigger radical spikes in user traffic and expose issues not encountered at more traditional levels. This new, unexpected workload can result in a flash or burst and spike traffic to dramatically high levels in a short period of time. Why does this happen? For instance, marketing departments often neglect to inform IT departments of aggressive campaigns. Businesses often don t take the key steps to ensure that IT can support these campaigns. Through Groupon, the Gap (one of the top 25 most visited retail web sites) offered $50 of merchandise for $25 and consumers bought more than 441,000 vouchers. Amazon (the number one Internet retailer) partnered with LivingSocial to make $20 vouchers available for $10 and shoppers scooped up more than 1.3 million of them. As a membership reward, Gilt and Rue La La schedule a limited time, private sale of high-end merchandise. Access patterns from mobile devices such as the iphone plus content aggregators such as Mint.com can drive system workloads to unexpected levels in a short period of time. These types of events can provide a cataclysmic rush to IT resources, and IT is often forced to react to a performance-engineering emergency instead of being prepared ahead of time. I. Changing online business workload Web sites that previously experienced smooth and steady traffic increases that could be accurately predicted over time are now subject to erratic day-to-day volumes and dramatic surges. IT groups are quickly trying to assess the impact to the infrastructure that these workload profiles generate. The load-testing infrastructure required to simulate these extreme burst loads can be prohibitively expensive to create and maintain. The traditional load testing approach of simulating virtual users can test many aspects of the system, but since the tests originate inside the company s network instead of on the Internet, many complicated factors are unable to be validated. This leaves a significant gap in the IT team s ability to accurately validate the web site s ability to absorb burst loads. A website is a complex system with many components. These components may not have been designed to handle sudden and extreme bursts of volume. As components in the system become stressed and begin to fail under this burst load, the web site often becomes unstable and users of the site are subjected to longer and longer response times. As this occurs, customer experience with the site will descend from satisfied to tolerable to frustrated and finally to abandonment. 2

3 This begs the question: how does your web site s response time change under sudden and increasing volume? Does it reach a tipping point and fail completely? Does it accept all requests and then do all users experience a significant slowdown? Do your best customers see the same slowdown as those just browsing or are they all treated equally? These questions can only truly be answered by validating your web site during burst levels. II. Traditional performance testing approach The traditional approach to performance and scalability testing relies upon a relatively small number of virtual users to generate the same number of transactions per second that the larger real user population is expected to generate. These virtual users are browserless, that is, the test application records the HTTP requests that a browser will make and stores them in a script that will be executed during a test. The virtual users execute inside the corporate firewall and often access a test environment that is also within the corporate network. This type of test is most effective for stressing the application and database servers but falls short of simulating the real production environment, or real world user experience. Figure 1: Flash events, and cloud performance testing measure the actual experience the end user will have while the system is under load. In most cases, all of this testing occurs in a test environment which is designed to mimic the production environment but is usually not an exact copy. In addition to load testing in a test environment, many organizations use third party monitoring services in production to characterize the real experience users have throughout the day. These services, provided by companies such as Keynote Systems or Gomez simulate users from across the Internet and record the results. A virtual user profile is designed and then executed from several locations around the country. These companies provide you daily reports of the end user experience. III. Responding to the new workload IT should consider some important questions in analyzing whether their performance strategy is on track to handle future needs, such as: Have you tested your web site for a burst situation? Have you measured what the user experience will be and can you communicate this to your business? Given the new workload to the web site, how does the performance engineering team respond? Do they have the ability to define and execute the volume of test scenarios required to simulate a burst for instance the load that 100,000 users will generate? How will the Internet between your corporate firewall and their browser impact the customer experience on your web site? Other techniques are used to simulate different aspects of the system and attempt to understand the user experience while the system is heavily loaded. Sometimes a network tool may be used to constrain the available bandwidth during the test for some of the virtual users and to simulate conditions outside the corporate firewall. Additionally, a functional testing tool may be employed during the performance test to exercise the web browser and to Can your load-balancing strategy accommodate a partial failure in one site while maintaining overall user experience? The answers to these questions may drive the performance team to assess or reassess their ability to implement a burst load scenario. The sheer scale and complexity of designing an environment capable of simulating burst loads is daunting. The virtual user often licenses traditional load testing tools. Simulating more virtual users requires additional licenses, which must be borrowed from other 3

4 parts of the organization or, in many cases, the team must work with their load testing tool vendor to buy temporary licenses. Some of the traditional vendors may offer a lease-based licensing model. This is usually a time based license for a much higher number of virtual users, though supplying the hardware to generate the increased number of users remains a challenge. To execute this burst load volume the team will also have to procure or buy or borrow enough hardware to run these virtual users. All of this requires significant coordination, overhead and leadtime. Setting up and managing this environment to support these virtual users can be prohibitively expensive. Burst load tests will likely occur less frequently than traditional performance tests, which means the licenses and servers will sit idle a majority of the time further reducing the economic attractiveness of this option. IV. Inside-in versus outside-in A vast majority of performance and stability tests are executed within the firewalls of the company or inside-in. This has the advantage of allowing the performance teams to run tests in a repeatable and controlled process in a closely managed environment. Unfortunately, the number of components between the corporate firewall and the customer s browser is large and is growing more complex every day. The types of traffic running over the Internet are growing and diverse. HTTP is no longer the only protocol on the Internet. Android and ipad/iphone apps are rapidly expanding in their diversity and proliferation and Skype and video conferencing are commonplace. Desktop sharing with WebEx and Join.me are becoming more popular. Video streaming from YouTube, Netflix, Amazon and other services are gaining market share from traditional bricks and mortar content delivery channels. Ignoring these variables leads to a false sense of security, while simulating them is nearly impossible. Web 2.0 sites typically have content sourced from multiple partners as well as from your own infrastructure. The only way to truly understand how your web site will handle burst loads across the Internet is to test from the Internet. V. Extending capability to the cloud Running large-scale volume tests from the cloud will help you uncover risks within your systems and provide you with response time measurement from outside your corporate firewall for your key business transactions. This will provide a key indicator of your customers expected web site experience during a burst load. The forward thinking performance team should incorporate cloud-testing capability into its toolkit. The cloud option will allow the performance team to quickly design and execute simulated burst tests on the web site by eliminating the lead time and overhead associated with building out the infrastructure to support such a test. The team should focus on value-added activities such as test design while cloud sourcing the overhead activities to the cloud provider. To build out cloud testing capability efficiently, several corporate processes and procedures will require change and the core performance processes, tasks and activities will need to adapt to this model. Teams must answer many questions such as how frequently will cloud testing be used? and what qualifies a web site to require cloud testing? Cloud-based burst tests against Internet-facing environments may closely resemble Denial of Service attacks to the network operations team, so before executing the burst performance test, you must coordinate with them. Burst tests from the cloud to a test environment will require firewall changes and will require review by the network security team (in order to efficiently monitor test environments). Sometimes sensitive data must be used to drive the performance test. The data formerly contained within the firewall will now be placed on the cloud. Risk and security teams must ensure that the processes are in place to protect this data. All performance tests need the ability to log in to the system undergoing the test; this necessitates having a population of test user ids for the application. The performance engineer must consider how to create the very large number of test accounts required for the cloud performance test. Creating 100,000 unique users is no simple task. A valid performance test requires realistic test data and oftentimes this is masked as production data. 4

5 Figure 2: Cloud testing All of these factors require careful planning to ensure successful cloud-based testing. There are a number of emerging options for leveraging the public cloud to extend your performance testing capability including SOASTA, CloudTest, Gomez RealityLoad, and HP LoadRunner in the cloud (currently Beta). Plus you can provision your own cloud service using Microsoft Azure, Amazon EC2 or Rackspace s public clouds and drive cloud-based loads using your own load generating software. Leveraging private cloud performance testing requires a significant financial commitment and may not be practical for your organization. Some vendors provide professional services along with their load-generating tool and actually participate in the test execution. They help design and execute the burst test with you leveraging their deep knowledge of their tool and experience with other clients. Other vendors provide only the tools and cloud infrastructure while the company designs and executes its own tests. Vendors such as SOASTA provide a dashboard along with their load test tool that reveals a real-time system level utilization and metrics in addition to the transaction response times all in one integrated console. Other vendors require you to build this capability yourself or to coordinate the system level metrics and the cloud test response times for correlation after the test. Whether you choose a fully inclusive vendor solution, grow your own using existing tools and public cloud infrastructure, or invest in a private cloud infrastructure, you will spend less time on expensive overhead activities such as managing licenses and provisioning physical servers and more time on the value-added activity of performance testing the system. VI. Conclusion Conventional performance testing plays a critical part of the ongoing process of validating the fundamental performance and stability of applications. Your current load-testing 5

6 environment allows you to test incrementally and to target key components. Testing with your current infrastructure at sub-burst levels can identify many issues. Conventional testing serves as the first gate in your comprehensive performance testing approach. Once the system has cleared the conventional testing gate, outside-in testing from the cloud at burst levels provides the best opportunity to efficiently validate your system at much higher volumes. Testing can be designed to validate performance at extreme burst levels or even to find the upper limit of a system s scalability and reliability before users see an unacceptable increase in response times and abandon your web site. The cloud performance testing market is rapidly emerging and undergoing changes every month. Many enterprises are reviewing the benefits that an internal cloud environment can provide. There will be the opportunity to leverage the internal cloud for performance testing in a shared services environment. The key to an internal performance lab is the utilization of the environment. A highly utilized (5 by 24) performance environment should be the goal of every performance team. While running load tests from the cloud cannot replace your current investment in traditional load testing, there are significant and real advantages to pursuing cloud based approaches. Both approaches work together to mitigate business risks related to system performance in a complementary way, and understanding both the risks of burst loads and the options to manage them will ultimately make the business of the web for both marketers and IT more successful. 6

7 Collaborative Consulting is a leading information technology services firm dedicated to helping our clients achieve business advantage through the use of strategy and technology. We deliver a comprehensive set of solutions across multiple industries, with a focus on business process and program management, information management, software solutions, and software performance and quality. We also have a set of offerings specific to the life sciences and financial services industries. Our unique model offers both onsite management and IT consulting as well as U.S.-based remote solution delivery. To learn more about Collaborative, please visit our website at us at sales@collaborative.com, or contact us at Copyright 2014 Collaborative Consulting, LLC. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. WP