Stop Cheating: Four Obsolete Performance Testing Work-Arounds

Transcription

1 Stop Cheating: Four Obsolete Performance Testing Work-Arounds Performance testing is often as much an art as a science. Traditionally, performance testing operates within such significant constraints that testers rely on incomplete data collected in isolated test labs, sophisticated mathematical models, past experience and gut instinct to make their predictions. This remains true even when using automated testing tools. The reality is that most production websites today go live without adequate performance testing. It s true even of big name sites that can afford the test beds and tools. Automated performance testing is underutilized, and the lack of adequate testing leads to slow-downs and failures that are costly for online businesses. In many cases, thorough performance testing at scale is seen as too difficult and expensive. Even in organizations with performance testing practices, many longstanding strategies are based on constraints and limitations that are no longer valid. In this paper we'll look at some of the testing strategies designed to work around constraints or limitations in testing technologies, and how they compromise the effectiveness of performance testing. Past constraints leave a legacy of inadequate testing Automated performance test tools first came on the market in the early 1990s. From a technology perspective, this was a different era before web applications became an essential sales and marketing channel for many businesses. Despite the power of these automated testing tools, testers had to work with serious constraints on their testing: - Hardware limits: Hardware for testing is always expensive, inadequate and time-constrained. Most test labs are comprised of aging, mix-matched load generation hardware that limits the quality and quantity of simulated load. The test tools that run on this hardware often have their own scalability and performance issues, so testers became creative in doing their best with what they had. - Virtual users: Most testing tools are priced per virtual user with prices reaching $1,200 per virtual user! This pricing model created a strong financial incentive to limit the number of virtual users in any test case. Because of the cost and complexity, automated performance testing was mostly the domain of mid-sized and large companies in the '90s. And, in the days before web applications, an application with 1,000 concurrent users was a pretty large application. Fast forward to present day. Web applications have to handle load in the tens or hundreds of thousands or more, coming from around the globe. User expectations for performance continue to climb, and consumers are likely to abandon online sites that are slow or unreliable. Even in this radically different environment, performance testing as a discipline

2 Four common work-arounds in performance testing Here are four common strategies that performance testers still use today, with traditional performance testing tools, to try to scale up their testing without increasing costs. 1. Modifying think time to get more testing out of a fixed number of virtual users 2. Extrapolating capacity by testing a smaller environment and calculating performance at scale 3. Singling out critical pages or silos instead of testing the entire workflow 4. Measuring performance by response time alone Each of the strategies is at best a kludge, leaving the potential for being blindsided by unexpected performance problems in production. And the tragedy is, it's no longer necessary to make these compromises. The current generation of cloud-based testing tools makes these constraints unnecessary. Let's look at each of these work-arounds in a little more detail. # 1 Cutting think time You have a license for 1,000 virtual users on your testing tool. You want to test to 2,000 concurrent virtual users. So you cut think time in half and double their actions. Problem solved? In the actual performance environment, users will have varying think times. You need to be able to randomize this variable. And shortening or condensing the think time unrealistically doesn't create an accurate test with realistic session states. When you set up your thousand virtual users each with a two-second think time, you can end up with one thousand virtual users virtually clicking the same button at the same moment:

3 Figure 1: View of the same test scenarios with varied think time to illustrate how a system experiences user impact. Credit: Scott Barber,PerfTestPlus This creates an interesting stress test. But it's not an accurate performance test for your site in the real world. People using websites think, pause, and spend varying amounts of time on pages based on the content. The supporting infrastructure responds by maintaining session states and managing resources based on these timings and behaviors. Running tests that accurately mimic real-world usage requires a 1:1 ratio of expected real users to virtual users and the test hardware to support it. Without this, you haven't tested the real production usage of the application, and can still get unexpected problems that testing did not uncover. # 2 Extrapolating capacity Another common strategy is extrapolating system capacity based on smaller tests. This constraint has practically been forced upon testers working with under-provisioned test environments that do not adequately represent the production environment. Examples include: Testing a small slice of the infrastructure (1/n) for capacity and assuming that the production capacity will be n times test capacity Assuming that a faster processor in the production environment will deliver a linear increase in speed Performance testers have developed sophisticated mathematical models to extrapolate performance. But the real world isn't a perfect place any differences between the model and production can invalidate the results. And even little things can make a difference. According to Scott Barber, founder of PerfTestPlus, something as minor as a difference in hard drive brand can have a measurable impact on performance. Most extrapolations assume that every component of the application will scale linearly. In reality, we know that performance testing is all about chasing bottlenecks. If you must extrapolate from a smaller environment, then you need to admit that you're making educated guesses. There are legitimate reasons for extrapolating for future capacity, but the best way to collect data for future models is testing to measure full capacity today.

4 #3 Unrealistic user flow modeling When testing resources are scarce and costly, performance testers may be forced to choose only a couple of user scenarios that represent the most common or critical use cases. Regrettably, many services and tools endorse this practice because running realistic workflows is hardware-intensive. To run the service, you are required to isolate individual pages among many, or single out the top few transactions that matter. This strategy creates unrealistic workloads that point all the load through a single path or page, flooding parts of the application in unusual ways while ignoring other parts. It often fails to catch the worst performance issues. Sites that repeatedly tune for the same few test cases end up hyper-tuning the site for one workflow and neglecting important performance issues. #4 Measuring with response time only Most automated performance testing tools measure response time as the primary metric. Response time indicates whether there is a problem or not. It doesn t tell you anything about what to do or how the system is operating end-to-end. Couple this with the other constraints, and the potential for failure is high. For example: - Are load balancing and failover tuned to respond properly? - Is one app server handling a disproportionate amount of traffic? - What are the primary backend contributors to overall response time database time, web and application server time, network contribution? - Are errors occurring in other components that might affect performance over time or at higher traffic levels? - How do 3rd party services like CDNs, transaction processors and ad servers play in the mix? Next-generation testing technologies lift constraints & change strategies A new generation of testing tools is removing the traditional limitations on virtual users and easing the constraints on hardware, making it possible to abandon outdated testing work-arounds. Today's testing tools can leverage cloud ecosystems like Amazon EC2 and Windows Azure, as well as private clouds, to generate load around the globe in minutes. Because of the cost efficiencies of the cloud, you can afford to scale out to actual expected load. Virtual users no longer need to be a gating factor in tests. It's also easy to test against production infrastructure using some combination of testing during light usage windows or maintenance windows and testing against a large slice of the production environment. By using the actual production infrastructure, you can test the entire application flow, and not miss components that otherwise fall outside a carefully constructed test environment.

5 Let's look at how these new realities affect the old, work-around strategies: Think time: If virtual users aren't a gating factor, you can use as many as you need, with realistic and randomized think times to more accurately simulate load. Extrapolating capacity: By leveraging the cloud, you can tap into a virtually limitless pool of load generation hardware, testing against production infrastructure. Today's test tools let you spin up and shut down virtual users quickly, with the ability to monitor and kill tests in real time, limiting risk to production environments. Without constraints and with total control, tests of 100% capacity are now a requirement. Most companies find that testing to well beyond their highest expected capacity is highly valuable for estimating worst- or best-case scenarios. Use cases: With faster, less expensive performance testing, you can test more use cases, creating realistic and varied workloads. Creating the actual tests can be as simple as running commands and capturing the generated HTTP to create a test script, leaving in or modifying actual user think time to reflect reality. Prioritize the most important test cases, and then continue iterative testing. Metrics: Today's test tools offer powerful analytics that let you drill down beyond response time to other metrics collected from the deepest database out to a virtual user in the farthest reaches of the globe. Best of all, you can do so while the test is executing, giving you the insight to fine-tune and adjust in real time as tests run. Figure 2. Today's real-time analytics can look beyond simple response time metrics to discover what's really happening during performance tests.

6 It's time to abandon outdated testing strategies The four testing work-around strategies were a necessity of their time. However, they are inadequate for predicting the kinds of performance problems that today's application complexity and massive user loads can inflict. There's no reason to keep guessing when the fundamental testing constraints have changed. Hardware constraints are minimized when you can test on the actual infrastructure, and virtual users are no longer a gating factor, so there's no longer any reason to: 1. Modify think time to stretch virtual users rather than testing with the right number of virtual users 2. Extrapolate capacity rather than testing to capacity 3. Test only critical pages or silos rather than the entire workflow 4. Measure performance by response time alone, rather than looking at all parts of the infrastructure during tests Next-generation performance testing tools, with powerful analytics and the ability to leverage cloud infrastructure, can provide highly scalable testing at a very low cost. Companies of all sizes can test web applications at scale, without huge hardware investments and teams of mathematicians. Test engineers still contribute important expertise and guidance, but given better tools they can deliver more valuable results, with less uncertainty and risk. For a deeper discussion of this content, watch the webinar Stop Cheating and Start Running Realistic Tests available from the SOASTA website. About SOASTA SOASTA's CloudTest products and services leverage the power of cloud computing to quickly and affordably test consumer-facing Web and mobile applications at scale, providing customers with the confidence that their business-critical Web applications will be reliable and scalable, even under extreme loads. SOASTA s customers include many of today s most successful brands, including American Girl, Chegg, Gilt Groupe, Hallmark, Intuit, Microsoft and Netflix. For more information about SOASTA, please visit