8 Common Myths About Performance Testing in Production Don t Let These Misconceptions Keep You From Production Testing.

Transcription

1 8 Common Myths About Performance Testing in Production Don t Let These Misconceptions Keep You From Production Testing White Paper

2 Table of Contents Myth #1: Production testing = live testing... 1 Myth #2: It s too risky... 2 Myth #3: Production testing is all about breaking the site... 2 Myth #4: It takes too long we re lucky to have time for QA testing... 3 we don t have the expertise... 3 Myth #6: Production testing isn t necessary if we do thorough testing in the lab... 3 Myth #7: I ll never get management buy-in... 4 Myth #8: We don t have the budget it costs too much... 4 Summary:... 4 About Soasta, Inc.:... 5 SOASTA, Inc. TESTING IN PRODUCTION MYTHS WHITE PAPER

3 In today s web environment, performance matters a lot. And one of the best ways to ensure that your application will perform as expected, under production loads, is to do production performance testing. This entails testing the application with production-level load in the actual environment in which it operates. ACCORDING TO GOOGLE, AN EXTRA 500MS OF LATENCY COSTS THEM 20% OF THEIR SEARCH TRAFFIC. Testing your application in the production environment uncovers performance issues at scale that simply don t exist in more limited lab tests. It calls on all of the components of your application ecosystem, including outside factors such as third-party services or content delivery networks (CDN). In addition, performance testing in the production environment uncovers problems outside the application itself but within your application infrastructure, such as: and bandwidth limits If you don t do performance testing in the production environment, then you re de facto outsourcing that testing to your production users which is probably not a wise decision. Despite its clear advantages, a surprising number of companies fail to performance test their applications in the production environment today. In part, this is due to misconceptions or myths about production testing that keep companies from even exploring the possibilities. This paper aims to dispel those myths and give you a better idea of what is possible with today s. Myth #1: Production testing = live testing users. Certainly, some companies regularly do production performance testing on their applications while real users are present, but there are many other ways to test on the production infrastructure, without risk to live data or actual users. Test during maintenance windows: If you have the luxury of scheduled downtime, test during this window. You need to work with the responsible teams to carefully plan and ramp the test Test before release: Test an application in the production environment before it s announced here a requirement. Involve all core team members across development, test and operations in these test events and soon the value of production testing will be clear to your entire organization. SOASTA, Inc. TESTING IN PRODUCTION MYTHS WHITE PAPER 1

4 Test on a slice of the infrastructure or on your redundant system: section of the production infrastructure that contains a representation of all of the components appear in a lab. Alternatively or additionally if you have a fully redundant system, you can technique is used by Amazon and Google and serves the dual purpose of also testing your production, keep reading. Myth #2: It s too risky Production testing simply sounds risky. But if you think about it, it is much less risky to run a that are easily replicated and preventable. The risks of performance testing in production can be mitigated by: A real-time dashboard gives you insight to application infrastructure performance. Myth #3: Production testing is all about breaking the site When you think of performance testing, do you think of testers trying to break the site? While stress and failure testing are clearly performance testing strategies, other types of performance testing are more common and less destructive: Baseline testing: Establish the level of load that the application can handle while meeting Spike: Simulate high, fast variance from normal load staying well under expected full capacity. Endurance: Diagnostic testing: to infrastructure. SOASTA, Inc. TESTING IN PRODUCTION MYTHS WHITE PAPER 2

5 Within production testing, you can measure things like: Transaction: How long does it take to complete a transaction under varying circumstances? Code-level performance: How fast are individual pieces of UI or Server code? Is there contention? Mobile performance impact: Does having many transactions over slower Edge or 4G networks result in more open connections than your application can handle? Site failure would be an unexpected outcome from any of the above tests. Performance testing is not all about breaking the site. Myth #4: It takes too long we re lucky to have time for QA testing This excuse might be valid if you re only planning to do production infrastructure testing before the application is live. But, as we saw when discussing Myth #1, it s possible to extend the test window and do performance testing when the application is in production, using maintenance windows, slices or silos of the infrastructure, or even low-usage times. environment, does not need to be time-consuming and complex. New cloud-based testing tools let you ramp up a global community of virtual users in a matter of minutes. And creating the actual tests can be as simple as running some commands and/or interacting with a mobile device and capturing the generated HTTP to create a test script. In fact, running smaller iterative tests with incremental objectives can allow more testing in the same or less time. Once this may have been true. But the technology available for performance testing has changed. New cloud-based testing applications provide you full control and visibility that make it simple to create the tests you need, spin up thousands of virtual users, and launch and closely monitor real-time results during tests as they run. But that s just the technology end. Any kind of performance testing requires planning, and production performance testing requires more than others. You should involve several key teams in your organization to plan and monitor the tests. One strategy is to start with those kinds of test that are easier to execute while still delivering a lot quite a bit about your site performance under load. Testing transactional scenarios requires more work to develop the test data and back out or otherwise negate testing transactions. As a result, transaction testing might wait until you have some success with other types of performance testing. Once your entire organization begins seeing the results from production testing, it s easier to gain buy-in for testing further, and often leads to modifying your applications and operations to make production performance testing easier and a core operational activity. Myth #6: Production testing isn t necessary if we do thorough testing in the lab As we ve already mentioned, testing in a lab environment is great for testing the application itself, Testing in production does not replace lab testing or testing in a staged environment but it can uncover problems that other tests cannot, including: Unbalanced web servers DNS routing problems Network bandwidth issues Mobile network performance issues SOASTA, Inc. TESTING IN PRODUCTION MYTHS WHITE PAPER 3

6 AMAZON LOSES 1% OF ITS SALES FROM EVERY 100MS OF LATENCY. Worried about those third-party application providers if you do a performance test? Choose testing approaches that enable proper load distribution, good monitoring and tight control over test execution to do this, and then check in with them ahead of time. They may well be happy to participate in the performance test and have ways to handle the testing on their end. Because ultimately, no one wants to be surprised by unexpected glitches in production. Myth #7: I ll never get management buy-in There are two key strategies for getting management buy-in for production performance testing: 1. Wait until there s a major problem, then suggest production testing. assessment and mitigation strategies, and get all of the relevant teams on board. you re trying to avoid. So, let s get right to #2. teams will embrace the value of performance testing in production. Production performance testing is in fact a wise risk mitigation strategy for the business as a whole, and should be presented as such. Myth #8: We don t have the budget it costs too much With traditional testing techniques, this is true. Building a test lab to support the application under To generate user loads common to most consumer sites with open source or legacy test tools requires hundreds or thousands of load servers again, an immense cost. Add to that the cost of test software licenses ample to drive a test of even moderate scale, plus the people to build and analyze the tests, and you quickly reach millions of dollars to test for just thousands of users. Again, existing tools that recognize the need to test the latest web and mobile technologies to realistic usage levels are reshaping the game when it comes to production testing. Using cloud platforms like Amazon EC2, IBM Smart Cloud, Rackspace Cloud, or Windows Azure, you can pay for only the capacity you need to run test platforms around the globe and at scale. To test mobile and changing diversity of the mobile device market. But of course there are other costs as well, in addition to the tools. If you need infrastructure you have and prevent you from investing in the wrong places. If problems arise outside of testing, your team will spend more time trouble- shooting and Performance testing helps you hold third-party applications accountable for their promised service levels. Minor changes in performance can have a major impact on your bottom line. Summary hopefully the ammunition to move forward with production testing plans. You ll be joining the rapidly growing number of companies that are running performance tests in a production environment. At SOASTA, we work with thousands of customers in the e-commerce, to implement performance test plans including production testing. With cloud testing services, low-risk manner. SOASTA, Inc. TESTING IN PRODUCTION MYTHS WHITE PAPER 4

7 About SOASTA, Inc. SOASTA is the leader in cloud testing. Its web and mobile app test automation solution, the CloudTest Platform, enables developers, QA professionals and IT operations teams to test with unprecedented speed, scale and precision. The innovative product set streamlines test creation, automates provisioning and execution and distills analytics to deliver actionable intelligence faster. held and headquartered in Mountain View, Calfornia. Contact us: Castro Street Mountain View, CA To learn more visit: soasta.com or us at 2012 SOASTA. All rights reserved. SOASTA, the SOASTA logo, and SOASTA CloudTest are trademarks of SOASTA. All other product or company names may be trademarks and/or registered trademarks of their respective owners. EST Connect with us: TESTING IN PRODUCTION MYTHS WHITE PAPER 5