Cloud Testing Bill of Rights:

Transcription

1 White Paper Cloud Testing Bill of Rights: Top 10 Decision Criteria for a Web & Mobile Application Testing Platform By Scott Barber, Chief Technologist, President & CEO, PerfTestPlus, Inc.

2 Table of Contents Why a Cloud Testing Bill of Rights... 1 Your Cloud Testing Rights... 1 Load Generation... 2 I. The flexibility to generate load both internally and externally... 2 II. Elastic, globally distributed test environment... 2 III. Dynamic and fully automated test grid provisioning on demand and within minutes... 3 Speed and Completeness... 3 IV. The ability to build and modify complex tests within hours... 3 V. Support for dynamic web and mobile technologies and protocols... 3 Control... 3 VI. Control test sessions to start, pause, stop, modify, re-start and analyze... 3 Analytics... 4 VII. Single integrated view of performance and monitoring metrics... 4 VIII. Rapidly isolate performance issues in real time at any scale... 4 Availability and Flexibility... 4 IX. Test platform that allows functional and load testing from lab to production... 4 X. Affordable enough for everyday testing... 5 Conclusion... 5 About the Author... 5 SOASTA, Inc. WHITE PAPER

3 Why a Cloud Testing Bill of Rights? When the governmental structure of the United States was initially put into place, it did not include the Bill of Rights. While many of the Founding Fathers were far from pleased about the absence of explicit citizens rights in the Constitution, they knew that if they didn t get the basic governmental structure in place quickly, the new country wouldn t last long enough for citizens rights to matter. This sounds very much like how development and IT groups were formed, with quick and dirty organizational structures that made sense to someone in the moment. My collaboration with SOASTA began in the summer of 2011 when I was asked to preview a beta version of CloudTest Lite. The positive experience I ve had with both the product and the SOASTA team has enabled our work together to continue. If you are familiar with my work then you know that I don t commonly write about vendors or tools aside from deliberately neutral educational pieces. SOASTA has proven to be one of those rare vendors that I find myself able to speak freely about without violating what mom taught me about refraining from saying anything at all if you don t have anything nice to say. This begs the question: In your development or IT group, do you feel like you have the right to voice dissatisfaction and organize to champion change? If your experience is like mine, your answer is that depends on what we re dissatisfied with and what change we are championing. I can think of no techniques, ideas or practices related to testing, which require support from management, that get shut down more quickly than those that include the cloud. The cloud is gaining a degree of credibility when it comes to operations, but when it comes to testing, there are a disproportional number of vociferous detractors as compared to champions. I have to wonder if the detractors are being driven by fear: fear of change and fear of being rendered irrelevant by advancement. This observation leads me to see the Cloud Testing Bill of Rights as a series of proposed revisions to commonly accepted software development practices. They ensure citizens of the nation of software development the freedom of expanded options and the right to pursue betterment of their current situation. These revisions are specifically related to using cloud-based technologies to improve and/or simplify application performance testing. Your Cloud Testing Rights I m hard pressed to imagine many individuals who would object to having the right to pursue these items appropriate to the needs and context of their project, application or organization without fear of being labeled a troublemaker. I. Is flexible to generate load both internally and externally II. Provides an elastic, globally distributed test environment on demand and within minutes III. Has dynamic and fully automated test grid provisioning SOASTA, Inc. WHITE PAPER 1

4 IV. Has the ability to build and modify complex tests within hours V. Supports dynamic web and mobile technologies and protocols VI. Controls test sessions to start, pause, stop, modify, re-start and analyze VII. Has a single integrated view of performance and monitoring metrics VIII. Rapidly isolates performance issues in real time at any scale IX. Allows functional and load testing from lab to production X. Is affordable enough for everyday testing In case these rights don t strike you as important or it s not immediately obvious to you what problems they address, read on. Load Generation A cornerstone of a business creating value by adding application performance testing is the ability to exercise the application under test with realistic loads. Realistic doesn t only refer to the ability to deliver the correct volume, but also to deliver that volume from the places where actual users will be using your application. The Cloud Testing Bill of Rights addresses this with three specific rights and further clarifies them with capabilities of their tool/service. Take the time to consider these before deciding which tool to use for your application performance testing needs. I. The flexibility to generate load both internally and externally I often talk about the benefits and limitations of performance testing internally, externally and from an end-user s perspective. Each of these perspectives add value that the other two don t, meaning that failing to consider and account for each of these will ultimately lead to unwanted surprises. Internal load generation is the most effective option for early performance assessments and for continually assessing and tuning hardware and software that will live in your hosted environment. External load generation is necessary to determine the impact of things like firewalls, content delivery networks, external components of your site or application and geographic proximity to the data center. Taking measurements from a true end-user perspective, or Last Mile testing, is necessary to assess and tune the front-end aspects of the site or application and determine actual user experience. When it comes to load generation, in almost all situations you ll want the ability to generate load both internally and externally. If you re part of an organization that hosts every component internally and access to those sites or applications will be available only to folks on the same internal network, then there is no need for external testing. There is rarely a need to generate load from actual client machines. Generating external load from the cloud while observing a sampling of end-user interactions is entirely sufficient, especially when front-end or user experience testing of the interface is already part of your overall testing program. II. Elastic, globally distributed test environment The specific numbers of locations or virtual users aren t what matter here. What matters is that you have the right to be able to generate realistic, production volume loads from geographies representative of your actual user base. While generating load externally from across the street is better than only being able to generate internal load, it s not nearly as valuable as concurrently generating load from a variety of geographic locations. It s far more challenging today to predict performance from a particular geography than it was even five years ago. And unlike five years ago, there are far more options available to the producers of sites and applications to improve performance in a particular geography. Implementing a cloudbased technology for load generation means that you no longer have to wait for users to complain before becoming aware of a problem. Cloud-based load generation can enable you to assess performance by user location by clicking a couple of extra buttons and running a couple of extra tests. SOASTA, Inc. WHITE PAPER 2

5 III. Dynamic and fully automated test grid provisioning on demand and within minutes Very much related to the previous two rights, you have the right to mix and match the generation points for your load quickly and simply. Having the ability to generate load from anywhere isn t enough, and isn t unique to cloud-based solutions. I, for one, was doing that ten years ago. Of course, it always seemed to involve a month to ship hardware with the right software pre-installed to remote locations, or to talk to people working in those remote offices through installing and configuring software they ve never heard of, coordinate network connectivity and the other hundred details. And all this before even trying to execute the first test. Imagine the reaction you d receive if you suggested such a solution today. Speed and Completeness Second only to a lack of sufficient load is the inability to generate realistic loads quickly enough to be relevant without needing to learn a new tool each time a new technology is implemented. Performance engineers have long been begging for rapid testing tools that can be used with a wider range of technologies. Today, traditional testing tools continue to take weeks and months and only test older technologies. Two rights speak to today s performance testers not being condemned to telling those same war stories ten years from now. IV. The ability to build and modify complex tests within hours Clearly, this right is not exclusive to cloud-based performance testing technologies. Rather, you have the right to demand this from your tool or service provider no matter what technology the tool or service is built on. Development teams and/or managers don t have days, let alone weeks, to wait for performance assessments. They have the right to results in enough time to do something more useful than shrug, mumble something about wishing they d known sooner and logging an action item for the next release. The only hope we have of providing results on their timeline is if we can build and modify realistic tests, complex tests, in hours or less. V. Support for dynamic web and mobile technologies and protocols One could say that we have the right to demand that our tools provide support for new technologies and protocols. At the time of this writing, the most challenging of those from a load generation perspective is related to asynchronous yet inter-related use of both client- and server-side communications within a single site or application. Next year, it may be something entirely different. What will remain the same, however, will be our desire to not purchase and learn a new tool or service due to a technological advance. Of course there will always be bleeding-edge technologies that tools don t support. And when one of those technologies catches on, vendors need not charge us obscene amounts of money for updates. The fact that some vendors still expect their loyal customers to pay exorbitant fees to Ajax or SOAP support, protocols that crossed from bleeding edge to simply not behind the times five years ago, is simply criminal. We have the right to be treated better than that. Control Following the ability to create and implement tests quickly, and building tests against leading technologies without purchasing new tools, tool extensions are the ability to control performance test execution. This is key to the accuracy and efficiency of the performance test effort. VI. Control test sessions to start, pause, stop, modify, re-start and analyze There was a time when it was a rare tool that allowed expert performance testers to write custom code to give them the kind of detailed control they needed to maximize the accuracy and efficiency of their tests, especially during test execution. Today, there is no excuse for making these testers spend weeks or months figuring out how to hack a tool so they can accomplish tasks like changing load levels on demand during test runs. There are plenty of tools on the market that come with this kind of control built in. SOASTA, Inc. WHITE PAPER 3

6 This speaks to performance testers having the right to spend the majority of their time doing performance testing instead of working around tool limitations. Any organization disabling their performance testing program by forcing their testers to work around their tools today is spending far more money in terms of wasted time, unfound performance issues and under-tuned sites and applications than it would cost to switch to a tool or service that just works. We most certainly deserve the right to ask our companies to stop wasting money. Analytics When I was starting out as a performance tester, virtually all performance analysis was done manually. By manually, I mean manually configuring logging software on every server, manually parsing log files for data of interest, manually massaging that data into the same spreadsheet of data that was manually exported from the load generation tool to manually create charts and graphs so the team could manually analyze the data in the hopes of finding some kind of pattern telling enough to tune the logging or modify the next test. VII. Single integrated view of performance and monitoring metrics Year after year my pleas for this capability fell on deaf ears. Today, most load generation tools come with built-in monitoring. Many even come with real-time, integrated monitoring on a single, customizable dashboard. To me, this is nothing short of a dream come true; for any performance tester this is at least a right, if not an expectation. Aside from saving an unbelievable amount of time, the increase in value per test run is off the charts. Again, the specific monitoring software isn t what s relevant to your rights, it s the right to have monitoring software that works as part of your performance testing program that matters. VIII. Rapidly isolate performance issues in real time at any scale I remember the first time I saw a demo of a tool that traced packets through the system and reported response time by layer, transition and/or activity. I was amazed! It was five years later before I had a chance to use one for a real performance testing project. Today, these tools are commonplace, affordable, approachable, and even easy to install/configure. In fact, many load generation tools come bundled with these capabilities. The ability to isolate performance issues in real time not only saves time, as the right s title implies, it all but eliminates the it s your tool, not my application response, enabling the team to get right to work on resolving issues instead of wasting time sometimes weeks of time arguing about whether or not there is an issue to fix. Availability and Flexibility The overwhelming majority of my clients performance testing efforts are completely hamstrung by tools, services and license agreements that make it impossible to do continual performance testing. This situation leads to the propagation of the belief that performance testing can be effective as a last-minute task. The reality is that treating performance testing as an isolated afterthought leads to unnecessary heroic efforts and frequently to epic failures in production. This is a sad and decidedly unhelpful state of affairs. IX. Test platform that allows functional and load testing from lab to production Good performance testing begins when a project is conceived and ends when a site or application is retired. Sites and applications that achieve consistent acceptable (or better) performance integrate performance testing into every aspect of design, development, deployment and maintenance of their product. It used to be that doing this took a large and expensive toolbox as well as a small army of tool specialists. That is no longer the case. With modern cloud-based performance testing tools, activities that used to be exceptionally painful, like moving a test from the lab environment to a staging or production environment, take minutes instead of days or weeks. Combine this with capabilities of individual tools to perform multiple functions means that fully and seamlessly integrating performance testing is entirely viable. Once again, we have the right to demand this kind of simple and seamless integration of performance testing into the entire product lifecycle and the right to demand that vendors support our efforts in this regard. SOASTA, Inc. WHITE PAPER 4

7 X. Affordable enough for everyday testing This right simply cannot be overstated. For the majority of my career as a performance tester, only the richest and most elite software development organizations were able to justify the expense of buying or leasing the software and infrastructure necessary to run tests at will. Shared licenses, temporary environment provisioning, and administrative overhead has been the bane of a performance testers existence for years. Any organization that is still facing this situation is doing so out of negligence. Advances in cloud-based technologies and pricing models have all but eliminated the need for additional hardware and infrastructure to be procured and maintained internally. This has reduced the total cost of enabling on-demand, daily performance testing, conducted by any member of any team, without the need to cross-coordinate hardware or license availability. Add that to the savings that an organization will realize by deploying and maintaining well-performing applications, and it seems nothing short of irresponsible to continue paying so much more for so much less value. Conclusion Cloud-based performance testing technologies have converted the dreams of many old-school performance testers into the rights of today s performance testers. Failure to take advantage of these rights wastes time and money. It ensures acceptable site and application performance will be sub-par, prohibitively time consuming and/or expensive, or result from sheer dumb luck. In short, not knowing and taking advantage of your rights as described by the Cloud Testing Bill of Rights is nothing short of irresponsible. About the Author A prominent thought-leader in the area of software system performance and testing software systems, Scott Barber, founder and Chief Technologist of PerfTestPlus & Director of the Computer Measurement Group, makes his living writing, speaking, consulting, and coaching with the goal of advancing the understanding and practice of software testing. Scott has contributed to four books, composed over 100 articles and papers, delivered keynote addresses on five continents, served the testing community as the Executive Director of the Association for Software Testing, and co-founded the Workshop on Performance and Reliability. Contact us: Castro Street Mountain View, CA To learn more visit: soasta.com or us at info@soasta.com 2012 SOASTA. All rights reserved. SOASTA, the SOASTA logo, and SOASTA CloudTest are trademarks of SOASTA. All other product or company names may be trademarks and/or registered trademarks of their respective owners. EST Connect with us: WHITE PAPER 5