Research Publication Date: 15 September 2009 ID Number: G00170552 What Is the Role of Quality Assurance in a SaaS Environment? Thomas E. Murphy, Daniel Sholler, Christian Hestermann Software as a service (SaaS) promises a mixture of flexibility and reduced operations costs, but it is misleading to think that all testing responsibilities go away. Understand the key practices and requirements for managing SaaS solutions. Key Findings Based on marketing messages by vendors, users are lured to assume that using a SaaS solution requires less testing and quality assurance (QA) efforts than when used on-premises, which is not true. Users of a SaaS solution have even less control over the change stream applied to their solutions; in many cases, they might not even know in detail what was changed, nor do they have visibility into the QA effort (or lack thereof) that is applied when change occurs. The changes can theoretically appear at any time, and the only way to not accept a change to the solution is to stop using it. This is unacceptable for core business applications like ERP. Recommendations Do not simply expect the vendor to be more accurate than you are, but understand where risk areas lie and the contractual or test mitigations you will need to apply. Make sure you have a good channel to vendors to notify them of issues, and to ensure that you have a way to have early access to changes. Establish error-handling and problem resolution processes as a part of negotiating the SaaS agreement. Think of SaaS as a partnership where you are helping vendors with early feedback and reports, and they are building enhancements and resolving issues. Reproduction and distribution of this publication in any form without prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Although Gartner's research may discuss legal issues related to the information technology business, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof. The opinions expressed herein are subject to change without notice.
ANALYSIS Software quality and testing is a constant source of grief in the production of software. When QA finds a defect, it delays the delivery of the software, and when a defect slips through, QA didn't get the job done. All in all, everyone looks for ways to reduce the cost of testing and seems to harbor the desire to forget about it all together. This is driving solid growth in off-shore outsourcing of testing, and has been one of the desired values of utilizing packaged software. Now, SaaS proposes to reduce the operating cost of software, on top of the development and testing savings of packaged software. However, there is a distinct difference in current SaaS offerings the loss of connection to the change stream. While the focus on testing of software varies greatly across organizations, with traditional software delivery, the consumer (or implementation partner) has some level of control over changes to the system. In a traditional upgrade, users should understand that they have to extensively test the versions before putting them into production. In a SaaS application, it is easy to be misled into neglecting testing because the upgrade happens in a transparent way. Indeed, vendors tend to make prospects believe that with a SaaS solution testing the stability and performance of a newer version is no longer needed. However, depending on the nature of the changes or how the SaaS or platform as a service (PaaS) solution is extended or integrated into other systems, customers may have to spend the same amount of diligence and prudence as with any other business-critical application (see "Ten Challenges for Successful Composite Multienterprise SaaS Integration"). But, without access to the change stream, instead of potentially having changes (and, therefore, bugs) introduced on a periodic basis, they can be introduced daily (and without your knowledge). As a whole, traditional package software vendors have not had a great record when it comes to change management and upgrade support, which does not bode well for the transition into cloud-computing platforms. Understand the Cost-Benefits Testing is a cost-benefit exercise. You can never test everything, because everything would literally be everything. A combinatorial number of possible uses, failures, data inputs, etc., that would theoretically need to be tested. The majority of organizations instead test based on the risks they are trying to control. Moving to the subject services, the picture seems to remove costs, but there are hidden costs here that you must be aware of that result from the loss of quality and change management control. So, the question in the SaaS model is that since you do not know the change stream, is the risk that a vendor-applied change will create an error in your process something that you want to spend time and money to control, or not? Most people will choose not to make these expenditures, because it means not accepting the change until it is tested, and that implies not using the system (since you do not have control over when the change is applied) in the meantime. With traditional software packages, you have some access and control over what is changing and an ability, through your own knowledge or a consulting party, to understand the risks and how to mitigate them. In SaaS, we are led to believe that all is good, but no software is bug free and changes to vendor-provided software have traditionally caused headaches for users. So, make sure you carefully measure the impacts and value of customization and integration, and how changes will affect the cost of using the software. Don't Lose QA Control The real discussion regarding testing and SaaS is what can you control? Obviously, you need to test those things that are affected by your configuration choices. You also want to periodically test nonfunctional aspects (performance, availability, security, recovery, etc.). The more you can automate this process, the better you can make a test harness into a change notification system. Publication Date: 15 September 2009/ID Number: G00170552 Page 2 of 6
But what happens beyond that? You cannot change the functionality of the product, so it's not worth testing, in most cases. This raises a dilemma when it comes to regulated financial systems where auditors will check both the correctness of the system and how it's used. Again, testing ends up being based on risk. Ensuring Compliance Vendors will have to take on a greater responsibility for the U.S. Sarbanes-Oxley (SOX) Act and other regulatory compliance issues in SaaS delivery. However, anywhere you can change a calculation or a workflow, or anywhere data comes out of the system and then re-enters it, requires end-user validation. Users will need to add regulatory issues as part of the buying criteria, e.g., has the vendor been certified? Certification of the vendor's practices, security, load/stress, and compliance should all be part of your decision on whether to go to SaaS. You are giving up control of these items and are implicitly trusting the vendor. Another potential area of concern is the quality of the vendor's change control practice and how it notifies users. When the large offshore outsourcing vendors started up, they sought out Capability Maturity Model Integrated (CMMI) Level 5 and International Organization for Standardization (ISO) 9001 compliance as a way to show they were trustworthy and reliable. If you are expending the effort to be Information Technology Infrastructure Library (ITIL)-compliant in your own operations, do you also expect it from your service provider? With financial systems, another challenge is that the fundamental nature of the application is correct, but something about the specific data set utilized causes an issue. In software testing, 100% path coverage may be obtained, but it is impossible to test all valid/invalid data inputs. In general, you are responsible for your data and for ensuring that it works correctly with the software, not the other way around. Auditors will not accept excuses or claims that you don't know what happened to the application. Change Management With traditional business application software in the field, the tools to manage change sets and testing are improving in response to changes creating upgrade issues for customers. For instance, SAP Solution Manager or other SAP change tools identify what has changed, which tests need to be rerun, etc. In SaaS, we will either need to get to that point, or strongly limit customization and integration. It also means that the test team should have a good set of automation around the key transactions, integrations and "customizations," and should run those potentially every day. These tests become an internal change notification system. But that still leaves a big window, when a change can occur and your software breaks. This is a risk you are agreeing to live with, unless you contractually protect yourself from vendor change without notice. This risk is currently mitigated by service-level agreements (SLAs). Most companies are not concerned about software quality; they are concerned about risk management, and the goal is to spend the least you can to have a level of risk you are willing to live with. In moving to SaaS, firms are trying to reduce testing costs, not increase them. The less you customize and the fewer integrations there are, the less need there is for concern about change management or testing. However, in all cases where the software in enterprise deployed, mission-critical, understanding the vendors change management and operational procedures and building a selected set of validation tests around core functionality is prudent. Costs can be controlled by limiting customization and integrations. Limiting these will curtail the need to be concerned that vendor updates will break your system and reduce the need for concern about change management or testing. However, in all cases where the software is enterprise-deployed, or mission-critical, understanding the vendors change management and operational procedures and building a selected set of validation tests around core functionality is prudent. Publication Date: 15 September 2009/ID Number: G00170552 Page 3 of 6
Application PaaS Note that PaaS/application platform as a service (APaaS) means that you take more responsibility back, and this "complicates" issues. Rather than just being software that you run (e.g., salesforce.com) with minor configuration changes, it is a hosted development platform (e.g., Force.com). In this case, you have software developed on top of the platform, and platform version changes could break functionality. In a traditional development world, this is similar to having a new version of Java or.net-only traditional applications built on multiple versions of.net, or Java can exist (along with the associated runtime) on a single machine with a servicebased solution there is only the service as it currently provided by the vendor. This creates the potential for breaking existing integrations as well as user skills if there isn't a provision for backward compatibility or support from the vendor to run an older version of the application. There are connections between this and testing for SOA and SaaS. This is especially true if the use of services is primarily externally provided services, such as credit authorization, payroll, shipping, etc. When the vendor changes a service (maybe not even a functional service), the performance characteristics change, and the system must deal with this (see "APaaS: A Step to a 'Killer App" for Cloud Computing"). An Example SaaS offers the benefit of helping you avoid upgrade pain; you just gain new features. For example, you don't worry about your Google docs working tomorrow, but you hope for improved functionality. One advantage that vendors gain in Web applications is knowing exactly how many people use specific functionality. Microsoft has been able to do this with its applications whenever you permit by selecting the "help provide feedback" option on product installation. This feedback lets the product manager understand how many people use a function, how often it is utilized, the potential problems that may occur, etc. This feedback may show the need to change a feature for usability, or show a feature that isn't "widely" used (but is still critical to you), and in the vendor's return-on-investment (ROI) assessment doesn't continue to make the cut. A major issue occurred when Microsoft introduced version 4 of Excel. Prior versions supported macros in foreign languages, e.g., German, so you could say "Berechne <xyz>" instead of "Calculate <xyz>." But in the next version, this support was dropped, so users had to manually go through and redo all their macros. Because the software was locally installed, users could control the rollout of the change and potentially revert to an earlier version while the broken formulas where updated. Returning to the example of Google docs, basic functionality should be okay. However, if you can use macros, formulas etc., then you hope they still deliver exactly the same results as before because you don't have a way to revert the change. Thus, when you move to SaaS, the pace of change is no longer yours to control, and your organization has to be able to respond to that change as quickly as possible. Because you can't protect yourself from change, a plan should be put in place to ensure a quick response in addressing issues that arise due to vendor changes. A critical element of this plan should be the establishment of a key internal contact person who works directly with the vendor when issues arise. We believe that these mechanisms will evolve, and we have seen in the world of constant betas the Web has delivered that there will be ways to opt into early access to changes. Vendors need the ability to understand the effects of these changes, and the only way to get the combinatorial advantage is to put users on the software. Do not simply expect the vendor to be more accurate than you are, but understand where risk areas lie and the contractual or test mitigations you will need to apply. In addition, make sure you have a good channel to vendors to notify them of issues, and ensure that you have a way to have early access to changes. Think of SaaS as a partnership where you are helping the vendor with early feedback and reports, and where they are building enhancements and resolving issues. Publication Date: 15 September 2009/ID Number: G00170552 Page 4 of 6
Who Has Ownership of Defect Management The issues of defects in off-the-shelf software and SaaS-provided solutions aren't really any different. The vendor is responsible for the defects in its software, and the user is responsible for the defects in any customization or integration. However, we believe that SaaS providers will be more willing and able to provide transparency regarding defect information than traditional vendors. Because these vendors are trying to disrupt the market and tend to make use of moreagile practices, and because they will need to evolve solutions to manage the impact of change on software users, of it will be in the vendors' best interest to make use of a more public stream of information. However, this will still require user resources to monitor and understand the impact on help desk trouble tickets, defects and feature requests, and to ensure that changes don't negatively impact running solutions. Bottom Line When moving to SaaS, don't expect defects to magically disappear. Even if you don't know exactly what has changed and cannot avoid using the wrong "version" until it is corrected, a minimal testing program will mitigate risks and help avoid damage. RECOMMENDED READING "Software as a Service: Component Development Challenges" "SAP Business ByDesign Offers Broad Functionality, but Depth and Integration Need Improvement" Publication Date: 15 September 2009/ID Number: G00170552 Page 5 of 6
REGIONAL HEADQUARTERS Corporate Headquarters 56 Top Gallant Road Stamford, CT 06902-7700 U.S.A. +1 203 964 0096 European Headquarters Tamesis The Glanty Egham Surrey, TW20 9AW UNITED KINGDOM +44 1784 431611 Asia/Pacific Headquarters Gartner Australasia Pty. Ltd. Level 9, 141 Walker Street North Sydney New South Wales 2060 AUSTRALIA +61 2 9459 4600 Japan Headquarters Gartner Japan Ltd. Aobadai Hills, 6F 7-7, Aobadai, 4-chome Meguro-ku, Tokyo 153-0042 JAPAN +81 3 3481 3670 Latin America Headquarters Gartner do Brazil Av. das Nações Unidas, 12551 9 andar World Trade Center 04578-903 São Paulo SP BRAZIL +55 11 3443 1509 Publication Date: 15 September 2009/ID Number: G00170552 Page 6 of 6