Five Must-Do s For Testing Quality At Speed

Transcription

1 For: Application Development & Delivery Professionals Five Must-Do s For Testing Quality At Speed by Diego Lo Giudice, January 23, 2015 Key Takeaways Don t Compromise Between Speed And Quality Any Longer The demand for speed and quality is unprecedented, and continuous delivery is the Holy Grail in terms of delivering new and better applications and features to your business faster. This affects all aspects of testing and means an unprecedented level of change. It s time to jump-start a new testing process or accelerate the one you have now. Support Your DevOps And Continuous Delivery Initiative With Continuous Testing Continuous testing is crucial for large-scale Agile adoption and is necessary for DevOps and continuous delivery. To create a continuous delivery capability, you must create a continuous testing one: Kick testing off at the beginning of each project and make it iterative, incremental, and continuous throughout the entire project life cycle. Today s Five Testing Must-Do s Are Tomorrow s Mobile-First Testing Practices Our five must-do s for testing quality at speed will help you improve your current delivery practices and gain some quick business wins. They will also set you on the right path for your mobile-first strategy, which you will need to develop soon. If you are already doing mobile-first, check our five must-do s to ensure you re not missing any. Forrester Research, Inc., 60 Acorn Park Drive, Cambridge, MA USA Tel: Fax:

2 January 23, 2015 Five Must-Do s For Testing Quality At Speed The Convergence Of Speed And Quality In Modern Application Delivery by Diego Lo Giudice with Holger Kisker, Ph.D,, Eveline Oehrlich, and Sophia Christakis Why Read This Brief Oh no! Our delivery date is just around the corner but we still have QA to run... let s push this to production anyway and deal with it in the next release. Sounds familiar? As digital disrupts business, we re seeing an unprecedented demand for speed, while quality, of course, is a given; as a result, old ways of testing don t cut it anymore. Agile and DevOps demand huge changes to the way application development and delivery (AD&D) and quality assurance (QA) teams test. Whether they are developing for the Web or for mobile first, they must transition to continuous testing. This report reveals continuous testing best practices for AD&D professionals in an age of high-speed digital delivery and explains how to turn QA from an impediment to rapid delivery into a new, converged model that tests quality at speed. Table Of Contents It s Not Just Speed; It s Not Just Quality; It s Quality At Speed Old QA Methods Don t Cut It Anymore Implement These Five Must-Do s To Overcome Poor QA At Speed 1. Organize Testing In A Lean Way 2. Use Shift-Left And Shift-Right Testing 3. Build A Practice For Testing Skills 4. Reduce Manual Testing In Favor Of Automation 5. Automate Test Data And Environments Provisioning WHAT IT MEANS If You ve Done The Must-Do s, You re Almost Ready For Mobile-First Supplemental Material Notes & Resources Forrester interviewed 19 companies: CA Technologies, Capgemini-Sogeti, Cigniti, Grid-Tools, HP, IBM, Magenic, Microsoft, Mindtree, Original Software, Parasoft, Syntel, Tasktop, Tata Consultancy Services (TCS), Tech Mahindra, Thoughtworks, Tricentis, Virtusa, and Wipro. Related Research Documents Overcoming Barriers To Modern Application Delivery August 4, 2014 Define A Software Delivery Strategy For Business Innovation July 25, 2014 The Forrester Wave : Service Virtualization And Testing Solutions, Q January 27, , Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. To purchase reprints of this document, please clientsupport@forrester.com. For additional information, go to

3 Five Must-Do s For Testing Quality At Speed 2 It s Not Just Speed; It s Not Just Quality; It s Quality At Speed Competing Formula One teams have figured out how to make pit stops an opportunity for differentiating and winning rather than a penalty. They ve learned that it s not just speed that determines the success or failure of the drivers during races but a number of factors: race strategy, real-time data analytics, teamwork, and, above all, quality. However, because every millisecond counts, reducing pit stops from 7 or 8 seconds to 3 or 4 seconds can change the outcome of a 2-hour race. Similarly, in the age of the customer, the speed of application delivery has a huge impact on the business and, as a result, is shrinking from months to weeks (see Figure 1). 1 This raises the question of whether quality assurance (QA) can also be reduced to frequent, fast pit stops that keep product quality high while meeting the business demand for speed. This can happen but only if AD&D and QA leaders challenge the traditional paradigm that more testing increases quality but decreases delivery speed (see Figure 2). According to IBM s business value research with 435 executives, 71% of the most successful advanced companies have made it a high priority to increase software development speed while maintaining quality; they regard doing this effectively as important for competitiveness compared with usability, flexibility, global integration, reliability, and business insights (see Figure 3). 2 Figure 1 AD&D Leaders Face Unprecedented Demand For Speed And Quality Delivery cycle time 1960s Mainframe 1980s Client/server 1990s The Web 2010 The demand for speed and quality are unprecedented. The age of the customer Time Continuous Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited.

4 Five Must-Do s For Testing Quality At Speed 3 Figure 2 Old-School Thinking Compromises Between Speed And Quality Testing improves quality but reduces delivery speed. Quality Speed Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited.

5 Five Must-Do s For Testing Quality At Speed 4 Figure 3 Quality At Speed Is A Must For Advanced Application Development And Delivery Effectiveness of capabilities important to competitiveness (Percentage who rated their organizations as effective or very effective ) Usability Creating a more usable and appealing customer experience Flexibility Delivering flexible systems to meet changing business needs 4% 4% 30% 22% 66% 63% Advanced Intermediate Foundational Speed Increasing software development speed while maintaining quality 2% 22% 71% Global integration Integrating systems globally to streamline business 5% 31% 70% Reliability Delivering reliable transaction processing with greater volumes 14% 40% 61% Business insights Providing business insights from social/ unstructured data 12% 4% 70% Base: 435 executives representing 18 industries and 58 countries Source: IBM Institute for Business Value Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited. Old QA Methods Don t Cut It Anymore Agile development has been around for more than 13 years, but it s taken AD&D and QA teams at least 10 years to realize that while Agile improves quality, thanks to the high degree of discipline required, it is testing that really needs to step up to help achieve overall improved quality. 3 Only 13% of 149 Agile adopters had implemented Agile testing when we surveyed them about their downstream Agile adoption in September Many organizations are still clinging to the old idea that testing means slowing down delivery consequently, they either use testing in the wrong way or don t test enough at all. What are the barriers to testing quality at speed? AD&D and QA teams have to: Break down organizational testing silos. Large enterprises in particular have tended to implement dedicated testing centers of excellence (COEs) with testers that focus day in and day out on defining and executing testing tasks. 5 This causes a variety of problems and antipatterns : 1) Testers get disconnected from development and delivery; 2) testing is set up as an afterthought to development (a waterfall approach); 3) it inhibits short, iterative development

6 Five Must-Do s For Testing Quality At Speed 5 and fast end user feedback; and 4) it makes quality the responsibility of a few testers, rather than the whole organization. Other organizations suffer from completely decentralized and distributed testing, which also leads to inconsistency in quality 6. Turn testing into a lean process. We see a variety of typical testing process challenges. Too much manual testing leads to increases in testing times, and our 2013 Agile survey showed that 71% of testing is often or always manual. 7 Testing strategies are rigid and predefined: The testers often decide upfront what type of functional and nonfunctional testing to execute and what tools and practices to use. Even worse, a centralized team that doesn t understand the business needs of your project often makes those testing choices for you. Finally, testing processes are mostly linear and defined using a waterfall approach: First you define the testing strategy; then you create the test plan with test cases; then you execute tests and do bug tracking; and then, much later on, you fix the bugs. In reality, requirements change often and you need to iterate all those steps in cycles but that hardly ever happens. Increase their level of automation testing. Automation testing levels are really low, although organizations have spent a fortune on tools and technology in an attempt to improve this. Our Agile survey showed that 38% of the participating organizations automated less than 18% of tests, while only 11% automated 80% or more. 8 Organizations have focused too much on graphical user interface (GUI) automation testing tools and not enough on the design and architecture practices needed to reach higher levels of automation. The efforts of services firms and systems integrators like Capgemini-Sogeti, Cigniti, Infosys, Mindtree, Syntel, TCS, Tech Mahindra, Virtusa, and Wipro also demonstrate this market need; they are all investing in accelerators to improve end-to-end automation testing. Manage growing architectural complexity. Application architectures are growing in complexity in response to current trends and customer needs around mobile, social, analytics, cloud, and the Internet of Things (IoT). Composite applications and systems of engagement built for mobile which provide, integrate, and interoperate with third-party application programming interfaces (APIs) are also changing the focus of QA. In these distributed architectures, integration testing becomes a major challenge. Provision an efficient test environment. Three main challenges exist: 1) the lack of standardization of test environments, 2) the low level of automation in the process of setting up and provisioning test environments for teams, and 3) the frequent misalignment between test environments and the production environment. These all reduce quality and increase the length of test cycles. DevOps faces major obstacles when trying to create and decommission test environments. 9 In conversation with us, AD&D leaders and QA managers claim that these obstacles are increasing the cost of QA (not including actually fixing the software) by 20% to 30%.

7 Five Must-Do s For Testing Quality At Speed 6 Implement These Five Must-Do s To Overcome Poor Qa At Speed QA and application development leaders should learn from Formula One and support the strong business desire for increased speed of delivery while ensuring quality (see Figure 4). Speed cannot come at the expense of quality. Based on conversations with dozens of AD&D pros who have improved or are improving their QA approach to support Agile, we ve determined the following five pit stop best practices that QA and AD&D teams should implement. Figure 4 The Five Must-Do s For Continuous Testing 1. Organize testing in a lean way Get testers and developers to work together. Reduce testing waste and wait times. Be more exploratory when you manually test. 2. Use shift-left and shift-right testing Get QAs and BAs to work together. Define user acceptance in your definition of done. Start functional and nonfunctional testing in sprint 0. Leverage data and analytics from production and end users. 3. Build a practice for testing skills Build a strong testing practice of excellence (not a TCOE). Improve your existing testers change management soft skills. Make Testing is fun and fast! your new refrain. 4. Reduce manual testing in favor of automation Shift automation testing more into the hands of developers. Automate tests for all of your architecture layers. Automate functional and nonfunctional tests. Automate the orchestration of test automation through CI. Automate from end to end provision, design, execution, feedback. 5. Automate test data and environments provisioning Standardize your test environments. Provision test environments and data continuously. Virtualize and simulate to tear down barriers Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited.

8 Five Must-Do s For Testing Quality At Speed 7 1. Organize Testing In A Lean Way Agile development needs Agile testing! Testing doesn t have to be process-heavy, be planned upfront, or involve bloated documentation and many handovers and control gates in order to be effective. Three actions will make your testing leaner: Bring testers and developers together. Form DevTest teams to encourage testers and developers to work together, and move testing responsibility and accountability to these teams. Turn testing into a continuous process. Reduce testing waste and wait times by allowing teams to build, change, and execute test plans and test cases iteratively. You can also reduce wait times by minimizing your dependence on external services.. Use manual testing only where needed. Some test scenarios might involve areas of high business or technical risk, which are too hard or too expensive to automate. For those areas, adopt exploratory manual testing that accommodates time-boxed approaches, as Agile usually requires Use Shift-Left And Shift-Right Testing Speeding up testing while keeping or increasing quality requires a totally new strategy: You must build in quality from the very beginning. Shift-left testing contributes to this; while it s not new to veteran testers, it s still rarely implemented, unfortunately. 11 To extend your testing in both directions, left and right, you need to: Align business and testing requirements. Get QA teams and business architects to work together from the very beginning to align their testing and business requirements; you can even define business and test requirements together through behavior-driven development (BDD) languages like Gherkin. 12 Align functional and nonfunctional testing from the start. Kick off the planning for functional and nonfunctional testing in an initiation step called sprint 0. Here you should define typical overall sprint and release testing goals. Define testing at each sprint. Define clear goals for all functional and nonfunctional testing in a sprint. The sprint definition of done is where you should clearly describe your testing goals. A common and safe approach is to associate definitions of acceptance criteria to the sprint backlog user stories, such as adopting a user acceptance test-driven development approach. Don t stop testing once you ve delivered your application. Finally, use shift-right testing as well by enabling developers to leverage data analytics from their app s production deployment. You should also enable feedback from end user testing by adopting innovative approaches, such as A/B or multivariate testing. 13

9 Five Must-Do s For Testing Quality At Speed 8 3. Build A Practice For Testing Skills QA or testing should no longer be unpopular; as software increasingly becomes your brand, better testing is essential. 14 If you have a centralized testing COE, federate it; if you don t, don t build one. However, in all cases, you will need a testing practice that can: Address specific testing needs via subject matter experts. Your testing practice needs to provide teams with specialized testing expertise in nonfunctional and test data management services. Multiply test automation skills. A core group of testers with great automation skills should help create a new role: test automation engineer. This core team should also define and provide teams with automation patterns and best practices. Foster best-practice testing processes. The testing practice should define, collect, and disseminate testing process best practices among your combined DevTest teams of business, development, and testing pros. Create and nurture a testing community. You must develop the skills of your existing testers so that they are more technically savvy; encourage them to become strong practitioners of new testing approaches like test-driven development, BDD, and automation engineering. Improve their change and soft management skills as well, so that they can lead DevTest teams rather than control them. Your new refrain? Testing is fun and fast! 4. Reduce Manual Testing In Favor Of Automation Manual testing will never go away at least, not until artificial intelligence or cognitive computing can replicate the smart, ad hoc testing that our brains can do. However, you should limit it to focused, exploratory cases. You can and should automate everything else. How? Automate testing at all layers. Automate tests at all layers of your architecture by going beyond GUI automation testing to testing at the API, service, and process levels. Shift automation testing to developers. Developers should make automation testing part of their daily job. Don t limit this to automating unit tests. Automate both functional and nonfunctional testing. Besides functional testing, extend automation to nonfunctional areas like performance and integration testing. Orchestrate your test automation. Use continuous integration servers to orchestrate test automation, and to make it an integral part of your automated continuous delivery. Think about test automation across the whole life cycle. Automate as much of the life cycle as possible from test definition/design to automation implementation/execution and to feedback collection.

10 Five Must-Do s For Testing Quality At Speed 9 5. Automate Test Data And Environments Provisioning To serve DevOps teams fast, you need to provision test data and have a structured test data management approach (TDM). In addition you will need to automate the test environment provisioning process in addition to the testing itself. Standardize your test environments. Reduce the number, set up a standard configuration, and implement version-control for all the testing assets that need automated provisioning. Set up a TDM governance processes and consider using technology to help. You will need to synthesize, subset, and provision the test data for the DevTest teams so that it is secure and business-aligned. Vendors like Grid-Tools, IBM (InfoSphere), Informatica, and Tricentis (Tosca Data Synthesizer) will help you work out your options. Align your test environments and sandbox with the production environment. Use thirdparty services and service virtualization and testing technology like CA Service Virtualization, HP Service Virtualizer, IBM Rational Test Virtualization Server, Parasoft Service Virtualization, and Tricentis Tosca Service Virtualization. 15 Consider cloud options like Skytap to spin test environments up and down quickly. Review your testing tool strategy to support these changes. You may see an impact on all of your testing tools more need for lightweight test management tools with an increased focus on analytics; automation tools that enable more API testing; integration with service virtualization tools; and performance testing tools that can help developers design and build in performance quality from the very beginning. 16 What It Means If You ve Done The Must-Do s, You re Almost Ready For Mobile-First In the past two to three years, AD&D organizations have responded to the need to deliver new and better products and application features faster by transforming testing so that it s more Agilefriendly. Many are developing systems of engagement for the Web and desktops first and then pushing those implementations to mobile, mostly through a mobile web approach. However, mobile is leading the charge, so you will need to shift to developing for mobile first. The good news is that most of the practices you ve learned for Agile and DevOps testing will also support your mobilefirst strategy in the next two to three years. In fact, while Agile testing has been a nice to have for your Agile strategy, it will be mandatory for your mobile-first strategy, with a few additions. You will need to: 1) define a multidevice testing strategy; 2) focus more heavily on nonfunctional testing, such as performance and security, and on dealing with the impact of the network or cloud on your apps, and 3) increase the level of shift-right testing thanks to greater feedback and data analytics from your app users once you ve launched to production.

11 Five Must-Do s For Testing Quality At Speed 10 Supplemental Material Survey Methodology Forrester s Q Global Agile Software Application Development Online Survey was fielded to 149 technology management professionals from organizations that are implementing or have implemented Agile. For quality assurance, we screened respondents to ensure they were at least Agile beginners, meaning that they have already started at least one project with Agile development. Forrester fielded the survey from July to August Respondent incentives included a summary of the survey results. Exact sample sizes are provided in this report on a question-by-question basis. This survey used a self-selected group of respondents with knowledge of their organization s Agile practices and is therefore not random. This data is not guaranteed to be representative of the population, and, unless otherwise noted, statistical data is intended to be used for descriptive and not inferential purposes. While nonrandom, the survey is still a valuable tool for understanding where users are today and where the industry is headed. In addition to sampling error, one should bear in mind that the practical difficulties in conducting surveys can introduce error or bias into the finding of opinion polls. Other possible sources of error in polls are probably more serious than theoretical calculations of sampling error. These other potential sources of error include question wording, question ordering, and nonresponse. As with all survey research, it is impossible to quantify the errors that may result from these factors without an experimental control group, so we strongly caution against using the words margin of error in reporting any survey data. Companies Interviewed For This Report CA Technologies Capgemini Cigniti Grid-Tools HP IBM Magenic Microsoft Mindtree Parasoft Syntel TaskTop Tata Consultancy Services (TCS) Tech Mahindra Thoughtworks Tricentis Virtusa Wipro Original Software

12 Five Must-Do s For Testing Quality At Speed 11 Endnotes 1 We entered the age of the customer in Since then, the desire to reduce delivery cycle times has become a reality. A consulting project we did for a large vendor shows that organizations have reduced their release delivery times from four to six months to six to eight weeks; some of the most disruptive organizations, such as Amazon, deploy features to production every 11.6 seconds. The growing trend within highly digitally disrupted organizations is to deliver new features to production continuously. For more on this concept, see the July 25, 2014, Define A Software Delivery Strategy For Business Innovation report. 2 The IBM study identifies three dimensions of software development performance necessary for high execution: 1) effective capabilities in software design, development, and delivery; 2) practice maturity in process, methods, and systems for managing the software development and delivery life cycle; and 3) results the ability to deliver valuable software outcomes for the business. The three dimensions determine the three states of foundational, intermediate, and advanced that organizations typically go through. Source: IBM Global Business Services, The software edge: How effective software development and delivery drives competitive advantage, IBM Institute for Business Value, March 2013 ( se sv_se products the_software_edge.pdf). 3 Agile disrupts everything that we know about testing, including what practices to use, what skills are needed, how to set up the testing organization, and how to leverage testing tools. Organizations adopting Agile need to act now: Testing leadership, skills, practices, and automation must move to the forefront of development. All of this represents a massive change not just for testing and testers but also for development and deployment organizations. For more information on how testing must change and how to introduce a more adaptive testing approach to preserve quality without compromising team performance, see the January 15, 2013, Consistent Performance In Agile Teams Must Include Testing report. 4 Our data shows that scaling Agile is hard but it can be done. This report helps application development professionals understand what organizations are doing to scale Agile and recommends some good practices for doing so. For more information, see the February 5, 2014, How Can You Scale Your Agile Adoption? report. 5 The World Quality Report also shows that the testing center of excellence is becoming a concept of the past, as it does not support Agile and DevOps well. Source: World Quality Report , Capgemini, October 7, 2014 ( 6 For an overview of the problems of traditional COEs for testing, see the July , Navigating The Agile Testing Tool Landscape report. 7 Forrester s Q Global Agile Software Application Development Online Survey provides an understanding of how practitioners, not planners, are using Agile. The survey primarily illustrates what organizations are doing to scale Agile at the enterprise level, including what benefits they are experiencing and how they measure success; how they are using Agile development together with other software development approaches and processes; which Agile practices are most common; what the status of scaling Agile is in organizations; and how mature Agile testing is and what that reveals about the overall maturity of Agile adoption. For more data, see the October 28, 2013, What Are Companies Doing To Scale Agile Development? report.

13 Five Must-Do s For Testing Quality At Speed 12 8 Source: Forrester s Q Global Agile Software Application Development Online Survey. 9 If the environment doesn t exist, you can create a new one based on your release package. This is helpful if you need to create environments in development or test. The benefit of this is that you are saving dollars on those systems when not in use. ARA tools can either do this creation and decommission as part of their base functionality or call on other environment management tools to do so. More information regarding this subject will be available in an upcoming report. 10 Here s a nice description of exploratory testing. Source: James Bach, Exploratory Testing Explained, Satisfice, April 16, 2003 ( 11 Shift-left testing is not new, as this article demonstrates. Source: Larry Smith, Shift-Left Testing, Dr. Dobbs, September 1, 2001 ( 12 Source: Gherkin ( Visit to download open source versions of Cucumber. 13 For more on A/B testing, read the following article. Source: Paras Chopra, The Ultimate Guide to A/B Testing, Smashing Magazine, June 24, 2010 ( 14 Your customers increasingly interact with your products and services directly through the software they touch: mobile apps, connected products, websites, and digital experiences. If the software fails or fails to delight the brand suffers. But software doesn t stop with the applications that customers touch. It pervades your entire product and service experience, from call center reps to analytically derived custom offers from marketing. More than ever, software either enriches or degrades your brand. It s time for companies to acknowledge the central role of software and elevate it from an IT-led application development group to a business-led product development function. For more information, see the January 23, 2014, Software Must Enrich Your Brand report. 15 In Forrester s 15-criteria evaluation of service virtualization and testing (SVT) vendors, we evaluated five solutions offered by CA Technologies, HP, IBM, Parasoft, and SmartBear Software. This report details our findings about how well each vendor fulfills our criteria and where they stand in relation to each other to help application development and testing professionals select the right SVT solution. For more information, see the January 27, 2014, The Forrester Wave : Service Virtualization And Testing Solutions, Q report. 16 For a broader understanding of the testing tools landscape, see the July 18, 2013, Navigating The Agile Testing Tool Landscape report.

14 About Forrester A global research and advisory firm, Forrester inspires leaders, informs better decisions, and helps the world s top companies turn the complexity of change into business advantage. Our researchbased insight and objective advice enable IT professionals to lead more successfully within IT and extend their impact beyond the traditional IT organization. Tailored to your individual role, our resources allow you to focus on important business issues margin, speed, growth first, technology second. for more information To find out how Forrester Research can help you be successful every day, please contact the office nearest you, or visit us at For a complete list of worldwide locations, visit Client support For information on hard-copy or electronic reprints, please contact Client Support at , , or clientsupport@forrester.com. We offer quantity discounts and special pricing for academic and nonprofit institutions. Forrester Focuses On Application Development & Delivery Professionals Responsible for leading the development and delivery of applications that support your company s business strategies, you also choose technology and architecture while managing people, skills, practices, and organization to maximize value. Forrester s subject-matter expertise and deep understanding of your role will help you create forward-thinking strategies; weigh opportunity against risk; justify decisions; and optimize your individual, team, and corporate performance. «Andrea Davies, client persona representing Application Development & Delivery Professionals Forrester Research (Nasdaq: FORR) is a global research and advisory firm serving professionals in 13 key roles across three distinct client segments. Our clients face progressively complex business and technology decisions every day. To help them understand, strategize, and act upon opportunities brought by change, Forrester provides proprietary research, consumer and business data, custom consulting, events and online communities, and peer-to-peer executive programs. We guide leaders in business technology, marketing and strategy, and the technology industry through independent fact-based insight, ensuring their business success today and tomorrow