Software Product Testing in Agile Environment Abstract The new economic reality puts spotlight on agile software product development. Agile provides the opportunities to reduce cost of development and achieve payback on the investment sooner than in the traditional approach by allowing decisions on the product future before making the full investment. Software Product Testing in an agile environment offers a completely different perspective and is packed with multiple challenges. Proteans was part of a product development effort, spanning over 3 years where in we gained considerable understanding of product testing in an agile environment. The lesson we learnt and applied has been brought into this White Paper. Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com
Introduction Proteans approach towards implementing an agile Testing method is very different from the traditional testing practices. We gained knowledge of testing products in an agile environment through a recent client engagement that gave us responsibility to develop, enhance, and maintain a suite of Add-on products for Microsoft Dynamics CRM versions 1.2, 3.0, and 4.0. The following sections of this paper identify Proteans methodology towards successful release and support of this suite of products. 2/8
The Challenges To Test a suite of 18 Add-on products and synchronize worldwide releases Support all previous released versions (via upgrades) of the products in multiple languages [English, French, German, and Dutch], which are deployed across different environments The product was used by end Users from wide array of business vertical s viz. Airlines, NBA teams, ISV s, Professional Services firms, Investment Banks etc - all aggregating to 4000+ clients with a installed user base of 200,000+ licenses The depth and breadth of complexity of the products and deployment options translated to 400+ Testing scenarios Overcoming Challenges Proteans, set-up a small group of Product Management Team (Client) to generate the Product Backlog together with prioritization of the features based on maximum business value delivered We scheduled releases, every 9 weeks - broken down into three iterations, each of 3 weeks duration. The first two iterations are the construction phase with third being the Product stabilization iteration We defined each feature or defect as User Story and assigned priority to it We organized the engagement team into two development teams and one QA Team. Each development team has a QA resource, called the Integrated QA and four developers. A team of 7 QA resources formed the Regression QA team The Integrated QA is involved in the iteration planning meetings where each User Story is estimated taking into account both the development effort as well as the testing effort. Testing is story focused and context based we did not test allied features but focused only on the feature/defect defined in the User Story At the end of the iteration, the Regression QA team picks up the build and does a regression of the product. This takes care of Integration Testing and Regression Testing of the products. The Regression QA team was assigned the task to create multiple environments, respond to Client Support issues, update Product Test Suites, and update the outstanding Defect Sheet which is base-lined Proteans used RALLY for Project Management to maintain all project artifacts viz. User Stories, Defects, Test Cases, Product Test Suite. Besides project management, Rally was useful in Agile Process Management and towards defining and measuring project metrics 3/8
Proteans Software Testing Approach At Proteans, testing of the software is operationalised and automated even before the project begins unlike in the traditional approach where you come to know of the glitches only close to the end of the project. Software Product Testing is all about efficient management of Product Releases. To do this we listed down a series of questions and tried to find answers: In a suite of 18 products, with 12 existing versions in the market, how do we ascertain what is an existing defect or a new defect? How do you handle 400 + test scenarios? How do you certify that a Product is good enough to be released with minimum testing? How do you manage Test Suites over a Products Life Cycle? In answer to the questions, we adopted the following steps: A. The quintessential QA artifacts 1. The Product Test Suite The Product Test Suite has Test Cases, and categorized as: - Setups ( Installer) - Vital functionality - Regression - L10n and I18n - User Interface Test Cases - MSCRM version Specific 2. The Product Baselined Defect Sheet All the existing defects in the product are categorized and logged into this sheet. This sheet acts as ready reference to ascertain the existence of a defect. This also prevents duplicate defect logging and acts as beacon pointing to existing defective functionality. 3. The Product Regression Testing Estimates A log sheet records the time taken to execute Regression Test Case per product on a particular Deployment Scenario and this forms basis for Test Execution Planning. 4/8
B. The Story of the Integrated QA; user acceptance testing of story cards The Integrated QA writes test cases for the defects/ stories / features (worked upon in the iteration) and executes them. Post execution the results are updated into RALLY and the Story Card is sent for acceptance to Product Management team. The Story Card is accepted based on test case results, screenshots provided, and by user acceptance testing on the client test server. Largely, user acceptance testing of every story card has been a keystone in achieving customer satisfaction. A story card may or may not pass the test, for every minor defect detected. Each defect is logged into RALLY and the Product Baseline Defect Sheet is updated accordingly. At the end of the iteration, the Integrated QA will update the Product Test Suite with the new test cases. This is the most important step in managing the Test Suites. C. The story of the Regression QA; the regression testing process and defined exit criteria The Regression QA tests all the products in multiple environments using the product test suite to verify defects/story cards / features worked in the previous iterations. They execute the test cases from the Product Test Suite in its entirety On their assigned priority - the vital test cases are executed first and then the other test cases Follow Exit criteria - Release Candidate has to pass the Vital test cases and test cases written for the priority defects/ stories designated for the Release. In addition, the Release Candidate has to pass the Setup test cases in all applicable scenarios Post execution of the test cases, the team first compare the defects found with those on the Product Baseline Defect Sheet. If the defect exists there, it is ignored, else it is logged into RALLY, and the Product Baseline Defect Sheet is updated with defect ID. At the end of Regression iteration, the updated Product Baseline Defect Sheet is shared with Product Management team. This approach helps in optimum utilization of available time and work force, and decreases stress on the team, leading to sustainable productivity in the end. D. What happens to a defect? The Defect Life Cycle White Paper A defect is identified in Regression Testing or from the Support Team A defect is always logged into RALLY A defect is always updated in the Product Baseline Defect sheet A defect has a priority and a severity attached to it. The Product Management team identifies the priority and severity of the defect. Based on the above two parameters a defect is scheduled to be resolved in a Construction Iteration Test cases are created; the defect is fixed and tested The Integrated QA marks it as fixed in RALLY and updates the defect status on the Product Baseline Defect Sheet The Regression QA verifies the defect in the regression iteration and marks it as To be closed The Product Management team / Support Team mark it as closed in RALLY 5/8
E. Test planning Plan for Regression Testing is mostly context based. Factors like number of products to be released and their complexity, number of setup scenarios, target markets involved, etc, are taken into consideration. For example, priority of Non English language test environments to be included in the test plan depends on the products being released in that region. The review and acceptance of Test plan by the product management team ensures clarity on the testing goals. F. Use of Virtualization Technology The testing involved high level of complexity. The products, including setups, required to be tested on a multiple environments. Hence, we created test environments that mimic customer environments. This includes major/minor upgrades, l10n/i18n environments [French, German, Dutch, Japanese, etc], Outlook clients [desktop, laptop], NLB, etc along with customizations as implemented by the customers. Virtualization is extremely cost effective and versatile technology. By creating Virtual PC s and Virtual Server, we were able to reduce capital costs incurred in development with the added advantage of creating mixed deployment scenarios with almost negligible incremental effort. G. Cross product training/responsibilities Each QA member is trained on all the products and there is absolute Product ownership. All QA members are responsible for testing, documentation and preparation/maintenance of test environments. This ensures that nonavailability of resources at times does not hinder the scheduled testing plan. H. Human factor: Team spirit The whole project team Developers and QA worked hand in-hand. Unlike in most projects, QA is not somebody s step-child". Developers and testers work in close co-ordination, which helped in minimizing conflict and keeping at bay, the all too common developer-tester divide. The members of the team trusted each other and look upon each other s roles and responsibilities as complementary rather than adversarial. Effective mentoring and knowledge transfer among team members are other key factors that contributed to our success. 6/8
Benefits of Proteans Software Product Testing Approach Efficient documentation We managed the defect lifecycle efficiently. To simplify understanding of features, we used story cards and shared the same with the Product Management team. The user acceptance of every story card resulted in enhancing our testing effort and helps avoid waste in terms of time and resources for the documentation process. Reduced capital costs We used virtualization technology. By creating Virtual PC s and Virtual Server we were able to reduce capital costs incurred in development with the added advantage of creating mixed deployment scenarios with almost negligible incremental effort. Sustainable productivity The classification and assigning priority to every identified defect played a vital role in achieving the testing goals. Further, this approach helped in optimum utilization of available time and work force and decrease stress on the team, leading to sustainable productivity in the end. Timely delivery We ensured our QA team members are trained on all products and inculcated a sense of product ownership in each of them. This ensured adhering to schedule inspite of any inadequacy in resources, if any. Thus, we were able to manage the testing activities efficiently and hence, give a green signal for the product to go live despite the tight deadlines for testing. 7/8
About Proteans Proteans is one of the leading outsourced Software Product Development Company and subsidiary of Norway based CAMO ASA (www.camogroup.com ). We specialize in software product research and development services and help ISVs, SaaS providers, and Software Development Organizations of global 1000 companies worldwide to bring software products faster to market while reducing R&D costs. For more information, visit our website www.proteans.com. If you are considering improving your software product testing effort in agile environment, call us today and we will be glad to offer you free assessment of your software testing and test automation needs. US Phone: +1-(410) 312 7625 Phone: +1-(410) 800-4679 UK Phone: +44 (1708) 440 319 Norway Phone: +47-223 963 00 Sweden Phone - +46-707296481 India Phone - +91 80 6618 6555 All registered trademarks acknowledged. 8/8