Table of contents. HP Software customer perspective: using HP TestDirector for Quality Center software to report and resolve software defects

Transcription

1 HP Software customer perspective: using HP TestDirector for Quality Center software to report and resolve software defects White paper Table of contents About the author Introduction Learning from experience Customized software quality defect-reporting and resolution process Defect reporting Defect-resolution meeting Defect fix Retest defects Defect-status definitions Defect-status modifications status modifications by user Configuring new processes into test automation tools

2 About the author During Punky McLemore's tenure as a quality assurance (QA) testing manager at HP, she worked with HP Software's legacy Mercury solutions prior to the Mercury acquisition. Before accepting her position at HP, McLemore was a QA testing manager at a large U.S. financial institution, where she was responsible for testing all new applications for this multi-billion-dollar company. The company s QA staff employed up to 80 testing specialists and 120 HP TestDirector for Quality Center software users. McLemore has 16 years of experience in QA testing and has used HP Software (legacy Mercury) solutions for seven years. Introduction In this white paper, I will share my unique viewpoint and my experience using HP TestDirector for Quality Center to support a variety of large-scale projects. I will discuss the setup and management of the processes, and how easily the processes can be changed and maintained using this HP software tool. I will also provide you with some of the best practices that I accumulated as the project lead for testing my company's new automated foreign exchange system a three-year project during which we extensively relied upon HP TestDirector for Quality Center. Learning from experience HP TestDirector for Quality Center is an extremely versatile product in all areas especially for tracking issues and discrepancies, and for managing defectresolution processes. I will share a set of best practices that you can use to design or streamline your own defect-tracking and resolution processes. One of my projects included creating and running a complex defect-tracking and resolution process for the company s newest application an automated foreign currency-exchange system. Previously, the company s financial managers had conducted currency exchanges by phone. With the new system, brokers would be able to trade currency electronically in real time. Because we had to electronically mimic a manual process, we studied the manual system to gather the majority of our requirements. We interviewed brokers, videotaped traders in action, and held many months of requirements-gathering meetings to examine and identify the business's needs. The system had to be very accurate to avoid misaligned deals, especially since there was the potential to lose hundreds of millions of dollars in a few keystrokes. My team did not have a current electronic system on which to base its expectations, so quality targets were vague. Communication among project group members was key to our success. The project was complex involving people from all over the world and testing was expected to take place 24 hours a day, seven days a week. The IT group also sought to automate the manual processes used in the development cycle to meet the project's challenges. My team needed a way to communicate issues, defects, resolutions and scheduling as well as to track activity. Likewise, the project team needed a system that would be highly reliable and easily customizable, and it needed to be able to support users all over the world. The system would have to be easy to use since the users were accustomed to manual processes and were not familiar with automated solutions. The brokers were also not used to completing transactions via computer. 2

3 After a few false starts with other automation tools, we discovered HP software. The QA department decided to try HP TestDirector for Quality Center as the foundation for all of its testing processes and found that it offered: High availability: HP TestDirector for Quality Center did not crash. Good performance: It was fast enough for longdistance users from Asia, Europe and Australia. Usability: It was easy to customize and use. After a favorable experience with HP WinRunner software, I decided to adopt HP TestDirector for Quality Center for full functional and regression testing. The company used HP TestDirector for Quality Center to model all of its processes for defect-resolution and scheduling releases. We defined everything in HP TestDirector for Quality Center along application lines all of the enterprise s applications were referred to as projects. The currency-exchange project was launched; it would ultimately take approximately three years to complete and it would deliver outstanding results. We recorded, tracked and resolved all discrepancies, deficiencies or other abnormal system behavior during the test phases of the software development lifecycle (SDLC) according to the company s newly defined problem-reporting procedures. In HP TestDirector for Quality Center, a discrepancy is defined as an unexpected event that occurs while testing or when the expected result does not equal the actual result. A discrepancy could be a defect, a request for an enhancement or new functionality, or any issue that needed to be corrected. We used the HP TestDirector for Quality Center Defects tab tool to report defects. The defect-resolution system tracked every issue and discrepancy in the project from system, hardware and software issues related to the application to process issues such as who was responsible for each step, how to obtain equipment, etc. I defined detailed steps for the company s new defect-reporting and resolution process (Figure 1). Customized software quality defect-reporting and resolution process It is important to first define all processes. Only with thorough planning can a defect-reporting and resolution process provide valuable information and streamline the processes for IT. In the following sections, I will describe how you can create reporting processes, define defect security levels, record defect status and track all modifications. 3

4 Figure 1. Software Quality Assessment (SQA) defect process flowchart Defect reporting START Discrepancy is identified Discrepancy is logged in HP TestDirector STATUS = NEW Use standard for defect documentation Defectresolution meeting Renew new open and reopen defects Need clarification? Adjust severity if needed STATUS = OPEN Assign defect Defect fix Retest defects Business representative Assign to? Development group Test group Close defect Behavior accepted? Environment group Assign defect to developer Receive build package and execute smoke test Smoke test passed? Log defect Change requirement Fix code and unit test Assign defects and execute defect retest Retest fail? STATUS= REOPEN Assign to development Fix in this release? Schedule for later release STATUS=FIXED Perform regression test defects? Log defect Assign to development Renew new open and reopen defects Deploy build with fixed defects Close defect Set STATUS = READY TO RETEST Set STATUS = READY TO RETEST Execute smoke test Environment fixed? Close defect STATUS = REOPEN 4

5 Defect reporting During each phase of the SDLC in HP TestDirector for Quality Center, we reported all discrepancies and issues relating to a project. When a defect was opened, my team would follow the process described below. 1. A release team member would log a defect report in HP TestDirector for Quality Center as soon as enough information was gathered to recreate or explain the situation. The default status was set to. I encouraged everyone on the team to log defects as soon as possible to allow optimum time for investigation and repair. 2. The person who logged the defect assigned an initial severity to the defect based on how it affected the application and/or the test process itself. Listed below are the requirements we used for defining the defect's severity. a. Show stopper: This level of defect prevented the team from proceeding in an area that would affect deployment schedules. workaround was available, and the team required an immediate solution. b. High: This level of severity was the same as show stopper, except that a workaround existed and the team could wait for the defect to be fixed until the next scheduled build. c. Medium: The team needed the defect to be fixed as soon as possible prior to deployment. d. Low: The team could wait for the defect to be fixed in a subsequent release (a later project). e. Enhancement: A defect that could be fixed any time. 3. The team used a defect writing standard to include all pertinent information. A team member then incorporated the specifics of the discrepancy, including print screens, associated test cases, selection criteria and any other critical information to help resolve the issue. 4. The team used the Assign To field to indicate who would be the likely choice to conduct an initial investigation, and the appropriate group leader was assigned to the defect. This fast-track assignment allowed defects to be reviewed and possibly resolved prior to the defect-resolution meeting described in the next section. If the team could not decide on the resolving group, the defect was assigned to the project manager. 5. If someone other than a release team member identified a discrepancy, that individual forwarded the specifics to the test lead so the data could be logged into HP TestDirector for Quality Center. Defect-resolution meeting We wasted time during the testing phase when defects were not acted upon in an efficient manner or went undetected. During defect-resolution meetings, we reviewed and resolved defects efficiently. So we decided to discuss defects in a periodic meeting with the test lead and representatives from the development, project management, product management (user acceptance) and business groups. The highest number of defects in the system approached 5,000, and we handled 25 to 50 defects per defect meeting. The test lead updated HP TestDirector for Quality Center during the meeting, which centered on a review of each defect report. Each report was prepared in HP Test- Director for Quality Center and contained all information needed to determine the severity of the defect and to whom it should be assigned. The defect reports were sorted by severity and status so that high-profile and new bugs would be addressed first. defects were first reviewed for the criteria listed below. Summary and description: If a defect lacked the information needed to adequately investigate the discrepancy, we assigned it back to the individual who found the defect to provide more information. The defect maintained status. The individual who found the defect then provided the necessary information by updating the defect in HP TestDirector for Quality Center. Next, the defect was assigned back to the project manager via the test lead so it could follow the process for defects. Severity: We reviewed a defect's severity for accuracy and required the team's consensus. Assign to: We assigned the defect to the appropriate team (environment, developer or business analyst) for investigation and resolution. te: Whenever an assignment was changed, HP TestDirector for Quality Center was set up to send out an to the assignee so the assignee immediately knew if a defect required attention. Status: We changed the defect's status from to. We set up HP TestDirector for Quality Center to send to the assignee each time a defect was assigned, and to send an to the test lead each time a defect changed status. The project manager and test lead also received an each time a new defect was entered. The test manager received an each time status was changed to User Error or Works as designed. The test manager then used this information for mentoring. The development manager received an for each status changed to Reopen. You can 5

6 easily customize HP TestDirector for Quality Center so that these changes can be made by a non-programmer; my team often made changes to support processimprovement initiatives. One tester used 50 percent of his time to administer HP TestDirector for Quality Center and make all of the experimental and real changes. This is one of the most powerful reasons to use this solution. Next, my team would review any defects, and this is where the Comments became useful. At each stage of defect-resolution from to, we entered comments to document the defect s journey. The Comments revealed the why, how and when a defect would be resolved. I would then make appropriate assignments to move the defect s resolution along. If the system s behavior was acceptable and the systems functioned as expected, we updated the defect with the acceptance information and assigned it to the test analyst who set the defect status to. My team then reviewed what was entered in a userdefined field called Cause. This enabled us to examine the root cause of the defect, as well as to pull reports at a later time for process-improvement purposes. We also had the option to review any disputed defects. Defect fix Because we held defect-resolution meetings, we assigned defects to the appropriate lead from one of the resolving groups: development, environment support or the business group. 1. We assigned the defect to the business group if there were any questions regarding a defect's requirements. At this point, we made any necessary requirement changes. 2. We assigned the defect to the development lead if we suspected there was a bug in the code, if any code changes were needed, or if it was identified as an enhancement for a later release of the application. This person then assigned the defect to a developer. When the developer fixed the code, the developer changed the defect status to Fixed. The fixed defects made their way into build packages and were handed off to the test group. HP TestDirector for Quality Center produced a report that reflected all defects fixed in a build. When we handed off a build, the test lead set the status of the Fixed defects to, and assigned the defect to a tester for retesting. te: A strength of HP TestDirector for Quality Center is its ability to define attributes for defects so as to support various processes. One of these is the Code Fix process. When configuring HP TestDirector for Quality Center, the code fix team took into account the attributes needed to track the defect as it went through the fixing process: setting developer priorities, keeping fix-time statistics, etc. The same procedure was applied by all the other groups. The team controlled the quality of the information for each defect by a system of attribute ownership. 3. We assigned the defect to the environment support group when configuration and set up errors were identified. These discrepancies usually surfaced when we ran a smoke test. (Smoke testing is non-exhaustive software testing that determines whether the most crucial functions of a program work without disturbing the finer details of the program. The term was derived from hardware testing, in which the device passed the test if it didn t catch fire the first time it was turned on.) The environment support group fixed the problem, set the status of the defect to and then assigned the defect back to the test lead for retest and subsequent closure. Figure 2. Defect R&D comments field Defect Status Fixed On Hold Reopen Goals of the ITIL process By default, the status of the new defect was when added by a test analyst. The test lead changed the status from to and updated the R&D comments field within an acceptable timeframe (dependent upon severity) and the defects were assigned to the appropriate teams. Developers updated the status from to Fixed when the defect was fixed and unit tested. The turnaround time for defect fixes was driven by the defect severity. Project/product/release managers updated status to when the defect had been fixed, the system tested and identified as being included in a build. Test analysts updated the status from to On Hold when the defect could not be tested in the build due to another defect that prevented the retest execution. Test analysts updated the status from to after re-testing on the new build of the application. Test analysts updated the status from to Reopen when the defect retest failed. Test analysts updated the status from to Reopen when the previously closed defect reappeared in a course of testing. 6

7 Retest defects We retested and immediately closed any defects we fixed without changing code. We included defect fixes requiring code changes in a build package and retested, and either closed or reopened them depending on the outcome of the test. All of this information was handled very efficiently in HP TestDirector for Quality Center: 1. Our test analyst reviewed any set up information and the Ready for Retest defects for retesting in the build. We also identified any Test Pending defects and reviewed them for retest consideration in the cycle. 2. Our test analyst then updated the status to Ready for UAT (User Acceptance Testing) when the defect detected in UAT/production was fixed and tested by integration test analysts. 3. Next, our test analyst retested the defect and updated its status appropriately. 4. If the defect was not resolved, the test analyst updated the status to Reopen, and the process resumed at Step Our test leads provided us with the status on all unresolved defects on a daily basis. 6. After the conclusion of each release, during the post-release phase, our test lead scanned the defect statuses for defects that were not yet resolved, to address them during the next release. Defect-status definitions Each defect had a status that we updated throughout the test cycle (Figure 2). Every time we updated the defect status, we added comments into the defect R&D comments field. Defect-status modifications status modifications by user The table below is a simplified version of the status changes allowed by HP TestDirector for Quality Center, which provides an easy learning curve and enforces a preferred workflow. Listed below are some of the status changes, also shown in Figure 3. Any changes to the system were made according to appropriate change control procedures. As necessary, the failed test or sections of the test were re-executed after being corrected. At any time during the process, our project managers, system analysts, developers and test staff could use HP TestDirector for Quality Center to retrieve the status of all defects stored in the HP TestDirector for Quality Center repository. The reporting capabilities of HP TestDirector for Quality Center provided a way to monitor the overall progress of defect-resolution. Figure 3. Status changes allowed by HP TestDirector for Quality Center User Test analyst Change from: Initial Entry Status to: Test Pending Reopen Ready to UAT Ready to UAT Project/Product/Release managers CR Created Fixed User acceptance analyst Initial Entry Ready to UAT Reopen Ready to UAT Developer Fixed Reopen Fixed 7

8 Configuring new processes into test automation tools All our processes evolved over time. It took us more than one year to transition from a chaotic process to a high-performance team that had: Contact information To find an HP Software sales office or reseller near you, visit Documentation Repeatable, measurable results A program for improvement HP TestDirector for Quality Center was pivotal in our success because it enabled us to model and remodel as we defined and redefined our processes. When we realized that HP TestDirector for Quality Center was successfully helping us to build the defectresolution process, we decided to tackle automation. To prepare for this next step, we decided to use the Test Plan in HP TestDirector for Quality Center as a repository for our manual tests, as well as the tool for managing and tracking all of our efforts. We used the concepts we learned in HP TestDirector for Quality Center to build robust test plans, as well as to build test sets that included manual tests and automated scripts that we originally built in HP WinRunner. To learn more, visit: Copyright 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. thing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. 4AA1-4226ENW, August 2007