The Quality Assurance Centre of Excellence

Transcription

1 The Quality Assurance Centre of Excellence A X I S T E C H N I C A L G R O U P A N A H E I M H E A D Q U A R T E R S, 300 S. H A R B O R, B L V D. S U I T E 904, A N A H E I M, CA PHONE :( 714) F A X: (714) W E B: W W W. A X I S T E C H N I C A L. C O M This document contains proprietary information. No part of this document may be photocopied, reproduced, or translated to another language without prior written consent of Axis Technical Group LLC. All rights are reserved. Copyright 2011 Axis Technical Group LLC

2 Table of Contents Introduction... 3 Engagement Models... 3 Advantages... 3 Areas of Expertise... 4 Quality Assurance Approach... 4 Quality Objectives... 4 Quality Assurance Activities... 5 Quality Assurance Metrics... 7 Measuring Project/Product Size... 7 Measuring Progress/Coverage of Testing... 8 Measuring Product Quality... 8 Measuring Test Efficiency... 8 Project Management Methodology Strategize Organize Transition Execute Manage About Axis Technical Group: Axis Technical QA COE Page 2 of 14

3 Introduction The Axis Technical Group Quality Assurance Centre of Excellence (QACOE) is configured to cater to the Quality Assurance and Quality Control needs of today s high performance IT enterprises. This document presents an overview of the activities and approaches that are adopted by the team at the QACOE to successfully assist the client s CIO office in maintaining an effective IT organization. Engagement Models At the Axis QA COE following engagement models are usually provided to the clients: 1. Captive QA units In this engagement model a captive QA team is made available to the client. The engagement duration is usually one year or more. This model provides the clients to have an extended off shore team to work on multiple project at a time. 2. Project specific QA service In this engagement model, the QA team is mobilized for a particular project or program. This model provides the flexibility to add the resources as needed to complete projects on schedule. In both the engagement models the typical team structure is as below: 1. On Site Coordinator: A senior technical QA resource with ample experience and expertise in off shore engagement model will be made available at the client s office. This resource works as the Single Point of Contact for client s team. All task requests, change requests, project artifacts, reports and other information are routed through this person. 2. Off Site QA team lead: A senior hands on QA engineer with experience in the client s domain and with expertise in various aspects of Quality Assurance is made available in the Axis office in Pune, India. This person leads all the project activities in Pune and will coordinate on a day to day basis with the On Site Coordinator for the inputs, provides the work artifacts 3. A team of QA Engineers: The highly qualified team with expertise in hands on Quality Assurance is mobilized at the Axis Pune office. Advantages 1. Greater control over system performance and issues 2. Greater control over processes 3. Improved and consistent system performance hour turnaround cycle 5. Reduced costs of QA Axis Technical QA COE Page 3 of 14

4 Areas of Expertise 1. Programming Languages: C, C++. C#.net. Javascript, VB.net, Windows Shell Script 2. Bug tracking tools: Team Foundation Server, Product Studio, Bugzilla, Sharepoint 3. DBMS: SQL-2005, SQL QA/QC Skills: Manual Testing, Functional Testing, Regression Testing, Integration Testing, System Testing, Test Strategy Development, Estimation, Test planning 5. Test Management Tool: Test Director, TFS 6. Automation tools: Quick Test Professional, Rational Robot,Load Runner, Xenu, Wink, Auto IT, Watin, Selenium, Visual Studio Test Automation, SOAP UI, nunit 7. Source Control Tools: SVN, Tortoise SVN Client, TFS, MS VSS 8. Other Skills: HTML, XML, MS Office, RoboHelp Quality Assurance Approach The approach to Quality Assurance provided by the Axis QA COE is based on the Quality Objectives, the Quality Assurance Activities and the Quality Assurance Metrics. The following sections describe these areas in detail. Quality Objectives From the onset of each project, the team at Axis QA COE works with the client to identify the Quality Objectives which need to be satisfied. Through a cooperative effort, the team establishes the key metrics to be tracked and reported as per the scope of the project. The following are the common objectives established for majority of the projects undertaken by Axis QA COE. Axis Technical QA COE Page 4 of 14

5 1. Ensure the end product is fit for purpose a. Satisfies the Functional Requirements b. Satisfies the Non-functional Requirements such as: i. Usability ii. Performance iii. Scalability 2. Ensure the end product meets the quality expectations Bugs are detected early and fixed before the end product goes in production 3. Ensure the end product is delivered as per the agreed upon schedule It is ensured that the other objectives set for the project, if any, are met as per client s expectations. Quality Assurance Activities The following Quality Assurance activities are carried out to meet agreed upon Quality Objectives. Some of the common activities are outlined below. 1. Review Requirements 2. Provide Requirement Analysis 3. Develop and Execute Test Plan 4. Develop and Execute Test Cases 5. Requirements Traceability and Gap Analysis 6. Automate Test Cases 7. Report Defects 8. Ensure defects are fixed and provide verification 9. Monitor Change Requests 10. Manage Risks to QA Objectives 11. Document Lessons Learnt 1. Review Requirements Requirements are reviewed by the Axis QA team as early as possible to gain better understanding of the application, as well as to plan the various types of testing conducted in order to satisfy the QA objectives. Axis Technical QA COE Page 5 of 14

6 Requirements include both functional (related to the behavior of the application) and non-functional (related to usability, scalability, performance etc) requirements for the end product. 2. Provide Requirement Analysis Whether or not this document is part of the deliverable, the QA team proactively provides an analysis document i.e. the understanding of the QA team of the requirements. This helps various groups (Engineering, BA and Product Management etc.) within the client organization to have confidence in QA group and overall 3. Develop and Execute Test Plan The Test Plan captures the following information, a. Scope of Testing b. Testing Approach and Strategy i. Types of Testing ii. Test Cycles (or Iterations) c. Testing Process d. Assumptions e. Risks 4. Develop and Execute Test Cases Test Cases are developed to effectively cover the requirements and test the application behavior under various scenarios from the end user s perspective. 5. Requirements Traceability and Gap Analysis After developing the test cases and before executing the Requirements traceability is performed. By tracing test cases back to the requirements, the QA team is able to account for every test case written and ensure that all requirements are covered. The by-product of this activity is a Gap Analysis report, where potential gaps in requirements are identified and notified to the appropriate group. 6. Automate Test Cases Depending on feasibility, time and budget allocation, the Axis QA team strives to automate all test cases which require iterative processes. Wherever feasible, the team tries to automate the core functionality and processes that do not change over a longer period of time. 7. Report Defects Defects are identified and reported as soon as possible to provide ample time for the development team to correct defects during the SDLC. Defects are reported in a clear and concise manner to help the development team reproduce and correct the issues in quick time. Axis Technical QA COE Page 6 of 14

7 8. Ensure defects are fixed and provide verification Axis QA team ensures that the reported defects do get fixed and then verifies these before the end product goes in production. 9. Monitor Change Requests Axis QA team proactively monitors all change requests. They analyze the impact of scope changes on the schedule and potential impact on quality and provide feedback to the Project Manager. 10. Manage Risks to QA Objectives Potential risks are progressively assessed, reported and mitigated in order to reduce negative impact as the project progresses. The Axis QA team plans and prioritizes testing in a manner which helps indentify and report the most severe bugs in early stages of the testing cycles. 11. Document Lessons Learned At the end of each project or project phase the QA team along with the Development team analyzes the reasons behind success or failure of every project or project phase. Achievements are rewarded and success stories are carved. Failures if any become stepping stones for process improvement and performance enhancements. Quality Assurance Metrics The QA team at Axis works with the client to create the Quality Assurance metrics that are essential for project. The Quality Assurance metrics are structured according to the purpose and nature of the project. Some of the metrics that are used as a part of the reporting are described below. Measuring Project/Product Size 1. Number of Requirements identified/reviewed High level Business Requirements are identified during project initiation and detailed requirements are created during the planning phase. 2. Number of Test Cases developed/reviewed High level Test Scenarios are indentified during the planning phase and the detailed Test Cases are prepared during the execution phase along the development of the code. 3. Effort in Person-weeks Based on the project timeline and resources in the schedule the effort in person-weeks is calculated. Axis Technical QA COE Page 7 of 14

8 Measuring Progress/Coverage of Testing 1. Number of Test Cases Executed vs. Developed (Percentage) This measurement indicates the test coverage achieved based on the number of Test Cases executed during a test cycle. 2. Number of Requirements Tested vs. Available (Percentage) This measurement indicates the test coverage achieved based on the number of Requirements covered by the execution of Test Cases during a test cycle. 3. Test Case Automation Percentage Automation of test cases is particularly useful when regression testing is involved, resulting in iterative, system wide retests for all new changes being made. 4. Rate of defect reporting vs. fixing This measurement is the number of defects reported in a fixed duration (weekly, biweekly) vs. the number of defects fixed during the same duration. 5. Average Turn-around time for fixing Defects This measurement is the time taken to fix defects once they are reported. The defects reported in Test Cycle 1 are fixed and verified in Test Cycle 2 and so on. Measuring Product Quality 1. Pass Percentage of Test Cases Pass Percentage is calculated using the following formula, Number of Test Cases Passed/Number of Test Cases Executed * 100 Higher percentage of passing would mean better quality. 2. Defect Quality Number, Type and Severity of Defects are reported by Testing Team in each phase/test cycle/environment. Status is shared with the customer after each test cycle. The more severe defects are found in test environment itself and fixed in the earlier test cycles. Thresholds are set for each test cycle to indicate whether these are within acceptable limits. 3. Defects Re-opened/Fixed Ratio This metric measures the count of defects fixed in previous cycle that are determined to be broken in a subsequent test cycle. These are classified as re-opened. Measuring Test Efficiency 1. Defects Acceptance Rate (Accepted/Total Reported percentage) Axis Technical QA COE Page 8 of 14

9 Most of the defects reported by QA team get accepted by the Customer (or development team). At the same time, there are few defects which get rejected due to various reasons like not-reproducible or inconsistent errors, typical behavior of the test environment etc. 2. Customer Reported Defects Number, Type and Severity of Defects reported by Customer that have escaped all software quality processes. These are commonly a result of a discrepancy between the production and the test environments. After all System and Functional Testing cycles are completed the application (product of the project) is deployed on the production environment for UAT (User Acceptance Testing) with real-time data. The product is tested by actual end users to gauge the production readiness. 3. Test Efficiency The counts as well as severity of defects reported by customer play an important role in determining the efficiency of the QA team. Axis Technical QA COE Page 9 of 14

10 Project Management Methodology Axis Technical s Project Management Methodology (PMM) will be inherent in the approach to staffing and managing the project. This will be used for managing both our deliverables for the project as well as project dependencies on any Third Party project deliverables. Axis Technical has a 5 point program management model that has helped the organization deliver projects successfully. Strategize A strategy is the most important step before the actual execution. The difference between a successful project and a failed one is the presence or absence of the right strategy. At Axilon a team of strategists brainstorm for the optimum solution based on client objectives, industry issues and value targets. Axis Technical QA COE Page 10 of 14

11 Organize Organization starts with fine-tuning the outsourcing process so that it benefits both the parties concerned. A Project Roadmap that outlines the three phases of the project lifecycle: Build a plan, Track and manage a project, and Close a project is prepared. This roadmap helps keep the project on track and monitor the progress. Finalize Outsourcing Processes Build Project Roadmap Finalize Pricing Sign Off SLA Identify Reporting Procedure Finalize governance Committee Formulate Business Continuity Plan Client Expectation Matrix Axis Technical QA COE Page 11 of 14

12 Transition Transition phase is a learning phase for both the parties involved. During this phase, agreed upon processes are implemented both at the customers end and the offshore location. Once processes are deployed they are verified and validated using audits and tools to establish their applicability to the project. Execute The executing phase of a project is where the actual development begins. Execution tests the business plan and the processes outlined therein. Monitoring the project is a continuous activity and it is to provide an understanding of the project s progress so that appropriate corrective actions can be taken when the project s performance deviates significantly from the plan. Axis Technical QA COE Page 12 of 14

13 Manage Analyzing team performance and measuring client s expectation is the important step in improving the relationship with the customer. SLA statistics are vital in performance measurement. Axis Technical QA COE Page 13 of 14

14 About Axis Technical Group: Axis Technical Group, LLC is a premiere Information Technology Services firm headquartered in Anaheim, California with offices in Pune India. Axis Technical Group has been built upon its core values of Quality, Consistency, Honesty, Loyalty, and Excellence through hard work. The company s focus on unparalleled customer service allows Axis Technical to provide every client with the cost-effective resources and solutions necessary to help them achieve their business goals. California Contact Information: 300 S Harbor Blvd, Suite 910 Anaheim, CA Phone: (714) Fax: (714) Colorado Contact Information: 4610 S. Ulster Street, Suite 150 Denver, CO Phone: (303) Fax: (303) For more information about the Axis QA COE and other value added services please visit or contact info@axistechnical.com. Axis Technical QA COE Page 14 of 14