Building QA Automation Using Agile. October 2, 2009

Transcription

1 Building QA Automation Using Agile October 2, 2009

2 Introduction Topic: Building QA Automation Using Agile Automated testing is a key requirement for effective Agile development teams. Automated regression test suites allow cost-effective regression testing with high coverage. But, these suites tend to be large software themselves, with quality and maintainability requirements higher than the application under test (AUT). Therefore, automation suites need to be built using a disciplined software development approach, such as Agile. However, the AUT itself is changing and this makes Agile approach an even more suitable development approach for building automated regression test suites In this webinar we will discuss how to effectively build automated test suites using Agile approach Presenter: Gireendra (Giri) Kasmalkar Giri is MD & CEO of SQS India. Giri founded VeriSoft, one of India s leading independent testing companies. VeriSoft was acquired in July 2008 by SQS Software Quality Systems AG, the global leader in independent testing. Giri did his BS in Mechanical Engineering from IIT Mumbai (India) and MS from Univ. of South Carolina. SQS Software Quality Systems AG Page 2

3 Contents Regression Automation Recap The Problem The Solution Methodology Standard Lifecycle Agile Lifecycle Sample Sprint In Conclusion... Questions? SQS Software Quality Systems AG Page 3

4 Regression Automation Recap SQS Software Quality Systems AG Page 4

5 Automation Suite For Regression Testing Faster Higher Coverage Better Process Adherence Expected Attributes Robust Data Driven Maintainable Ultimate Goal Should reduce total cost of regression testing and risks for the AUT SQS Software Quality Systems AG Page 5

6 Automation Suite Structure Application Under Test Object Map Utilities Test Script Test Results CSV XML EXCEL DB Test Data Store Test Script Base state Action Verification Report SQS Software Quality Systems AG Page 6

7 Automation Framework For more robust, data-driven and maintainable automation Can handle more complex functionality, alternate work-flows Tracking and reproducing functionality is easier Higher productivity Reused (pre-tested) utilities and wrappers Organization-wide usage Across applications Across tools Typical frameworks Data-driven Test Script Modularity Framework Test Library Architecture Keyword driven or Table driven Framework Hybrid Test Automation Framework SQS Software Quality Systems AG Page 7

8 The Problem SQS Software Quality Systems AG Page 8

9 The Problem The peculiar nature of automation suites Large and critical software Need formal methodologies Need frameworks Quality requirements higher than AUT, esp. Robustness Maintainability Dependence on Tools limits on automation AUT which keeps on changing SQS Software Quality Systems AG Page 9

10 The Solution SQS Software Quality Systems AG Page 10

11 The Solution Overview Premises AUT will not be static New features Bugs Regression testing is for the stable parts of AUT These may keep on changing Solution Agile paradigm of incremental delivery Broad plan for the entire automation, but detailed plan for each sprint Benefits Can be nimble about which test cases to add to automation The use of automated regression starts much earlier than traditional methodology Automation suite always in sync with AUT SQS Software Quality Systems AG Page 11

12 The Solution Track Record Customers Products Enterprise Apps Tools Commercial tools QTP, Silktest, Rational Open Source Selenium, AutoIT Projects 2 people x 3 months to 14 people x 18 months Iteration 2 weeks to 4 weeks Paper Presented and published at QAI International Testing Conference, Pune, SQS Software Quality Systems AG Page 12

13 Methodology SQS Software Quality Systems AG Page 13

14 Methodology Standard Automation Life Cycle Tool Evaluation Comparative evaluation of different test automation tools Object recognition, ease of learning, ease of use, capability of the scripting language Requirement Analysis To study the existing test cases to estimate the percentage of automation. To determine requirement coverage of existing test cases and improve it Design Scripts Data preparation for base states, input and master output Utilities SQS Software Quality Systems AG Page 14

15 Methodology Standard Automation Life Cycle (contd) Development Recording Programming for robustness (exception handling), data-drivenness and as per coding guidelines Development and documentation of utilities Coding for reports logs, messages and test reports Testing Unit testing Batch testing Deployment Creating configuration files Use Test Management tool SQS Software Quality Systems AG Page 15

16 Methodology Agile Overview 12 9 Implement Test Test Code Code Analysis Analysis SQS Software Quality Systems AG Page 16

17 Methodology Agile Automation Lifecycle Pilot / Planning phase Development phase Maintenance phase Objective: Select tool, establish the process, proof of concept, estimate percentage of automation, estimate efforts for the subsequent automation. Deliverables Deploy automation for the POC scenarios Detailed plan for development and maintenance phases Duration 3-4 weeks Objective: Automating the existing test scenarios to the estimated percentage Deliverable: Bi-weekly deployment of automated test scenarios, execution of updated automation suite and test report Objective: Maintain automation suite with enhancements with each release Deliverable: Updated automation suite, execution of the updated automation suite and test report. SQS Software Quality Systems AG Page 17

18 Methodology Activities in Pilot / Planning Phase 1. Application understanding 2. Review of test cases 3. Prioritization of modules and their test cases for automation based on stability 4. Tool selection 5. Feasibility study to determine percentage of automation possible 6. Design of automation suite directory structure, utilities 7. Pilot implementation of automation for a module 8. Creating a broad plan for subsequent development and maintenance phases. Assumption: Scenario where product development is underway but test automation has not started. SQS Software Quality Systems AG Page 18

19 Methodology Activities in Development Phase For each iteration (bi-weekly) 1. Selection of modules and test cases based on stability and criticality 2. Determination of backlog from previous iteration and customer feedback 3. Execution of automation suite defect reporting and determination of maintenance requirements 4. Effort estimation and detailed planning for the iteration 5. Design, development, testing of utilities / framework elements 6. Design, development, testing and deployment as per typical SDLC the latest release of the AUT is used for testing and deployment. Iterations repeated till backlog of manual test cases automated to expected coverage SQS Software Quality Systems AG Page 19

20 Methodology Activities in Maintenance Phase For each iteration (each release of AUT) 1. Selection of modules and test cases based on stability and criticality 2. Determination of backlog from previous iteration and customer feedback 3. Execution of automation suite defect reporting and determination of maintenance requirements 4. Effort estimation and detailed planning for the iteration 5. Design, development, testing of utilities / framework elements 6. Design, development, testing and deployment as per typical SDLC the latest release of the AUT is used for testing and deployment. Tips: Automation should lag behind manual testing by a few iterations Test cases chosen for automation are not the ones newly added in the iteration, but the ones that have become stable. A smaller team can be used as the time-lines are longer in this phase. SQS Software Quality Systems AG Page 20

21 Methodology Sample sprint time-table No Activity Deliverable Day By 1 Release of AUT to automation team 1 Release Team 2 Selection of test cases for automation Product Owner 3 Preparation of test scenario and datasheets Test case and datasheet documents Coverage of automation metric 1 Product Owner 4 Effort estimation for backlog from earlier cycle 1 Automation Team 5 Effort estimation for designed test cases 2 Automation Team 6 Run the existing automation suite to determine maintenance requirement and effort estimation 2 Automation Team 7 Defect Reporting 2 Automation Lead off-shore 8 Development work on backlog 1-2 Automation Team 9 Deciding on scope for the cycle Coverage of implementation metric 3 Automation Lead off-shore 10 Development of Automation scripts Updated automation suite 3-9 Automation Team 11 Deployment (adding to the existing suite), execution Performance metrics Rework requirements 10 Automation Lead off-shore Acceptance 10 Product owner SQS Software Quality Systems AG Page 21

22 In Conclusion... Using Agile ensures success of Automation Incremental delivery means Early usage Continuous sync with AUT Concrete metrics of progress of the project Can be used for any development methodology used by AUT Using Agile facilitates out-sourcing / off-shoring of Automation (Otherwise) Automation always takes a back-seat when release is near Clear division of labour Manual QA team working on release can focus on new features Automation team can focus on regression of much larger number of old features (as regression is only for the stable parts, anyway) Clear visibility of progress and success of Automation SQS Software Quality Systems AG Page 22

23 Questions? SQS Software Quality Systems AG Page 23

24 1. Software development partner for small/mid-sized technology companies Exclusive focus on small/mid-sized technology companies Deep experience in full SDLC design, dev, QA/testing, deployment Technology and industry domain agnostic 2. Dedicated team of high caliber software professionals Seamlessly extends your local team, offering full transparency NOT just staff augmentation, but provide full mgmt oversight 3. Actually reduces risk of development/delivery Experienced team - uses appropriate level of engineering discipline Practices Agile development responsive, yet disciplined 4. Reduces cost dual-shore team, 50% cost advantage 5. Offers long term flexibility allows (facilitates) taking offshore team captive SQS Software Quality Systems AG Page 24

25 Synerzip Clients Representative List SQS Software Quality Systems AG Page 25

26 SQS Software Quality Systems At a glance: SQS is the leader in independent software testing and quality management services.» The global leader in independent software testing and quality management services - majority of its business in Europe Financial Times, 21 August 2007 «More than 27 years of prosperous operations Operations in 14 countries globally Over 5,000 successfully completed projects The customer base includes 36 FTSE-100 companies, half of the DAX 30 companies and nearly a third of the STOXX-50 companies The SQS philosophy is to increase success and efficiency of IT projects by using efficient testing solutions SQS is listed on the AIM London SQS Software Quality Systems AG Page 26

27 About SQS SQS India Overview Leading independent testing company in India 11+ year track record of off-shore services 150+ testing consultants 300+ projects for more than 100 customers globally Captive off-shore centers, IT companies Game companies, product companies Acquired by SQS in July 2008 Primary destination for scalable off-shore Mature offshore delivery process (SQS Global Offshore Delivery Model) QA Partnership with Synerzip Last 5 years Automation and Load testing projects Commercial as well as freeware tools ISO 9001:2000 certified for the last 5 years ISO27001 in progress SQS Software Quality Systems AG Page 27

28 Thank You! Please contact us for free consultation Hemant Elhence SQS Software Quality Systems AG Page 28