Testing Process Models

Testing Process Models Process Model of a Test Factory EECS 814 Fall 2009 Jennifer Kaufman

Agenda 1. Introduction & Abstract 2. Organizational Models 3. Testing Process Models 4. Process Model of a Test Factory 5. Conclusion 12/12/2009 2

Introduction A proposal of a process model to create a Test Factory by: Ana Sanz Javier Garcia Javier Saldana Antonia Amescua Presented at the ICSE 09 Workshop on Software Quality 12/12/2009 3

Abstract Software testing is being outsourced to Test Factories Test Factories must implement an organizational model that includes a defined testing process model This research presents a testing process model to manage a Test Factory 12/12/2009 4

Software Quality Assurance According to Wikipedia, Software Quality Assurance (SQA) consists of a means of monitoring the software engineering processes and methods used to ensure quality. SQA strives to minimize the cost of guaranteeing quality throughout the development process 12/12/2009 5

SQA & Testing Validation ensures the right functions are performed Verification ensures the correct performance of these functions Testing is part of validation and verification Inspections Technical review Unit testing Integration testing All are SQA activities 12/12/2009 6

Outsourcing Testing Software reliability is demanded by customers Complex systems require efficient SQA activities including more testing Companies that do not have the testing resources are beginning to outsource testing These organizations have been termed Test Factories 12/12/2009 7

Test Factories Designed to correctly perform validation and verification activities Dedicated resources and infrastructure for complex testing Can help reduce software development costs Lack organizational model to manage the testing process 12/12/2009 8

Hypothesis If we have a well-defined organizational model which allows the creation of Test Factories, outsourcing testing-related activities could be managed efficiently. A well-defined organizational model includes: Organizational structure Process model Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 9

Organizational Models Software testing can be organized in various ways Developers as testers Independent test teams Integrated test teams QA/QC Outsourcing & Test Factories 12/12/2009 10

Testing Process Models Testing process improvement increases client satisfaction, reduces defect rates, and increases fulfillment of quality objectives There are numerous testing process reference models 12/12/2009 11

Framework Quagmire Ahern, D. M., Clouse, A., & Turner, R. (2001). CMMI Distilled. New York: Addison-Wesley. 12/12/2009 12

CMMI Capability Maturity Model Integrated Developed by the Software Engineering Institute (SEI) at Carnegie Mellon University Works organizations to continually improve software-intensive systems Helps organizations to develop or acquire the right software, defect free, within budget and on time, every time Carnegie Mellon University. (2009). CMMI Overview. Retrieved October 31, 2009, from Software Engineering Institute: http://www.sei.cmu.edu/cmmi/index.cfm. 12/12/2009 13

CMMI WHAT to do Ad Hoc & Immature Mature & Discipline Business Goals Not HOW and WHO should do it A benchmark of the maturity of an organization s processes with industry state of practice 14 Carnegie Mellon University. (2009). CMMI Overview. Retrieved October 31, 2009, from Software Engineering Institute: http://www.sei.cmu.edu/cmmi/index.cfm. 12/12/2009

CMMI It helps integrate once separate organizational functions Set process improvement goals and priorities Provide guidance for quality processes Provide a point of reference for appraising current processes CMMI does not fully support testing process improvement 15 Carnegie Mellon University. (2009). CMMI Overview. Retrieved October 31, 2009, from Software Engineering Institute: http://www.sei.cmu.edu/cmmi/index.cfm. 12/12/2009

TMM Testing Maturity Model Developed at the Illinois Institute of Technology by professors Bob Carlson and Ilene Burnstein and their graduate students It provides a guideline for test process improvement Compliments Capability Maturity Model IIT Center for Professional Development. (2009). Software Testing Maturity Model (TMM). Retrieved December 10, 2009, from IIT Center for Professional Development: http://www.iit.edu/cpd/itm/research/software_testing.shtml 12/12/2009 16

TMM Six levels of maturity 1. Initial Rush to code from requirements No clear separation between design and code phases No attempt is made to use well known design techniques or methods 2. Phased Clear separation between the requirements phase and design is phase IIT Center for Professional Development. (2009). Software Testing Maturity Model (TMM). Retrieved December 10, 2009, from IIT Center for Professional Development: http://www.iit.edu/cpd/itm/research/software_testing.shtml 12/12/2009 17

TMM 3. Systematic Defined design process with a methodical approach to design and the incorporation of design activities in the project plan Process can be taught and mastered, highly repeatable 4. Integrated Defined design process integrated with development and testing processes Portion of the code can be automatically generated and is consistent in structure Can be consistently maintained IIT Center for Professional Development. (2009). Software Testing Maturity Model (TMM). Retrieved December 10, 2009, from IIT Center for Professional Development: http://www.iit.edu/cpd/itm/research/software_testing.shtml 12/12/2009 18

TMM 5. Managed Design process managed by measures Large scale reuse and use of components leads to a high level of consistency Minimize the knowledge gap between management and software development staff 6. Optimal Systems are based on composition of designs Methodology is repeatable across people Process lends itself to self-analysis seeking corrections, efficiency, and improvements to the product 12/12/2009 19

TMMi Developed and promoted by the TMMi Foundation Compliments CMMI Based on TMM Guideline and reference framework for test process improvement TMMi Foundation. (2009). Test Maturity Model Integration. Retrieved December 10, 2009, from TMMi Foundation: http://www.tmmifoundation.org/downloads/tmmi/tmmi%20framework.pdf 12/12/2009 20

TMMi 5. Optimization Defect Prevention Test Process Optimization Quality Control 4. Management and Measurement Test Measurement Software Quality Evaluation Advanced Peer Reviews 3. Defined 2. Managed 1. Initial Test Organization Test Training Program Test Life Cycle and Integration Non-Functional Testing Peer Reviews Test Policy and Strategy Test Planning Test Monitoring and Control Test Design and Execution Test Environment TMMi Foundation. (2009). Test Maturity Model Integration. Retrieved December 10, 2009, from TMMi Foundation: http://www.tmmifoundation.org/downloads/tmmi/tmmi%20framework.pdf 12/12/2009 21

TPI Test Process Improvement model Develop by Sogeti, an IT service provider Supports the improvement of test processes Offers insight into the maturity of the test processes within your organization Helps define gradual and controllable improvement steps Sogeti. (2009). Test Process Improvement (TPI ): A step-by-step guide for improving your test process. Retrieved December 10, 2009, from Sogeti: http://www.ie.sogeti.com/documents/publications/sogeti_testing_test_process_improvement_tpi_step_by_step_guide_150609.pdf 12/12/2009 22

TPI 20 key areas for improvement Life cycle model Moment of involvement Estimating and planning Test specification techniques Static test techniques Metrics Test automation Test environment Office environment Commitment and motivation Test functions and training Scope of methodology Communication Reporting Defect management Testware management Test process management Evaluation Low-level testing Sogeti. (2009). Test Process Improvement (TPI ): A step-by-step guide for improving your test process. Retrieved December 10, 2009, from Sogeti: http://www.ie.sogeti.com/documents/publications/sogeti_testing_test_process_improvement_tpi_step_by_step_guide_150609.pdf 12/12/2009 23

TPI Each key area is given a maturity level ranging from A to D A Test Maturity Matrix established the relations among areas A set of checkpoints and improvements are made after the assessment Sogeti. (2009). Test Process Improvement (TPI ): A step-by-step guide for improving your test process. Retrieved December 10, 2009, from Sogeti: http://www.ie.sogeti.com/documents/publications/sogeti_testing_test_process_improvement_tpi_step_by_step_guide_150609.pdf 12/12/2009 24

TPI Sogeti. (2009). Test Process Improvement (TPI ): A step-by-step guide for improving your test process. Retrieved December 10, 2009, from Sogeti: http://www.ie.sogeti.com/documents/publications/sogeti_testing_test_process_improvement_tpi_step_by_step_guide_150609.pdf 12/12/2009 25

TestPAI Integrated with CMMI level 3 engineering processes Includes and defines all practices related to testing Same structure as CMMI Proposes 5 specific goals and related practices Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 26

What s Lacking The 4 models discussed are helpful to create a Test Factory, but not complete No model examined integrate organizational structure and a well-defined process model The authors proposed a process model of a Test Factory Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 27

Proposed Process Model 3 categories of testing processes Management Technical Support Each process was defined by identifying these elements: Purpose Description Activities Inputs/Outputs Roles Work Elements Work Products Technical Instructions Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 28

Proposed Process Model Management Testing Requirement Development Testing Planning Testing Design Support Technical Quality Assurance Control and Monitoring Configuration Management Measurement and Analysis Testing Execution Validation Tests Verification Tests Acceptance Tests Testing Report 12/12/2009 29

Management Testing requirement development Inputs: Analysis document, requirement document, specification of client needs Outputs: Contract between the organization and the Test Factory, testing requirements Description: If the feasibility study and project scope are agreed upon, a contract is signed and testing requirements are captured and managed Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 30

Management Testing Planning Inputs: Analysis document, testing requirement document Outputs: Test plan Description: A test plan is developed and maintained. In addition to the normal elements of a test plan, it includes sections of the project plan which are necessary manage the testing phase. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 31

Management Testing Design Inputs: Analysis document, design document, testing requirement document, test plan Outputs: Test plan, Test design specification, test procedure specification Description: Establish the information to execute testing effectively and efficiently including test plan tasks, schedule, risk, resources, and cost Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 32

Technical Testing Execution Inputs: Test work package including artifacts to be tested, project documentation and testing documentation Outputs: Result reporting Description: Test cases and scripts and data are required to prepare and execute tests. A result report is prepared after execution. This is comprised of verification tests, validation tests and acceptance tests. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 33

Technical Verification Tests Inputs: Artifacts to be tested Outputs: Result reporting Description: Inspections, walkthroughs and technical reviews are performed to review the artifacts. Checklists can be used as a simple and useful verification tool. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 34

Technical Validation Tests Inputs: Project documentation and testing documentation Outputs: Result reporting Description: Unit tests, integration tests, system tests, regression tests, stress tests, performance tests and usability tests are executed to check the system. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 35

Technical Acceptance Tests Inputs: Requirement document and testing document Outputs: Result reporting Description: Acceptance tests are executed by the customer or end user to ensure the system meets the client s needs. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 36

Technical Testing Report Inputs: Result reporting Outputs: Result summary reporting Description: Testing results are analyzed and evaluated and a brief report is drafted. This process controls and manages incidents. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 37

Support Quality Assurance Inputs: Test work package Outputs: Result audit reporting Description: Verification that the activities defined in the current process have been carried out and that the appropriate intermediate or final product is obtained. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 38

Support Control and Monitoring Inputs: Test plan, result monitoring reporting Outputs: Result monitoring reporting Description: To control and monitor the state of the projects in regards to schedule, costs, resources, milestones, commitments and risks. This is a continuous process. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 39

Support Configuration Management Inputs: Any element involved in the project Outputs: Information about the change Description: Changes in the configuration items must follow a formal process of change. The impact is assessed and the change is accepted or rejected. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 40

Support Measurement and Analysis Inputs: Project and process specifications Outputs: Measurement reporting Description: To control the processes implemented in the organization. Information collected is measured and analyzed based on set indicators. A measurement report is drafted. Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 41

Use of The Model The proposed model has been implemented at several companies in Spain The proposed process model was adapted for each implementation to include: Justification for using a Test Factory Metrics to measure the performance of the model Methodological guides to be considered when testing Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69). Vancouver, Canada: IEEE. 12/12/2009 42

Conclusion Test Factories is one way to structure testing in organizations Successfully managing a Test Factory requires a customized management model that includes A process model An organizational structure 12/12/2009 43

Conclusion The proposed model defines: The processes required to manage a testing factory Key elements to develop a contract for outsourcing testing activities The proposed model has been implemented in several companies but the results were not presented in this article 12/12/2009 44

Future of Software Testing 12/12/2009 45