F15. Towards a More Mature Test Process. Anne Mette-Hass. P r e s e n t a t i o n

Transcription

1 Towards a More Mature Test Process Anne Mette-Hass International Conference On Software Testing, Analysis & Review November Stockholm, Sweden P r e s e n t a t i o n F15 Friday 23rd November, 2001

2 Friday 23 November 2001 F15 Towards a More Mature Test Process Anne Mette Jonassen Hass Mrs. Anne Mette Jonassen Hass, M.Sc.C.E., has more than 20 years experience in IT. She has been involved in all aspects of software development: analysis, design, coding, test, quality assurance, and management. Mrs. Hass has worked in various types of business such as hospitals, the oil industry, telecommunication, hardware producers, and the space industry, in Denmark, Norway, England, and France. Since 1995 she has worked as a consultant in Software Process Improvement, assessments, configuration management, and test. Mrs. Hass has been involved in several process improvement projects in different companies. She also performes third party test and validation of software, especially safety critical software. Mrs. Hass is a certified BOOTSTRAP V.3.0 lead assessor, having performed more than 30 BOOTSTRAP assessments in Denmark, Poland, and Canada for companies of all sizes and in many different branches. Mrs. Hass has resently published a book about professional configuration management.

3 Towards a more Mature Test Process: Configuration Management EuroSTAR 2001 DELTA, Anne Mette Jonassen Hass, amj@delta.dk

4 What is it all about? Configuration management is reportable identification, controlled storage, and control of all changes of the items that a (software) system is produced on the basis of and consists of through the entire lifecycle of the system.

5 Configuration Management from a Bird s View THE WORLD AT LARGE Company m Company n Customer STANDARDS MODELS Supplier Product n Product 2 Product 1 Req. COMPANY PERSPECTIVE PRODUCT PERSPECTIVE Development Design Coding Test Project management Preparation Maintenance PEOPLE PERSPECTIVE Programmer Project manager Librarian Process responsible Quality assurance INSTITUTIONS Subcontractor management Configuration management PROJECT PERSPECTIVE 5 - Optimizing 4 - Managed 3 - Defined 2 - Repeatable 1 - Initial (Performed) PROJECTS TOOL SUPPORT PROCESS PERSPECTIVE

6 Contents Configuration Management in the Light of CMM A Definition of Configuration Management Configuration Management Data The Tester s Role in Configuration Management Implementation and Improvement of Configuration Management Configuration Management Tools Conclusion Useful Web-sites

7 Configuration Management in the Light of CMM - 1 CMM (Capability Maturity Model) version 1.1 staged Configuration Management at level 2 Software Configuration Management according to CMM: Goal 1 Software configuration management activities are planned. Goal 2 Selected software work products are identified, controlled, and available. Goal 3 Changes to identified software work products are controlled. Goal 4 Affected groups and individuals are informed of the status and content of software baselines. 10 activities are defined for fulfillment of goals in the model.

8 Configuration Management in the Light of CMM - 2 CMMI (Capability Maturity Model Integration) staged Configuration Management at level 2 continuously Configuration Management is a process area in its own right with goals and activies (almost) as for CMM V1.1 part of level 2 for all process areas Level 5: Optimizing Level 4: Managed Level 3: Defined Level 2: Repeatable Level 1: Initial (Performed)

9 SPICE Compatible Model - BOOTSTRAP v. 3.2 BOOTSTRAP 3.2 Organisation ORG.1 Business Engineering ORG.2 Human Resource Management ORG.3 Infrastructure Management Life cycle dependent Methodology Technology TEC.1 Technology Innovation TEC.2 Technology Support for Life Cycle Processes TEC.3 Technology Support for Life Cycle Independent Processes TEC.4 Tool Integration ENG.0 Development Definition ENG.1 System Requirements Analysis ENG.2 System Architecture Design ENG.3 Software Requirements Analysis ENG.4 Software Architecture Design ENG.5 Software Detailed Design Management ENG.6 Software Implementation MAN.0 Management and Testing ENG.7 Software Integration and Testing ENG.8 System Integration and Testing ENG.9 Maintenance ENG.10 Migration ENG.11 Retirement Life cycle independent MAN.1 Project Management MAN.2 Quality Management MAN.3 Risk Management MAN.4 Subcontractor Management MAN.5 Reuse Management Support SUP.1 Documentation SUP.2 Configuration Management SUP.3 Quality Assurance SUP.4 Verification SUP.5 Validation SUP.6 Joint Review SUP.7 Audit SUP.8 Problem Resolution Process related PRO.1 Process Definition PRO.2 Process Improvement PRO.3 Process Assessment PRO.4 Measurement Customer Supplier CUS.1 Acquisition CUS.2 Customer Need Management CUS.3 Supply CUS 4. Software Operation CUS.5 Customer Support

10 A Definition of Configuration Management Need from plan Production (Dynamic) Configuration Management Identification CMitem Item approval CMitem Release request CMitem Storage (Controlled) Audit Usage (Static) Change control Event registration Metadata Metadata Change request Metadata Metadata database Metadata Status reporting Change control data Reports

11 Detailed Definitions Identification Unambiguous definition of a configuration item and its relations to the outside world and to other configuration items. Storage Ensurance that a configuration item can be found at any time and delivered in the condition you expect to find it; and control of who has been given what. Change Control Full control of all change requests for a product and of all implemented changes in each configuration item relative to its predecessors. Status Reporting Making the information necessary for effective management of the development and maintenance of a product, available in a usefull way.

12 Configuration Management Data What may be placed under configuration management For configuration items: What do you need to know description of metadata What do you need to register list of forms What information is available examples of status reports

13 What may be placed under CM for test administrative documents letters, contracts, process description, sales material, templates, standards etc. hardware cables, mainframe, PC, workstation, network, storage, periferals etc. code header files, include files, source code, system libraries, object files etc. environments compilers, linkers, operating systems, tools, word processors etc. project documentation user manuals, build scripts, data, event registrations, installation procedures, plans etc. technical documentation requirements (all levels), design (all levels), technical notes test material drivers, stubs, test data(base), test reports, test specifications and test procedures The scope must be tailored to the individual project and company.

14 Releases Individuel configuration items may be assembled in releases, which may and should be configuration items in their own right Releases are hierarchies of configuration items. Test specification, V2.1 System Test, V6.4.3 Testcase-203, V2.1 Testcase-204, V2.0 Software, V E.25 Test Plan, V5.3 Test Specifikation 7, V1.1 PC, V1.8 Printer, V4.6 Testcase-205, V1.7

15 What do you need to know - Description of Metadata Product belongs to People produced by under responsibility of approval by may be distributed to has been distributed to Configuration item name version status date storage place storage medium tracing to produced using ancestor is assembled of Other configuration items

16 What do you need to register - List of forms Item approval configuration item concerned, dated signatures by producer, responsible, and approver, condition(s) for approval, and related metadata Release request configuration item concerned, dated signature(s) by orderer or orderes, possibly dated signature by responsible, reason for release request, and release medium and/or destination Event registration configuration item concerned, for the event: identification, type, and short title phase information for the event: phase, time, name of responsible, name(s) of other involved, description, and classification Change request Production Storage reporting the change: identification, identification of the underlying event, configuration item concerned and priority phase information for the change: phase, time, name of responsible, description Ident. A Metadata database Usage C C

17 What information is available - examples of status reports Release note Date, configuration item concerned, description, release composition etc. Item list: status Date, configuration item concerned, released to, open event registrations, open change requests Item list: history Date, configuration item concerned, for each related version: id, status, dato, responsible, released to, event registrations, change requests Item list: composition Date, configuration item concerned, for each included c.i.: name, version, status, date, responsible - this may be recursive Trace report Date, releases concerned, detailed To <-> From traces, missing traces either way Configuration management is a valuable supplier of project metrics.

18 Roles in Configuration Management A (software) project is likea play, where all roles must be filled in order forthe play tobe performed. IMPORTANT: Roles are filled by people! Configuration management is an everyday tool for testers. Personal requirements for testers to benefit from and contribute to configuration management: some attention to detail some disciplin interest in contributing to adding common value some knowledge of existing configuration management processes

19 Role in relation to configuration management - Tester 1 Responsibility - general: to test a system according to the applicable test plan to be able to re-perform already Production Ident. reporting performed tests in connection with later re- or regression tests in the entire lifetime of the product A Storage Metadata database Usage C C Responsibility - configuration management: to use the appropriate configuration management system in connection with the performance of all test activities, such as planning, performance, and reporting

20 Role in relation to configuration management - Tester 2 Contribution identifying relevant configuration items, e.g. test plans, test descriptions, test scripts and test data, and releases in terms of an entire test including test environment etc. placing these configuration items in storage after appropriate approval produce event registration for relevant configuration items in connection with performance of test, e.g. for source code or (sub-)systems

21 Role in relation to configuration management - Tester 3 Advantages extract related configuration items as the basis for the testing work, e.g. individual configuration items or, more importantly, releases in terms of integrated (sub-) systems getting information about the status and history of these configuration items getting information about relevant event registrations and their progress

22 Configuration Management in Practise - 1 Development phases (live cycle dependent) specifications, design, coding, integration, test, production, and maintenance; including emergency changes Support functions (life cycle independent) e.g. project management, configuration management, quality assurance, subcontractor management Special conditions e.g. frequent builds, iterative development, multi-platform, multi-site, multivariants, parallel development, composite systems, safety critical products, size of product, Web Cross-organizational products company infra-structure, internal and external reuse-components development, quality system, documents The art is to tailor the configuration management activities to support each phase, function, special condition, product, and business.

23 Configuration Management in Practise - Test 1 Possible test configuration management items: test plans test specifications, comprised of: test cases test procedures test environment, including: drivers stubs test data(base) tools machines and other hardware test reports From a configuration management point of view the differences between test levels and test types are not significant, and configuration management for test may therefore be described in one go.

24 Configuration Management in Practise - Test 2 Releases documents with e.g. test cases test object and the associated test configuration items, e.g. software module + test specification + test environment Connections to other process areas test performed on products from other phases test performed on the basis of products from other phases test planning may relate to project management

25 Configuration Management in Practise - Test 3 Identification mostly as for documents and code tracing is very important! Storage as and with associated configuration items possibly entire isolated test environments, including data Change Control should be in place for test objects (e.g. modules, system) don t forget test configuration items (e.g. test cases and data) Status Reporting must be comprehensive and fast a means for test management and follow-up

26 Configuration Management in Practise - Test 4 Tracing: The ability to determine why a configuration item exists, e.i. what it is based on. For test typically: Example: Software requirements specification V.2.0 : : Req.-1 V.2. 0 Req.-2 V.1.8 Req.-n V.1.8 Tracing System test V.1.A : : Testcase-5 V.1. A acceptance test < > user requirements system test < > software requirements integration test < > architectual design module test < > detailed design : : Testcase-6 V.1. A Testcase-11 V.1. A

27 Configuration Management in Practise - Test 5 Benefits to testing from tracing: early initiation of test planning, test case definition, and environment preparation extra review of requirements or design in view of testability notification of testers when requirements or design change requirement/design coverage reports ensuring everything covered by test cases (or other activities) test case status report showing traceability of all test cases (no unnecessary test cases)

28 Implementation and Improvement of CM - 1 Scoping of the configuration management task - cost/benefit analysis Degree of formalism Min. Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 Support function a Support function b Support function c Time Project phases Support functions Configuration items The total cost of c.m. is a function of the shaded volume.

29 Implementation and Improvement of CM - 2 From level 1 and onwards: Planning Configuration management Process description Metrics - statistical control On and on and on Document and Analyze Lessons. Revise Organizational Approach. Define Processes and Measures. Plan and Execute Pilots. Plan, Execute and Track Installation. Establish Improvement Infrastructure Set Context and Establish Sponsorship Stimulus for improvement Establish Process Action Teams and Plan Act ions Set Strategy and Priorities Develop Recommendations and Document Phase results Apprais e and Characterize Current Practice Initialise - get going Diagnose the current state START Establish a plan Act according to the plan Learn from the experience SEI

30 Configuration Management Tools Types of tools for configuration management Version Control - most primitive: storage, simple forms of control PVCS from Intersolv Visual SourceSafe from Microsoft Development Support - version control + create, merge, change, and release products in a distributed environment ClearCase from Rational Total Process Oriented - automate total process models, i.e. support workflow, roles, and responsibility in an integrated configuration management system Harvest fra Computer Associates Total Process Orientated Development Support Version Control

31 Conclusion Configuration management is: very complex very easy difficult to live with impossible to live without hard work easing work a pain (if you hate it) fun (if you don t)

32 Web-sites Yellow pages BOOTSTRAP Institute CMII brugergruppe CMM version CMMI European Software Institute Institute ofcm Ovum Software Engineering Institute Standarder Tool ClearCase Tool Harvest ca.com/products/ccc_harvest.htm Tool PVCS Tool StarTeam Tool Visual SourceSafe