Do 4.3 January 21-25, 2008, Munich, Germany ICM - International Congress Centre Munich Deriving Value from a Defect Management System Wolfgang B. Strigel
Defect Management Deriving Value from a Defect Management System Wolfgang Strigel Vancouver - Munich Slide 1 The Role of Testing COMMUNICATION about the quality of the product about defects and all associated details and attributes feedback about the product and the process Slide 2
Information Flow Business Decisions Development/SQA Product Information Process Information Testing Slide 3 Result of Testing Better product Happier customer Is that all? Slide 4
Internal Results Higher productivity in development Visibility of development process Improvement of development process Happier staff Need a Feedback Mechanism Need a Defect Management Process Slide 5 Defect Management Definition All activities related to the capture, storage, communication and analysis of defect information Slide 6
Defect Life Cycle Defect life cycle is one of the foundation processes Describes how a quality report or defect is handled Defines how severity codes are defined and handled Defines interaction between team members related to defect management Slide 7 Defect Life Cycle Defect not fixed Start Open Resolved Closed Defect reported Deferred Defect fixed Defect confirmed fixed End Defect to be addressed later Slide 8
Detailed Defect Life Cycle QA Labs Inc. Slide 9 Defect Review Board (DRB) A body to manage defects Purpose: Review, prioritize and update defects ( Defect Triage ) Attendees: DRB Chairperson, Project Management, Development & Test Manager(s), Customer Support Frequency: daily, weekly, etc. Slide 10
Roles - Responsibilities Role Responsibility DRB Chairperson Project Manager(s) Development Manager(s) Test Manager(s) Customer Support Chair defect review meetings Table list of defects to be reviewed Track list of actions/decisions arising from meeting Prioritize defect fixes Manage defect assignments Update defect information as required Support the classification and prioritization of defect fixes Discuss availability of resources to implement fixes Update defect information as required Support the classification and prioritization of defect fixes Discuss availability of resources to test fixes Update defect information as required Support customer view as part of defect classification and prioritization process Update defect information as required Slide 11 Defect Review Board Arbitration DRB Testing Defect DB Signif. Defect Yes Set Priority Deferred or stored For future consideration No Assigned Devel. Slide 12
Defect Classification Good classification is the foundation for cause-effect analysis Requirements are: Orthogonality Consistency across phases Uniformity across projects/products Keep it simple Slide 13 Defect Attributes State Type Severity Priority Slide 14
Defect States Open: Ready for triage Assigned: in the queue to be fixed Cannot Reproduce: Add details Need Info: Programmer needs more info As designed: program works as designed Deferred: It s a bug, to be fixed later Duplicate: Repeat of another bug, cross reference it Withdrawn: Person who reported withdraws the bug Fixed: Programmer says it s fixed. Re-test Slide 15 Defect Types Function: affects significant capability Assignment: initialization, data structure Interface: component interaction Checking: data validation Timing: shared real-time resources Build: library systems, version control Documentation: notes about the system Algorithm: efficiency, correctness Slide 16
Process Inference All defects that Meet C1&C2 Process Space Classification 2 Classification 1 C1: Defect Type C2: Life Cycle Phase Slide 17 Defect Type Distribution Defect Type by Life Cycle Defect Volume 90 80 70 60 50 40 30 20 10 0 Design Unit Test Integration System Test Function Assignment Interface Timing Source: http://www.chillarege.com/odc/articles/odcconcept/odc.html Slide 18
Severity Classification Critical-1 Major - 2 Minor-3 Cosmetic-4 Keep it simple Define meaning clearly Slide 19 Priority Classification Emergency must be fixed immediately. Often results in a patch release High first priority. To be addressed before all lower priority defects/issues Medium second priority. To be addressed after high priority defects/issues Low only gets addressed after all higher priority defects have been resolved Typically assigned at defect triage Slide 20
Defect Log Name Type Status Precondition Repro Steps Priority Severity Environment Description Defect fix descr. Slide 21 Defect Management Process Defect Prevention Baseline Defect Discovery Defect Resolution Process Improvem. Management Reporting Slide 22
Role of Management All defects are an opportunity to learn Each post-baseline defect can be a symptom of process failure Defect analysis is a rich source of hints for process improvement Each defect is an indication of others not yet detected Slide 23 Defect Management Tools Categories: Test management Defect/issue tracking for traceablity Related tools Configuration management Requirements management Project management Slide 24
Central Repository DRB Chair SQA Developers Project Management Tool and Database Customer Support Product Management Corporate Testers Slide 25 Defect Management Tools Features Workflow engine Status transitions Email notifications Task assignments Links to other DBs Issue history Data analysis and data mining Defect tracking (traceability) Slide 26
Test Progress Test Progress 350 300 250 200 150 100 50 0 Def. Expected Def. Found Def. Found Def. Found Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Slide 27 Residual Risk Residual Risk 250 Total Defects 200 150 100 50 Ship Def. Expected Def. Found 0 Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Slide 28
Defect Density Defect Density Density 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 - A B C D E F G H I J K Defects/FP Component Slide 29 Project Health Project Health New found/week 40 35 30 25 20 15 10 5 0 Healthy Unhealthy Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Slide 30
Defect Backlog Defect Backlog Cumulative Defects 200 180 160 140 120 100 80 60 40 20 0 Defects Found Defects Repaired Week 1 Week 3 Week 5 Week 7 Week 9 Week 11 Week 13 Week 15 Slide 31 People and Defects Everybody likes data nobody likes bad news Treat defects as software assets, not personal performance indicators Effectiveness of defect managers hinges on reliable and timely data Dispute resolution is a valuable skill Slide 32
References - Acknowledgements Some of the material is derived from the QA Labs (now UST GLOBAL) Defect Management process IBM Research (www.chillarege.com) Slide 33 Discussion And Questions Slide 34