Early Software Product Improvement with Sequential Inspection Sessions: An empirical Investigation of Inspector Capability and Learning Effects

Transcription

1 Early Software Product Improvement with Sequential Inspection Sessions: An empirical Investigation of Inspector Capability and Learning Effects Dietmar Winkler, Bettina Thurnher, Stefan Biffl Institute of Software Technology and Interactive Systems Vienna University of Technology

2 Motivation The construction of high-quality software products requires (a) professional approaches (e.g., processes and methods) and well-trained engineers. Early detection and removal of defects, e.g., in the design phase, helps increase software quality and decrease rework effort and cost. Prior empirical studies showed that UBR (software inspection with a usagebased reading technique approach) can focus on most important use cases and spot on the detection of crucial and important defects. Inspection promises to be a vehicle to support learning. Questions: How is the impact of inspector qualification on inspection performance? Is there any notable difference of learning effects regarding inspection performance in a sequence of sessions? 2

3 Defect Detection with Inspection Software Inspection is a static analysis technique to verify quality properties of software. does not require executable code (applicable to design documents). focuses on defect types and location in the inspected object. Guidance of inspectors with reading techniques and guidelines (how to traverse a software document). Best-practice approach: Usage-Based Reading (UBR) Well-investigated reading technique approach. Goal: focus on most important defects first (classes crucial and important ). User focus: use cases lead the inspection process. Application of use cases and scenarios from requirements documents in a pre-defined order (prioritized by a group of experts) to design documents Institut für Softwaretechnik und Interaktive Systeme

4 Learning with Inspection Inspection supports learning due to a systematic and structured process (inspection process) which is repeatable and traceable Active guidance to support individual inspectors in defect detection tasks (guidelines, checklists, etc.) We refer learning as an improvement of individual inspection performance in a sequence of inspection sessions within a similar application domain. Research Questions: Is there any difference of inspection performance regarding system complexity and inspector capability? Can we identify differences of gained additional inspection experience in a second inspection session? Institut für Softwaretechnik und Interaktive Systeme

5 Dependent Variables and Hypothesis Inspection effort includes individual preparation time and inspection duration (we did not consider inspection pre-work, e.g. use case prioritization). Effectiveness is the number of defects in relation to the overall number of seeded (important) defects. Efficiency is the number of defects found per time interval (e.g., defects found per hour) False Positives is the share of "wrong defects found" by individual inspectors. Hypothesis: Effectiveness and efficiency will increase in a second inspection session. False positives will decrease in a second inspection session. Inspectors will perform better in the less complex part of the system Institut für Softwaretechnik und Interaktive Systeme

6 Experiment Description The system represents a snapshot of the development process of a taxi management system including requirements and design documents and source code fragments. Two parts of the system at different levels of system complexity (amount of inspection material). Complexity (Central) > Complexity (Taxi). Total number of 56 seeded important defects within the design specification and the source code. Three experiment phases processed: (a) training & preparation, (b) individual inspection, and (c) data submission.

7 Study Arrangement (2x2 study design) Subjects 104 graduate students in a class on quality assurance and software engineering: 18 less, 22 medium and 12 higher-qualified inspectors per group. The subjects were randomly assigned to 2 groups. Data Set 1 (Central part) is the more complex part and Data Set 2 (Taxi) is the less complex part of the system.

8 Threats to Validity Internal validity: Avoidance of communication between individuals during the study execution. Participants could take individual brakes, whenever necessary (break durations noted). Limitation of the overall inspection duration (3h for taxi, 5h for central due complexity reasons). Experience questionnaire to get an insight on prior knowledge. Feedback questionnaire to see if the participants followed the study process properly. External validity: Well-known application domain to avoid domain-specific interpretation problems. Pilot test and reviews to assure correctness of experiment material. Control of variables due classroom setting Institut für Softwaretechnik und Interaktive Systeme

9 Results: Effectiveness 100 Effectiveness: number of defects found in relation to the number of seeded defects Session 1 Session 2 Central Taxi Central Taxi Mean 34.2% 51.0% 55.7% 74.6% Std.Dev. 17.2% 25.7% 5.5% 11.6% System Complexity: Significant differences between Group A and B in both sessions. Only small advantages for high-qualified inspectors. Similar system parts: significant differences for all qualification classes and both groups. Advantages for less- and medium-qualified inspectors. Group A (Central) Session 1 Group B (Taxi) Effectiveness [%] Effectiveness [%] Group B (Central) Session 2 Group A (Taxi) Qualification low medium high Qualification low medium high

10 Results: Efficiency 14 Efficiency: number of found defects per hour. 12 Session 1 Session 2 Central Taxi Central Taxi Mean Std.Dev System Complexity: We observed significant differences between less- and higher qualified inspectors in both session within the more complex part. Efficiency [per hour] Group A (Central) Session 1 Group B (Taxi) No significant differences in the taxi part Similar system parts: There is a notable learning effect (p<0.040) in session 2 for all qualification classes. We observed the highest improvement factor for medium qualified inspectors and the lowest for higher-qualified inspectors. Group B (Central) Session 2 Group A (Taxi) Efficiency [per hour]

11 Results: False Positives 100 False Positives (FP): share of wrong defects found Session 1 Session 2 Central Taxi Central Taxi Mean 63.3% 36.9% 55.7% 41.7% Std.Dev. 19.4% 10.9% 21.9% 9.6% System Complexity: No significant differences within both sessions. False Positives [%] Group A (Central) Session 1 Group B (Taxi) Similar system parts: 80 The FP decreased for group A (starting with the more complex central part of the system) and increased for group B (starting with taxi). False Positives [%] Group B (Central) Group A (Taxi) 11 Session 2

12 Summary and Further Work Summary: Software inspection is an appropriate method for defect detection in early software development phases and learning. The results showed that inspectors who started with a simple (less complex) system part are more successful regarding effectiveness, efficiency and false positives. We observed a smaller learning effect for higher-qualified inspectors. One reason might be that they apply their own suitable inspection approach. Further Work: More detailed investigation of the results regarding inspector qualification, different defect types and document locations will lead to a deeper insight in learning with inspection. 12

13 Backup 13

14 Study Artifacts System Overview: Taxi Management System Software artifacts per part Central (DS1) Taxi (DS2) Requirements 4 pages 4 pages definitions 1,400 words 1,100 words 16 use cases 11 use cases Design document Source Code in Lines of Code (LoC) Common system information 5 pages 3 pages 1,600 words 800 words 750 LoC 650 LoC 2,100 words 1,400 words 1 component diagram 1 class diagram 11 sequence charts Seeded defects per part Central (DS1) Taxi (DS2) Total No. of Defects Share of Defects 54% 46% 100% Data Set 1 (Central part) is the more complex part (higher amount of inspection material) of the system.