Software Quality Assurance Slide 1 Objectives Understand different approaches to software quality assurance Understand the nature of software defects Be able to record and track defects in your project Slide 2
Quality Assurance Quality Assurance Fault Avoidance Fault Detection Fault Tolerance Slide 3 Software quality definitions - relevance high Relevance to producer Productivity LOC or FP per month Process maturity/stability capability index Conformance to schedule deviation from planned budgets/ requirements Timeliness Time to market Conformance to requirements delivered defects per KLOC low Relevance to customer high Slide 4
Software quality attributes Reliability Functionality Usability Efficiency Product in operation (user perspective) Maintainability Reusability Portability Testability Product revision (developer perspective) Slide 6 Reliability Probability the system operates without failure in a given period of time under a given operational environment But what exactly is a failure? Slide 8
What is a software failure? Formal view deviation from specified behaviour Engineering view deviation from required, specified or expected behaviour Slide 9 Human errors, faults, and failures? can lead to can lead to human error fault failure Human Error: Designer s mistake Fault: Encoding of an error into a software document/product Slide 10
Processing errors Human Error Fault Input Processing Error Failure Slide 11 Relationship between faults and failures (Adams 1984) Faults Failures (sized by MTTF) 35% of all faults only lead to veryrare failures (MTTF>5000 years) Slide 12
The relationship between faults and failures Small proportion of faults lead to most frequent failures Most faults are benign Removing faults may not lead to greatly improved reliability Does not mean we should stop looking for faults But do not equate fault counts with reliability Slide 13 Using fault data to predict reliability Pre-release testing Post-release operation delivery Slide 14
Pre-release vs post-release faults? 30 20 Post-release faults? 10 0 0 40 80 120 160 Pre-release faults Slide 15 Pre-release vs post-release faults: actual 30 20 Post-release faults 10 0 0 40 80 120 160 Pre-release faults Slide 16
The problem with problems Defects Faults Failures Bugs Anomalies Crashes Slide 17 Incident Types Failure (in pre or post release) Fault Change request Slide 18
Recording incidents Applicable to all incident types What: Product details Where (Location): Where is it? Who: Who found it? When (Timing): When did it occur? What happened (End Result): What was observed? How (Trigger): How did it arise? Why (Cause): Why did it occur? Severity/Criticality/Urgency Change Slide 19 Example: Failure Data What: ABC Software Version 2.3 Where: Norman s home PC Who: Norman When: 13 Jan 2000 at 21:08 after 35 minutes of operational use End result: Program crashed with error message xyz How: Loaded external file and clicked the command Z. Why: <BLANK - refer to fault> Severity: Major Change: <BLANK> Slide 20
Example: Fault Data (1) - responsive What: ABC Software Version 2.3 Where: Function <abcd> in Module <ts0023> Who: Simon When: 14 Jan 2000, after 2 hours investigation What happened: Caused reported failure id <0096> How: <BLANK> Why: Missing exception code for command Z Urgency: Major Change: exception code for command Z added to function <abcd> and also to function <efgh>. Closed on 15 Jan 2000. Slide 21 Example: Fault Data (2) - reactive What: ABC Software Version 2.3 Where: Help file, section 5.7 Who: Norman When: 15 Jan 2000, during formal inspection End result: Likely to cause users to enter invalid passwords How: The text wrongly says that passwords are case sensitive Why: <BLANK> Urgency: Minor Change: Suggest rewording as follows... Slide 22
Example: Change Request What: ABC Software Version 2.3 Where: File save menu options Who: Norman When: 20 Jan 2000 End result: <BLANK> How: <BLANK> Why: Must be able to save files in ascii format - currently not possible Urgency: Major Change: Add function to enable ascii format file saving Slide 23 Slide 24
Slide 25 Tracking incidents to components Incidents need to be traceable to identifiable components - but at what level of granularity? Unit Module Subsystem System Slide 26
The SEI Capability Maturity Model Level 2: Repeatable Level 1: Initial/ad-hoc Level 3: Defined Level 4: Managed Peer reviews Training programme Intergroup coordination Integrated s/w management Organization process definition/focus S/W configuration management S/W QA S/W project planning S/W subcontract management S/W requirements management Level 5: Optimising Process change management Technology change management Defect prevention Software quality management Quantitative process mgment Slide 29 Results of 1987-1991 SEI Assessments Column Title All 59 46 self 13 SEI Level 1 81% 87% 62% Level 2 12% 9% 23% Level 3 7% 4% 15% Level 4 0% 0% 0% Level 5 0% 0% 0% Slide 30
In-process defects/maeloc Process improvement at Motorola 1000 800 600 400 200 0 1 2 3 4 5 SEI level Slide 31 History of Inspections...a formal evaluation technique in which software requirements, design or code are examined in detail by a person or group other than the author to detect faults, violations of development standards and other problems IEEE Standard for Software Reviews and Audits Original method developed by Fagan at IBM In use since early 1970s Highly formalised method Reliant on generation of quantitative defect data Slide 32
Motivation for Inspections: Cost to Fix a Fault Cost to fix a fault by development phase Relative cost to fix an error 1 3-6 10 15-40 30-70 40-1000 Reqmts. Design Coding Dev. Testing Acc. Testing Operation Slide 33 The Inspection Process check - lists process ideas change requests exit Edit and Follow-up Logging Meeting Checking Kick-Off entry Planning Slide 34
Inspections: Defect Logging Meeting Listens Re-works material Author Reader Read material Spot defects Moderator Distributes material Schedules meeting Chairs meeting Writes report Chases rework Decides pass/fail Secretary Records defects Slide 35 Inspections: Management Issues Only the quality of the product should be considered when evaluating an individual s work Goal for inspections is to detect as many defects as possible Remember the number of defects will depend on the problem s complexity and the skills of staff and on thoroughness of inspection Don t use defects found as a measure of individual performance Do not punish individuals for defects Slide 36
Inspections: Critical Success Factors Logging meeting must focus on defect detection not discussion Logging meeting should strive to create synergy Optimum checking rates should be respected (approx. 1 page per hour) Data collection should be structured and automated for later analysis Slide 37 Inspections: Costs and Benefits Benefits claimed productivity increases of up to 100% claimed 10 fold reduction in testing costs 2/3 reduction in maintenance costs team more likely to deliver system on time Costs consume up to 15% of development budget high training costs (1 week per person) data analysis effort meetings perceived as overhead Slide 38
Inspections Vs Testing Not mutually exclusive alternatives Testing done on executable code - Inspection performed on documents or source code Important to inspect test cases and documentation before testing is performed Slide 39 Summary Software quality is multi-faceted Do not equate faults and failures Use standard method for tracking incidents Reviews and inspections are efficient QA procedures Slide 40