Lecture Software Engineering CHAPTER 06 TESTING Lecture Software Engineering Topics Introduction Quality Issues Execution-Based Testing What Should be Tested Testing versus Correctness Proofs Who Should Perform Execution-Based Testing When Testing Stops Lecture Software Engineering Reference Schach, S.R. (2010). Object-Oriented and Classical Software Engineering, Eighth Edition. McGraw-Hill, ISBN: 978-0-07-337618-9. Introduction Testing Integral component of software process Testing must be carried out throughout life cycle: Continual testing by development team while it performs each workflow More methodical testing at the end of each workflow Verification versus validation Verification Process of determining whether a workflow has been correctly carried out Validation Evaluation process before the product is delivered Prior definition Verification Are we building the product right Validation Are we building the right product Terms defined in IEEE software engineering glossary [IEEE 610.12] Term testing is used instead
Introduction Two types of testing Execution-based testing Testing by running executable code Non-execution-based testing Testing of code, documents and reports Importance of testing cannot be stressed too strongly Quality Issues Definitions (IEEE Standard s terminology) [IEEE 610.12] Fault Results when a human makes a mistake Failure Observed incorrect behavior of product as a consequence of a fault Error Amount by which a result is incorrect Defect Generic term for a fault, failure, or error A specific failure may be caused by several faults Some faults may never cause a failure Quality of software Extent to which product satisfies its specifications Product must be well designed and coded Task of every software professional is to ensure high-quality software at all times Quality cannot be added afterward by SQA group SQA ensures that developers are doing high-quality work Quality Issues Software Quality Assurance Software Quality Assurance (SQA) group ensures that developers product is correct Ensure that each workflow has been carried out correctly Ensure that complete product is correct Develop standards to which software must conform Establish monitoring procedures for assuring compliance Role of SQA group is to ensure the quality of software process: Thereby ensure the quality of product Quality Issues Managerial Independence There should be managerial independence between development team and SQA group Development team manager: Delivering faulty software on time SQA team manager: Delivering good software late Senior manager deciding appropriate action SQA group separate In an organization with 100 software professionals If each spends n% on quality assurance Divide into n for SQA group and 100-n for development Very small organizations (four or fewer) Maybe not economically viable to have a separate SQA group
Non-execution-based testing Testing software without running test cases Reviewing software Analyzing software mathematically Review Review task must be done by someone other than original authors of documents Having only one reviewer may not be adequate Walkthroughs and inspections are two types of reviews Walkthroughs A walkthrough team should consist of four to six individuals An analysis walkthrough should have at least one representative from Team responsible for specification Manager responsible for analysis workflow Client representative Representative from next workflow team (design team) SQA (chaired by SQA representative) Members should be experienced senior technical staff Material must be distributed well in advance Each reviewer must study the material Develop two lists Items reviewer does not understand Items reviewer believes to be incorrect Managing Walkthroughs SQA representative has most at stake and should therefore chair the walkthrough Person leading the walkthrough guides other members Faults are detected and recorded, not corrected (1) Correction may be inferior due to time constraints (2) Inefficient to have all participants time spent (3) Some correct items could be flagged as faults (4) Not enough time to detect and correct Two ways of conducting a walkthrough Participant-driven: Members presenting lists of items Document-driven: Walking through the document Likely to be more thorough Leads to detection of more faults IEEE Standard for Software Reviews [IEEE 1028]: Prescribes thorough document-driven review Primary role of walkthrough leader is to elicit questions and facilitate discussion: Interactive process Inspections An inspection goes far beyond a walkthrough and has five formal steps (1) Overview: Present document before distribution (2) Preparation: Understand document and list faults (3) Inspection: Walk through document, cover entire fault list Leader (moderator) produces a written report for followthrough (4) Rework: Responsible individual resolves all faults noted in report (5) Follow-up: Moderator ensures resolution of all issues Fixing document or clarifying items incorrectly flagged Check that fixes do not introduce new faults If more than 5% reworked, then 100% is reinspected
Inspection should be conducted by a team of four IEEE standards recommendation [IEEE 1028] Between three to six participants Moderator Reader: Leads the team through process Recorder: Produces a written report of detected faults An essential component of an inspection is checklist of potential faults E.g., do actual and formal arguments correspond for interfaces An important component of the inspection procedure is record of fault statistics Fault are recorded by severity (major or minor) and Fault types (e.g., interface and logic faults) Fault statistics information used Comparison with other products for early warnings Identifying troubled artifacts (disproportionate number of faults) An artifact with far more faults can be redesigned Used at later stages (design inspection used in code inspection) Surveys show that 67% to 93% of faults can be discovered during inspection: Programmer productivity increases One advantage of code inspection over running test cases is that testers need not deal with failure Fault causing failure when running code must be located and fixed In non-execution-based testing, a fault is logged and process continues A risk of inspection process is that it might be used for performance appraisal: Top management should be aware of potential problem Comparison of Inspections and Walkthroughs Walkthrough is a two-step process: Preparation followed by team analysis of document Inspection is a five-step process: Overview, preparation, inspection, rework, and follow-up Inspection procedures are formalized Inspection process take much longer Strengths and Weaknesses of Reviews Major strengths of a review Review is an effective way of detecting a fault Faults are detected early in software process Effectiveness of review can be reduced if software process is inadequate Large-scale software is difficult to review unless it consists of largely independent components: A strength of the objectoriented paradigm Unless documentation of previous workflows is complete, updated, and available, the effectiveness of team review is hampered Design review team may refer to analysis artifact Code review team may refer to design documents Metrics for Inspection Inspection rate Number of pages or lines of code inspected per hour Fault density Faults per page inspected or per KLOC
Fault detection rate Number of faults (major and minor) detected per hour Fault detection efficiency Number of faults (major and minor) detected per person-hour The purpose of these metrics is to measure the effectiveness of the inspection process Results may reflect deficiencies of development team As fault detection rate increases E.g., from 20 faults per KLOC to 30 Inspection team more efficient or Quality of code has decreased Execution-Based Testing Program testing can be very effective in detecting faults, but it cannot prove the absence of faults What Should be Tested Utility Extent to which a user s needs are met when a correct product is used under conditions permitted by its specifications Reliability Measure of frequency and criticalness of product failure How often product fails (mean time between failures) How bad effects of that failure can be How long it takes, on average, to repair it (mean time to repair) Robustness Function of a number of factors such as range of operating conditions, possibility of unacceptable results with valid input, and acceptability of effects of invalid input Product with wide range of operating conditions more robust than a more restrictive one What Should be Tested Performance Extent to which product meets its constraints Response time Space requirements Embedded computer systems Integral part of a larger system Embedded software controls device in which computer is embedded E.g., computer chip in a washing machine Space constraints Real-time software Characterized by hard time constraints If a constraint is not met, information is lost E.g., a nuclear reactor control system sampling core temperature Use of a simulator: Working model of the environment
What Should be Tested Correctness A product is correct if it satisfies its output specifications, independent of use of computing resources, when operated under permitted conditions Issue of incorrect specifications E.g., specifications for a sort Figure 6.1 Figure 6.2 and Figure 6.3 What Should be Tested Testing versus Correctness Proofs Correctness proof A mathematical technique for showing that a product is correct: It satisfies its specification Use of Input specification Output specification Assertion A mathematical expression that holds Loop invariant A mathematical expression that holds irrespective of how many times the loop is executed E.g., sum of elements of an array Show that after code is executed S will contain sum of n elements of array y Figure 6.4 6.5 6.6 Testing versus Correctness Proofs Figure 6.5 and Figure 6.6
Testing versus Correctness Proofs Correctness Proofs and Software Engineering Software engineers lack adequate mathematical training computer science graduates with sufficient mathematical skills Proving is too expensive to be practical Essential in life-critical situations determined using cost-benefit analysis Proving is too difficult Many nontrivial successfully have been proven correct E.g., operating system kernels, compilers, communication systems Difficulties Ensuring that a theorem prover is correct Finding input and output specifications and the loop invariants Specifications themselves can be incorrect Testing versus Correctness Proofs Proving products correct is an important, and sometimes vital, software engineering tool When a full formal proof is not justified: Quality of software can be improved through informal proofs Model checking A new technology that may take the place of correctness proving of software Testing technology for hardware that is starting to be applied to software Who Should Perform Execution-Based Testing Testing is a destructive process with intention of finding faults: Programmers should not test their own code artifacts Programmer must desk check code artifact: Before it undergoes systematic testing by others SQA group: Independent from development team Test cases should never be thrown away Test cases with expected outputs Used during postdelivery maintenance Used for regression testing When Testing Stops After being successfully maintained for many years: A software product is decommissioned and removed form service Product may lose its usefulness Cost of porting to new system may be prohibitive Only after product is discarded: It is time to stop testing