Software Quality Assurance: Introduction

Transcription

1 Software Quality Assurance: Introduction Room E Tel [email protected]

2 Outline I Introduction II Software Life Cycle III Quality Control IV Infrastructure V Management VI Standards VII Conclusion & Summary I-2

3 I Introduction I.1 Motivation I.2 Definitions & Terminology I.3 Quality Management I.4 Components of a SQA System I.5 Organizing for SQA I.6 Discussion & Summary I.7 Bibliography I-3

4 I.1 Motivation [Galin2004] The Software Crisis: IBM Consulting group estimates that 55% of large distributed systems projects cost more than expected, 68% overrun their schedules, and 88% require redesign. The Standish group estimated the cost of bad software for US businesses at $85 billion for The Y2K problem was estimated to cost $1 to $2 trillion. W.W Gibbs, in Software's Chronic Crisis in the Scientific American, September 1994 estimates that the average software project overshoots its schedule by half. How to develop quality software? I-4

5 Software's Chronic Crisis Denver's new international air port was to be the pride of the Rockies, a wonder of modern engineering. Twice the size of Manhattan, 10 times the breadth of Heathrow, the airport is big enough to land three jets simultaneously-in bad weather. Even more impressive than its girth is the airport's subterranean baggage-handling system. Tearing like intelligent coal-mine cars along 21 miles of steel track, 4,000 independent "telecars" route and deliver luggage between the counters, gates and claim areas of 20 different airlines. A central nervous system of some 100 computers networked to one another and to 5,000 electric eyes, 400 radio receivers and 56 bar-code scanners orchestrates the safe and timely arrival of every valise and ski bag. At least that is the plan. For nine months, this Gulliver has been held captive by Lilliputianserrors in the software that controls its automated baggage system. Scheduled for takeoff by last Halloween, the airport's grand opening was postponed until December to allow BAE Automated Systems time to flush the gremlins out of its $193-million system. December yielded to March. March slipped to May. In June the airport's planners, their bond rating demoted to junk and their budget hemorrhaging red ink at the rate of $1.1 million a day in interest and operating costs, conceded that they could not predict when the baggage system would stabilize enough for the airport to open. W. W. Gibbs. Software's chronic crisis. Scientific American, 271(3):72--81, September I-5

6 Denver International Airport Baggage System The Denver International Airport has a modern, automated baggage-handling system designed by BAE Automated Systems, Inc. (In June, 2003 G & T Conveyor Company, Inc. acquired BAE) See: See: Schloh, Schloh, Michael. Michael. Analysis Analysis of of the the Denver Denver International International Airport Airport baggage baggage system system t/csc463.html t/csc463.html Eipe, Eipe, Rohit. Rohit. The The importance importance of of software software architecture: architecture: Denver Denver International International Airport's Airport's automated automated baggage baggage handling handling system: system: A report report :+Denver+Intnl+Airport:+Baggage+System :+Denver+Intnl+Airport:+Baggage+System Nice, Nice, Karim. Karim. How How Baggage Baggage Handling Handling Works, Works, HowBIZWorks HowBIZWorks I-6

7 Automated Baggage-Handling System (1/3) check-in conveyer belt plane unloading load DCV The baggage handling system at an airport plays a crucial role. It makes the difference in an airport's ability to attract or keep a major airline hub ("an airport that serves as a central connecting point) A baggage-handling system has three main jobs: Move bags from the check-in area to the departure gate Move bags from one gate to another during transfers Move bags from the arrival gate to the baggage-claim area Conveyors equipped with junctions and sorting machines automatically route the bags to the gate. I-7

8 Automated Baggage-Handling System (2/3) a DCV (telecar) tunel system load DCV unload DCV Destination-coded vehicles (DCVs), unmanned carts (also named telecars) propelled by linear induction motors mounted to the tracks can load bags (without stopping) can unload bags (without stopping) I-8

9 Automated Baggage-Handling System (3/3) unload DCV DIA has: More than 5 miles (8 km) of conveyor belts More than 19 miles (30 km) of DCV tracks About 4,000 DCVs enabling it to handle over 1,000 bags per minute. baggage-claim load plane I-9

10 Observation: First System Test (March 1994) Telecars jumped tracks, crashed into each other; pieces of baggage were flung off telecars, in some cases chewed to pieces. Baggage was sent to the wrong places. Line balancing algorithm problems arose, where there were bags lined up waiting for telecars to come by, and yet empty telecars would just go by without picking up these bags. In peak conditions, the system quickly became overloaded with tracking all the telecars in transit. Head of line blocking occurred at popular telecar intersections in the tunnel network, and rerouting of approaching telecars didn t always take place. Poorly printed baggage tags led to the laser scanners at one point rejecting 70% of all baggage tags, sending them to manual baggage stations. Remark 1: This famous demo of the system in which hundreds of bags were destroyed was not done by BAE, but was done by the City of Denver when BAE told them not to do it because the system was not working. BAE claimed that the city violated the contract many times, making it nearly impossible for them to finish the job. Eventually the city quit managing the project and United Airlines took over, and BAE was able to finish the project. I-10

11 Observation: Timeline (1/2) The project was begun in the mid 80s by US Transportation Secretary Fredrico Pena, the previous Denver mayor. Construction was supposed to begin in September 1989 with an opening date set for October 31st On March 2nd, 1993 the first delay was announced, changing the date to December 19th, October that year was the second delay. March 1994 was the third. After this Logplan was brought in, yet the fourth delay was announced on May 2nd, DIA finally began operations on February 28th, I-11

12 Observation: Timeline (2/2) DIA finally began operations sixteen months behind schedule, with a reduced number of gates operational, concourse A being postponed indefinitely, the only having been implemented in concourse B, an inability to handle transfer flights, and with a capacity for handling only 30 bags a minute. As of November 1996, DIA s automatic baggage handling system was still not working. The airport was not even increasing its air traffic; in 1995 the number of passengers dropped by 2 million from when Stapleton was running, in High fees associated with Denver charging each airline a $20 fee per passenger, which must have been passed on to the customer, are supposedly the reason for this disappointing performance. I-12

13 Observation: Financial Damages Automated baggage handling system: Originally planed budget was $193 million the amount finally spent was close to $311 million (included the cost of installing a manual system) DIA planed DIA costs were $1.7 billion the final cost was more than 2.5 times that, at $4.5 billion Delay costs: estimated costs for every month that the DIA remained closed was about $33.3 million I-13

14 Cause: Lack of Management The baggage handling system has been implemented almost as an afterthought! Thus the system of tunnels was not designed with a particular baggage handling system in mind, and as it turned out, the small tunnels and many sharp turns made BAE s job rather difficult. The airport designers didn t even design their system with a baggage handling system in mind, so BAE had to work around all kinds of problems. This was the reason that the management team never saw the baggage handling system, of all things, to be so important that it could have such a profound economic impact on the project. The planning of the system was done in a completely haphazard way. Little communication took place between DIA s airport designers, the city officials, the airlines, and BAE. The planning instead was driven by higher level management with little or no understanding of the complexity of the system, and by consultants contracted to develop a specification of the system, but who had no direct responsibility to the team(s) that would actually develop the system. I-14

15 Cause: Impossible Schedule/Plan One reason, if it might be called that, that BAE didn t perform a review of their own design was that they were being made to work under an impossible tight schedule. BAE claim that they protested at the outset that the timeline Denver city officials proposed for the completion of the project was utterly unrealistic. Denver on the other hand claims that BAE committed to delivering the system within a certain period of time, and should have delivered a bug free system in that time. Whoever the culprit, one thing is for sure, the time required to fully understand and do justice to a system of this size and complexity was grossly underestimated, and this had direct bearing on the efficiency of the design that was put forth, and ultimately implemented. identify problems with the design, and take decisions about them early, instead of reacting to those problems after they were uncovered in testing BAE and Denver didn t heed the advice of others, and grossly underestimated the time for developing and testing the system (The Munich airport, on which DIA was modeled, spent two years testing the system and six months of running the system 24x7 before opening) I-15

16 Cause: Complexity Problems As it turned out, when a change needed to be made to one part of the system, it was not clearly understood how that change would impact the rest of the system. When the system was eventually test-run, BAE found it incredibly difficult to even understand why things were going wrong. BAE s design was such that the number of components working together was very high, and they were very tightly coupled, and there was little redundancy. sheer size of the system made it so complex that not even the people who designed it were able to understand it very well, leave alone a third party to weed out problems at the design stage than to implement a design that had rather glaring faults, and then try to perform frantic firefighting I-16

17 Cause: Many Interfaces One extremely difficult problem that BAE faced was the task of conversing with United Airlines Apollo reservation computers. BAE s computers had to speak the same software language as each of the airlines The process of language translation that was necessitated was painful and bug-filled at best. I-17

18 DIA Conclusions Missing communication between the stakeholders The schedule didn t allow enough time to fully design the system before jumping into implementation. The design of the baggage handling system was not integrated into the design of the airport. Not entirely inappropriately, the DIA was coined as being DOA dead on arrival! I-18

19 How can we circumvent such disasters? Observation: Quality problems prevent the system from functioning Management decisions results in wrong reactions Missing design reviews Fixing rather than analyzing Erratic debugging rather than systematic testing Therefore even larger delays result Systematic approach for the development of high quality software is required I-19

20 I.2 Definitions & Terminology Software Definition & characterization Why do the classical approach to QA not apply? Software quality Definition Quality models Quality attributes Measure quality Reliability, Availability, safety, maintainability, Defect counts Software errors, faults and failures Classification of the causes of software errors I-20

21 Basic Terminology Software is: Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system. [IEEE_Std_ ] I-21

22 What is special about Software? Invisibility of the product [Galin2004] Limited opportunities to detect defects ( bugs ) Often new demanding functionality has to be realized Often software has to realize extraordinary high complexity I-22

23 Classical Quality Assurance (Hardware) Requirements: complete set of external quality characteristics complete set of internal quality characteristics and a detailed and complete set of requirements and specifications Design: Design that satisfies the requirements and specifications design for reliability (wear out) design for manufacturability design for maintainability (e.g., self-diagnosis) Manufacturing: statistical production process control with acceptance sampling Operation: collect failure data for continuous improvement and predictions (intelligent maintenance) I-23

24 Software Quality Assurance? Requirements: Completeness is hard to achieve (complexity) Design: design for reliability only in special cases design for manufacturability is not required design for maintainability in special cases Focal area of software quality assurance! Manufacturing Implementation: limited success with statistical process control (metrics) Operation: Fixing found bugs I-24

25 What is quality? [Sommerville2004] Quality, simplistically, means that a product should meet its specification. This is problematical for software systems There is a tension between customer quality requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.); Some quality requirements are difficult to specify in an unambiguous way; Software specifications are usually incomplete and often inconsistent. I-25

26 Approaches to Tackle Quality Transcendental view: quality is universally identifiable, absolute, unique and perfect Product view: the quality of a product is measurable in an objective manner User view: quality is fitness for use Manufacturing view: quality is the result of the right development of the product Value-based view (Economic): quality is a function of costs and benefits I-26

27 Software Quality IEEE View Software quality is: (1) The degree to which a system, component, or process meets specified requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations. [IEEE_Std_ ] I-27

28 Software Quality Software quality is : Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software. [Pressman2004] I-28

29 Software Quality Quality the degree of excellence of something. We measure the excellence of software via a set of attributes. [Glass1992] ( satisfying requirements )! Quality is absolute or at least independent of the requirements underlying the product. It is part of the software development to get the right requirements. ( user satisfaction )! McDonald s restaurants are popular, but I-29

30 Quality Models Such general definitions of software quality are not sufficient in practice Thus, software quality is described by specific quality models causal relationship from intangible quality views to tangible software measures Two main approaches: Standard Models: McCall ISO/IEC 9126 Application or company specific quality models FURPS GQM Approach [ISO/IEC 9126] I-30

31 Factor-Criteria-Metrics-Model Classification into : Factors (to specify): They describe the external view of the software, as viewed by the users. Criteria (to build): They describe the internal view of the software, as seen by the developer. Metrics (to control): They are defined and used to provide a scale and method for measurement. Quality factor 1 Quality criterion 1 Software Quality Quality factor 2 Quality criterion 2 Quality factor 3 Quality criterion 3 Quality indicators (metrics) I-31

32 McCall s Factor Model Tree [MaCall+1977] [Galin2004] A quality factor represents a behavioral characteristic of the system. Operation Revision Transition A quality criterion is an attribute of a quality factor that is related to software production and design. A quality metric is a measure that captures some aspect of a quality criterion. I-32

33 The Six Quality Characteristics of a Software (ISO/IEC 9126) Software quality: The totality of features and characteristics of a software product that bear on its ability to satisfy stated or implied needs. (ISO 9126: 1991, 3.11) Software quality characteristics: A set of attributes of a software product by which its quality is described and evaluated. A software quality characteristic may be refined into multiple levels of sub-characteristics. (ISO 9126: 1991, 3.13) [ISO/IEC 9126] Each characteristic is refined to a set of sub-characteristics Each sub-characteristic is evaluated by a set of metrics. Some metrics are common to several sub-characteristics. I-33

34 Characteristics Functionality Reliability Usability Subcharacteristics Suitability Accurateness Interoperability Compliance Security Maturity Fault tolerance Recoverability Understandability Learnability Operability Definitions Attributes of software that bear on the presence and appropriateness of a set of functions for specified tasks. Attributes of software that bear on the provision of right or agreed results or effects. Attributes of software that bear on its ability to interact with specified systems. Attributes of software that make the software adhere to application related standards or conventions or regulations in laws and similar prescriptions. Attributes of software that bear on its ability to prevent unauthorized access, whether accidental or deliberate, to programs or data. Attributes of software that bear on the frequency of failure by faults in the software. Attributes of software that bear on its ability to maintain a specified level of performance in case of software faults or of infringement of its specified interface. Attributes of software that bear on the capability to re-establish its level of performance and recover the data directly affected in case of a failure and on the time and effort needed for it. Attributes of software that bear on the users effort for recognizing the logical concept and its applicability. Attributes of software that bear on the users effort for learning its application. Attributes of software that bear on the users effort for operation and operation control. I-34

35 Characteristics Efficiency Maintainability Portability Subcharacteristics Time behaviour Resource behavior Analyzability Changeability Stability Testability Adaptability Installability Conformance Replaceability Definitions Attributes of software that bear on response and processing times and on throughput rates in performances its function. Attributes of software that bear on the amount of resource used and the duration of such use in performing its function. Attributes of software that bear on the effort needed for diagnosis of deficiencies or causes of failures, or for identification of parts to be modified. Attributes of software that bear on the effort needed for modification, fault removal or for environmental change. Attributes of software that bear on the risk of unexpected effect of modifications. Attributes of software that bear on the effort needed for validating the modified software. Attributes of software that bear on the opportunity for its adaptation to different specified environments without applying other actions or means than those provided for this purpose for the software considered. Attributes of software that bear on the effort needed to install the software in a specified environment. Attributes of software that make the software adhere to standards or conventions relating to portability. Attributes of software that bear on opportunity and effort using it in the place of specified other software in the environment of that software. I-35

36 Hewlett Packard: F.U.R.P.S. (1/2) Result of a statistical project survey at Hewlett Packard 1987 to improve its products: Factors: Functionality: functions it performs, their generality and security Usability: aesthetics, consistency, documentation Reliability: frequency and severity of failure, accuracy of output Performance: response time, resource consumption Supportability: can it be extended, adapted, corrected? FURPS is originally a company specific quality model I-36

37 Hewlett Packard: F.U.R.P.S. (2/2) Quality Functionality Usability Reliability Performance Supportabilit y Feature set Human factors Frequ./severity of fail. Speed Testability Capabilities Generality Security Aesthetics Consistency Documentation Recoverability Predictability Accuracy Efficiency Rsc. consumption Thruput Extensibility Adaptability Maintainability Mean time to failure Response time Compatibility Configurability Serviceability Installability Localizability I-37

38 GQM: Goal-Question-Metric A measurement program can be more successful if designed with the goals in mind. GQM approach provides a framework with 3 steps: 1. List the major goals of the development/maintenance project 2. Derive from each goal the questions that must be answered to determine if the goals are being met 3. Decide what must be measured to answer the questions adequately [Basil&Rombach1988] I-38

39 Example GOAL: Evaluate effectiveness of coding standard QUESTIONS: Who is using Standard? What is coder productivity? What is code quality? METRICS: Proportion of Coders -using standard -using language Experience of Coders -with standard -with language -with environment etc Code size Effort Errors I-39

40 GQM Discussion Benefits : generates only those measures relevant to the goal Several measurements may be needed to answer a single question. A single measurement may apply to more than one question. The goal provides the purpose for collecting the data. Not evident from the GQM The model needed to combine the measurement in a sensible way so that the question can be answered. It must be supplemented by one or more models that express the relationship among the metrics. (equation definition is not clear) Disadvantages: Additional efforts to derive the goals and metrics Error prone compared to standard models I-40

41 Measuring Quality? User s view: Reliability (number of failures over time) Availability Usability etc. Often directly measurable Manufacturer s view: Defect counts Rework costs I-41

42 Reliability Reliability is the probability of a component, or system, functioning correctly over a given period of time under a given set of operating conditions. Quantitative: Reliability R(t) is the probability that the system will conform to its specification throughout a period of duration t. Note that t is important If a system only needs to operate for ten hours at a time, then that is the reliability target. I-42

43 Availability The availability of a system is the probability that the system will be functioning correctly at any given time. Quantitative: Availability A (or V) is the percentage of time for which the system will conform to its specification. Literally, readiness for service I-43

44 Safety [Storey1996] Safety is a property of a system that it will not endanger human life or the environment. A safety-related system (safety-critical system) in one by which the safety of equipment or plant is assured. I-44

45 Maintainability Maintenance is the action taken to retain a system in, or return a system to, its designed operating conditions. Maintainability is the ability of a system to be maintained (ability to undergo repairs and modifications). Mean time to repair (MTTR) Problem: Maintenance induced failures I-45

46 Defect Counts software errors are the cause of poor software quality [Galin2004] In software, higher quality (in the form of lower defect rates) and reduced development time go hand in hand. [McConnell 1996] Unlike the low-defect kind of quality, attention to this kind of quality tends to lengthen the development schedule. [McConnell 1996] Intuitively, the occurrence of defects is negatively related to functionality and reliability. Defects also interfere, to some degree, with other dimensions of quality. I-46

47 Relative Cost of Correcting Defects [Pressman2004] I-47 9

48 Faults, Errors and Failures A fault is a defect within the system. (Error cause, German: Fehlerursache) An error is a derivation of the required operation of the system or subsystem. (Manifestation of a fault in a system, German: Fehlzustand) software errors are a human action which results in software containing a fault A system failure occurs when the system fails to perform its required function. (Transition to incorrect service delivery, German: Ausfall) If the distinction between fault and failure is not critical, defect can be used as a generic term. [Storey1996,Liu1996] I-48

49 Recursive Nature of Faults A fault in a component can infect all other components which depend on it. Chain of Faults/Failures User Operator??? Physical process/system System (server)? component (server) component (server) System? component (server) component (server) component (server) I-49

50 Fault/Failure Chain Fault Error Failure Fault Error Failure Fault error next hierarchy level a fault which has not been activated by the computation process is dormant a fault is active when it produces an error Error failure an error is latent when it has not been recognized an error is detected by a detection algorithm/mechanism Failure fault a failure occurs when an error passes through and affects the service delivered a failure results in a fault for the system which contains or interacts with the component I-50

51 Examples for Fault/Failure Chain Program error (software): a dormant fault in the written software (instruction or data) upon activation the fault becomes active and produces an error (system state) if the erroneous data affects the delivered service, a failure occurs Electromagnetic interference (hardware): leads to faulty input value (either digital or analog) by reading the input the fault becomes active and produces an error if the erroneous input value is processed and becomes visible at the interface a failure occurs I-51

52 Fault/Failure state transitions I-52

53 Fault Classification Nature (Critical distinction): random/hardware faults logical/systematic/design faults E.g., Degradation (wear-out) vs. design faults Duration: permanent, transient, intermittent Extent: localized, global I-53

54 Nature: Degradation Faults The system used to work but now it does not Something broke for some reason: Infant mortality End of useful life wear out Physical damage vibration, humidity, temperature Examples: Light bulb burns out Disk head crashes Power supply fails I-54

55 Nature: Failure Bathtub Curve Infant mortality wear out I-55

56 Nature: Design Faults The system never worked Nothing broke The design is flawed Examples: Incorrect electrical wiring Software defect! I-56

57 Dealing With Faults Fault Prevention: Development techniques that prevent the introduction of faults Fault Elimination: Development techniques that find and remove faults already introduced Fault Forecasting: Estimate current number, future incidence and likely consequences Fault Tolerance (not considered here): Execution-time techniques that cope with the effects of faults I-57

58 Fault Avoidance Fault prevention Fault removal Fault forecasting int i; while(i<10){ if (i>3) { } } Activities: Preventing the introduction or occurrence of faults (fault prevention): SWT, formal methods, high level languages, CASE tools, Reducing the number and seriousness of faults (fault removal): Analysis diagnosis correction Evaluating the present number future incidents, and severity of faults (fault forecasting): Statistics & management I-58

59 Main causes for software faults: 1. Faulty requirements definition 2. Client-developer communication failures 3. Deliberate deviations from software requirements 4. Logical design errors 5. Coding errors 6. Non-compliance with documentation and coding instructions 7. Shortcomings of the testing process 8. User interface and procedure errors 9. Documentation errors [Galin2004] I-59

60 I.3 Quality Management [Sommerville2004] Quality Management System [ISO 9000]: The organizational structure, responsibilities, procedures, processes and resources for implementing quality management Concerned with ensuring that the required level of quality is achieved in a software product. Involves defining appropriate quality standards and procedures and ensuring that these are followed. Should aim to develop a quality culture where quality is seen as everyone s responsibility. I-60

61 Environment Characteristics [Galin2004] Being contracted Subjection to customersupplier relationship Requirement for teamwork Need for cooperation and coordination with other development teams Need for interfaces with other software systems Need to continue carrying out a project while the team changes Need to continue maintaining the software system for years I-61

62 The Cost of Quality Cost of Quality includes all costs incurred in the pursuit of quality or in performing quality related activities such as appraisal costs, failure costs and external failure costs. [Pressman2004] The Quality Compromise: We cannot wait for specifications to improve before paying attention to quality management. We must put quality management procedures into place to improve quality in spite of imperfect specification. [Sommerville2004] I-62

63 Scope of Quality Management Quality management is particularly important for large, complex systems. The quality documentation is a record of progress and supports continuity of development as the development team changes. For smaller systems, quality management needs less documentation and should focus on establishing a quality culture. [Sommerville2004] I-63

64 Quality Management and [Sommerville2004] Software Development Software development process D1 D2 D3 D4 D5 Quality management process Standards and procedures Quality plan Quality review reports I-64

65 Quality Management Activities (1) Quality assurance Establish organisational procedures and standards for quality. (2) Quality planning Select applicable procedures and standards for a particular project and modify these as required. (3) Quality control Ensure that procedures and standards are followed by the software development team. Quality management should be separate from project management to ensure independence. [Sommerville2004] I-65

66 (1) Quality Assurance Quality Assurance [ISO 9000]: All those planned and systematic actions necessary to provide adequate confidence that a product or service will satisfy requirements for quality Software quality assurance [IEEE]: 1. A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements. 2. A set of activities designed to evaluate the process by which the products are developed or manufactured. Contrast with: quality control. I-66

67 Quality Assurance Software quality assurance is [Galin2004] : A systematic, planned set of actions necessary to provide adequate confidence that the software development process or the maintenance process of a software system product conforms to established functional technical requirements as well as with the managerial requirements of keeping the schedule and operating within the budgetary confines. Quality assurance consists of the auditing and reporting functions of management [Pressman2004] I-67

68 (2) Quality Planning Quality planning is the process of assessing the requirements of the procedure and of the product and the context in which these must be observed. Quality assurance plan is the central aid for planning and checking the quality assurance. [Pressman2004] I-68

69 Quality Assurance Plan [Sommerville2004] A quality assurance plan sets out the desired product qualities and how these are assessed and defines the most significant quality attributes. The quality assurance plan should define the quality assessment process. It should set out which organisational standards should be applied and, where necessary, define new standards to be used. I-69

70 Quality Assurance Plans [Sommerville2004] Quality assurance plan structure: Product introduction Product plans Process descriptions Quality goals Risks and risk management Quality assurance plans should be short, succinct (If they are too long, no-one will read them) I-70

71 Example: SQA Plan Purpose of Plan References Management organization structure, SQA tasks, their placement in the process roles and responsibilities related to product quality Documentation project documents, models, technical documents, user documents. Standards, Practices and Conventions Reviews and Audits Test test plan and procedure Problem Reporting and Corrective action Tools, Techniques and Methodologies Code Control Media Control Supplier control Records Collection, Maintenance and Retention Training Risk Management [IEEE_Std_ , Pressman2004] I-71 22

72 (3) Quality Control Quality Control [ISO 9000]: The operational techniques and activities that are used to fulfil requirements for quality Quality Control is the series of inspections, reviews and tests used throughout the development cycle to ensure that each work product meets the requirements placed upon it. [Pressman2004] I-72

73 Quality Control [Sommerville2004] This involves checking the software development process to ensure that procedures and standards are being followed. There are two approaches to quality control Quality reviews; Automated software assessment and software measurement. I-73

74 Quality Control Objective: minimize the produced defects increase the product quality Implementation approaches: Fully automated Entirely manual Combination of automated tools and human interactions I-74

75 Quality Control Quality control includes a feedback loop to the process: provide management with the necessary data about product quality. gain the insight and confidence of product quality Two types of quality control: Quality design: the characteristics that designers specify for an item (includes: requirements, specifications, and the design of the system). Quality of conformance: the degree to which the design specification are followed. It focuses on implementation based on the design. [Pressman2004] I-75

76 Quality Assurance System Quality assurance system is the organizational structure, responsibilities, procedures, processes and resources for implementing quality management. [Pressman2004] I-76

77 Process and Product Quality The quality of a developed product is influenced by the quality of the production process. This is important in software development as some product quality attributes are hard to assess. However, there is a very complex and poorly understood relationship between software processes and product quality. [Sommerville2004] I-77

78 Process-based Quality [Sommerville2004] There is a straightforward link between process and product in manufactured goods. More complex for software because: The application of individual skills and experience is particularly important in software development; External factors such as the novelty of an application or the need for an accelerated development schedule may impair product quality. Care must be taken not to impose inappropriate process standards - these could reduce rather than improve the product quality. I-78

79 Process-based quality [Sommerville2004] I-79

80 Practical Process Quality [Sommerville2004] Define process standards such as how reviews should be conducted, configuration management, etc. Monitor the development process to ensure that standards are being followed. Report on the process to project management and software procurer. Don t use inappropriate practices simply because standards have been established. I-80

81 I.4 Components of a SQA System (1) Pre-project components (2) Software project life cycle components (3) Infrastructure components for error prevention and improvements (4) Management SQA components (5) SQA standards, system certification and assessment components (6) Organizing for SQA the human components and considerations guiding construction of organization s SQA system I-81

82 (1) Pre-project Components Pre-project Contract reviews Development and quality plans (see Chapter II and III) I-82

83 (2) Project Life Cycle Components Development Reviews Expert opinions Software testing Assurance of the quality of external participants work Maintenance Software maintenance components (see Chapter II and III) I-83

84 (3) Infrastructure Components Procedures and work instruction Templates and checklists Staff training, retraining and certification Preventive and corrective actions Configuration management Documentation control (see Chapter IV) I-84

85 (4) Management SQA Components Project progress control Software quality metrics Software quality costs (see Chapter V) I-85

86 (5) Standards, Certification, Assessment Project process standards Quality management standards Objectives: Utilization of international professional knowledge Improvement of coordination with other organizations quality systems Objective professional evaluation and measurement of the organization s SQA achievement (see Chapter VI) I-86

87 (6) Organizing for SQA Management s role in SQA The SQA unit SQA trusties SQA committees SQA forums (see Chapter I.5) I-87

88 Software Maintenance University of Paderborn The Software Quality Shrine [Galin2004] (1) Pre-project SQA components Contract review Project Development plan and Quality Plan (2) Project Life Cycle SQA components Formal Design Reviews Peer Reviews Experts Opinion Software Testing SQA of External Participants Procedures (3) Quality Infrastructure components Supporting Devices Training Instruction Preventive Actions Configuration Management Documentation Control (4) Quality Management Project Progress Control Software Quality Metrics Software Quality Costs (5) Standards Quality Management Standards Project Process Standards (6) Organizational Base Human components Management SQA Unit SQA Trustees SQA Committees SQA Forums I-88

89 Software Maintenance University of Paderborn I.5 Organizing for SQA [Galin2004] Formal Design Reviews (1) Pre-project SQA components Contract review Peer Reviews Project Development plan and Quality Plan (2) Project Life Cycle SQA components Experts Opinion Software Testing a) a) Management b) b) SQA Unit Unit c) c) SQA SQA Trustees d) d) SQA Committees e) e) SQA Forums SQA of External Participants Procedures (3) Quality Infrastructure components Supporting Devices Training Instruction Preventive Actions Configuration Management Documentation Control (4) Quality Management Project Progress Control Software Quality Metrics Software Quality Costs (5) Standards Quality Management Standards Project Process Standards (6) Organizational Base Human components Management SQA Unit SQA Trustees SQA Committees SQA Forums I-89

90 Exec. Exec. Management Exec. Executive responsible for software quality The SQA framework SQA unit Other Departments Software Testing Department Software Development and Maintenance Department SQA Committees Legend Line of authority line for SQA issues SQA Forums Flow of Forum s recommendations line [Galin2004] Software Testing Teams Software Development Teams SQA Trustees I-90

91 The SQA Framework: Participants Managers Top management executives, especially the executive in charge of SQA Software development and maintenance department managers Software testing department managers Project managers and team leaders of development and maintenance projects Leaders of software testing teams Testers Members of software testing teams SQA professionals and interested practitioners SQA trustees SQA committee members SQA forum members SQA unit team members I-91

92 a) Management Overview: Top management s quality assurance activities Software quality policy The executive in charge of software quality Management review Department management responsibilities for quality assurance processes Project management responsibilities for quality assurance I-92

93 TOP Management Responsibilities [Galin2004] Assure the quality of the Company s software products and software maintenance services. Communicate the importance of product and service quality in addition to customer satisfaction to employees. Assure full compliance with customer requirements. Ensure that SQA objectives are established and accomplished. Initiate planning and oversee implementation of changes to adapt the SQA system to changes related to the organization's clientele, competition and technology. Intervene directly to resolve of crisis situations and minimize damages. Ensure availability of resources required by SQA systems. I-93

94 SQ Policy Requirements [Galin2004] Quality policy refers to the basic aims and objectives of an organization regarding quality as stipulated by the management. [Pressman2004] Conformity to the organization purpose and goals Commitment to: General software quality assurance concepts The quality standards adopted by the organization Allocate adequate resources for software quality assurance Continuous improvement of the organizations quality and productivity I-94

95 Responsibilities (Executive in Charge) Responsibility for preparation of an annual SQA activities program and budget Responsibility for preparation of SQA system development plans Overall control of implementation of the annual SQA regular activities program and planned SQA development projects Presentation and advocacy of SQA issues to executive management [Galin2004] I-95

96 Management Reviews Def.: Management review is the name given to the periodic meeting convened to allow executives to obtain an overview of their organization s software quality issues. Typical items: Periodic performance reports, including quality metrics Customer satisfaction feedback Follow up reports for SQA annual regular activity program and SQA development projects Summary of special quality events related to customers, suppliers, subcontractors, etc. Review of significant findings of internal and external quality audits as well as special surveys Identification of new software quality risks and unsolved pre-existing risks Recommendations for software quality management improvements. [Galin2004] I-96

97 Management Reviews: Objectives Assess achievement of quality objectives set for the organization s software quality management system Initiate updates and improvements of the software quality management system and its objectives Outline directions for remedying major SQA deficiencies and software quality management problems. Allocate additional resources to the software quality management system. [Galin2004] I-97

98 Department Responsibilities (1/2) The quality system-related responsibilities: Preparation of the department s annual SQA activities program and budget, based on recommended SQA unit program. Preparation of the department s SQA systems development plans, based on recommended SQA unit plan. Control of performance of the department s annual SQA activities program and development projects Presentation of the department's SQA issues to the executive in charge of software quality. [Galin2004] I-98

99 Department Responsibilities (2/2) Project-related responsibilities Control of compliance to quality assurance procedures in the department's units Detailed follow up of contract review results and proposal approvals Review of unit performance of planned review activities; approval of project documents and project phase completion Follow up of software tests; approval of project s software products. Follow up of progress of software development project schedules and budget deviations. Advise and support project mangers in resolving difficulties. Follow up of quality of maintenance services Detailed follow up of project risks and their solutions Follow up of project's compliance with customer requirements and customers satisfaction. Approval of large software change orders and significant deviations from project specifications. [Galin2004] I-99

100 Project Management Responsibilities Professional hands-on tasks: Preparation of project and quality plans and their updates. Participation in joint customer-supplier committee Close follow up of project team staffing, including recruitment, training and instruction. Management tasks The follow up issues: Performance of review activities and the consequent corrections, including participating in some reviews. Software development and maintenance units performance with respect to development, integration and system test activities, corrections and regression tests and acceptance tests Software installation in customer sites and the running-in of the software system by the customer SQA training and instruction of project team members Schedules and resources allocated to project activities. Customer requests and satisfaction Evolving project development risks, application of solutions and control of results. [Galin2004] I-100

101 b) The SQA Unit Overview: Activities Responsibilities Tasks performed by the head of the SQA unit SQA sub-unit tasks related to the project life cycle SQA sub-unit infrastructure operations tasks SQA sub-unit audit and certification tasks SQA sub-unit support tasks SQA sub-unit standards and procedures: Development and maintenance tasks SQA sub-unit information system tasks I-101

102 SQA Unit Tasks [Pressman2004] Quality assurance planning oversight, record keeping, analysis and reporting Participates in the development of the projects software process Reviews software engineering activities to verify compliance with the defined software process. Audits designated software work products to verify compliance with those defined as part of the software process. Ensures that deviations in software work and work products are documented and handled according to a document procedure. Records any noncompliance and reports to senior management. I

103 Unit: Organizational Structure [Galin2004] Head SQA Unit SQA Operations SQA Development and Maintenance Project Life Cycle SQA Internal and Certification SQA Audits SQA Standards and Procedures SQA Information Systems SQA Infrastructure Operations SQA Support SQA Engineering I-103

104 SQA Unit Head Tasks (1/2) Planning tasks Preparation of proposals for the Unit s annual activity program and budget Planning and updating the organization s software quality management system and recommended annual SQA activities programs for the software development and maintenance departments. Preparation of recommended SQA systems development plans for the software development and maintenance departments. Management tasks Management of SQA team's activities Monitoring implementation of the SQA activity program Nomination of team members, SQA committee members and SQA trustees Preparation of special and periodic status and performance reports. [Galin2004] I-104

105 SQA Unit Head Tasks (2/2) Contacts with customers and other external bodies and the executive in charge of software quality Serving as the customer s address for software quality issues of software products and services supplied Representation of the organization before external bodies regarding software quality issues Drafting the management review reports Raising SQA organizational issues and preparing requested material for top management's consideration SQA professional activities Participation in project joint committees Participation in formal design reviews Review and approval of deviations from specifications Consultation to project managers and team leaders Participation in SQA committees and forums [Galin2004] I-105

106 Life Cycle Tasks (Sub-Units) Project life cycle control tasks Follow up of development and maintenance teams compliance with SQA procedures and work instructions Approval or recommendation of software products (design reports and code). Monitoring delivery of software maintenance services to internal and external customers Monitoring customer satisfaction (surveys, etc.) and maintaining contact with customer s SQA representatives Participation tasks participation in: Contract reviews Preparation and updating of project development and project quality plans Formal design reviews Subcontractors formal design reviews Software testing, including customer acceptance tests Software acceptance tests of subcontractors software products Installation of new software products [Galin2004] I-106

107 Infrastructure Tasks (Sub-Units) Publication of updated versions of procedures, work instructions, templates, checklists, etc., with their circulation. Training and instruction to new and current staff and SQA trustees regarding SQA procedures, work instructions, new and revised procedures, development tools and methods, etc. Monitoring and supporting implementation of new and revised SQA procedures Follow up of staff certification activities Proposal of subjects requiring preventive and corrective actions Follow up of configuration management activities Follow up of compliance with documentation procedures and work instructions [Galin2004] I-107

108 Types of Audits (in or by SW Org) Internal audits Audits of subcontractors and suppliers to evaluate their SQA systems External audits performed by certification bodies External audits performed by customers who wish to evaluate the SQA system prior to accepting the organization as a supplier [Galin2004] I-108

109 Audits and Certifications (Sub-Units) Preparation of annual programs for SQA audits Performance of SQA audits Follow up of corrections Preparation of periodic summary reports Collection of data on the performance of the audited organization from internal and external sources Periodic evaluation of the audited organization Coordination of the external audit's contents and schedule Preparation of documents as specified by external auditors Instruction of the audited teams and performance of preparations for external audits Participation in the audit [Galin2004] I-109

110 Support Tasks (Sub-Units) Preparation of project development plans and project quality plans Staffing review teams Choice of development methodologies and tools that reflect the accumulated failure experience Choice of measures to solve identified software development risks Choice of measures to solve schedule delays and budget overruns Choice of SQA metrics and software costs components Use of SQA information systems [Galin2004] I-110

111 Standard and Procedures (Sub-Units) Prepare an annual program for development of new procedures and procedure updates Responsibility for development of new procedures and procedure updates, including participation in appropriate committees and forums Follow up of developments and changes in SQA and software engineering standards; introduction of additional relevant procedures and changes Initiation of updates and adaptations of procedures in response to changes in professional standards, including adoption or deletion of standards applied by the organization. [Galin2004] I-111

112 Engineering (Sub-Units) Testing quality and productivity aspects with respect to new development tools and new versions of currently used development tools Evaluation of quality and productivity of new and improved development and maintenance methods Development of solutions to difficulties confronted in application of currently used software development tools and methods Development of methods for measuring software quality and team productivity Provision of technological support to CAB committees during analysis of failures and formulation of solutions [Galin2004] I-112

113 Information Systems (Sub-Units) Development of SQA information systems for software development and maintenance units for: Collection of activity data. Processing of information delivered by the units: periodic reports, lists, exception reports, queries and estimates of software quality metrics and software quality costs. Updating of SQA information systems Development and maintenance of the organization's SQA Intranet/Internet site [Galin2004] I-113

114 SQA Unit Plan [Pressman2004] Evaluations to be performed Audits and reviews to be performed Standards that are applicable to the project Procedures for error reporting and tracking Documents to be produced by the SQA group Amount of feedback provided to software project team I-114

115 c) SQA Trustees Unit-related tasks: Support their colleagues' attempts to solve difficulties in the implementation of SQA procedures and work instructions Help their unit manager in performing his or her SQA tasks Promote compliance and monitor implementation of SQA procedures and work instructions by colleagues Report substantial and systematic non-compliance events to the SQA unit Report severe software quality failures to the SQA unit Organization-related tasks Initiate changes and updates of organization-wide SQA procedures and work instructions Initiate organization-wide improvements of development and maintenance processes and applications for solutions to recurrent failures observed in their units Identify organization-wide SQA training needs and propose an appropriate training or instruction program [Galin2004] I-115

116 d) SQA Committees Permanent committees commonly deal with: SCC (software change control), CA (corrective actions), Procedures, Development of method, tools and quality metrics. Ad-hoc committees commonly deal with specific cases: Updates of a specific procedure, Analysis and solution of a software failure, Elaboration of software metrics for a targeted process or product, Updating software quality costs, Data collection methods for a specific issue. [Galin2004] I-116

117 e) SQA Forums SQA forums typically focus on: SQA procedures improvements and implementation Quality metrics Corrective actions analysis of failure and success cases Quality system issues development and implementation of new tools Quality line management problems daily operational software quality problems Members of an open forum may include: SQA unit members SQA trustees Software development and maintenance staff SQA and software engineering consultants/experts Customer representatives [Galin2004] I-117

118 Other Relevant Structures [Pressman2004] Requirements Control Board All requirement changes must be formally reviewed and approved Software Control Board All design changes must be formally reviewed and approved Interface Control Board I-118

119 I.6 Discussion & Summary General quality definitions of quality are not sufficient in practice. Thus, software quality is described by specific quality models which determine the causal relationship from intangible quality views to tangible software measures [ISO/IEC 9126] We cannot wait for specifications to improve before paying attention to quality management. We must put quality management procedures into place to improve quality in spite of imperfect specification. Quality management activities consist of quality assurance, quality planning, and quality control. I-119

120 Discussion & Summary The quality of a developed product is influenced by the quality of the production process. This is important in software development as some product quality attributes are hard to assess. However, there is a very complex and poorly understood relationship between software processes and product quality. The main instrument for quality is the software quality assurance system. Components of a software quality assurance system for the pre-project phase, each life cycle phase, management, infrastructure, standardization, and organization exist. Organizing for SQA involves mainly the management, SQA Unit, SQA Trustees, SQA Committees, and SQA Forums I-120

121 I.7 Bibliography (1/4) [Basili&Rombach1988] V.R. Basili, H. D. Rombach, "The TAME Project: Towards Improvement-Oriented Software Environments," IEEE Transactions on Software Engineering, vol.se-14, no.6, June 1988, pp [Glass1992] R.~L. Glass, Building Quality Software. Englewood Cliffs, NJ, USA: Prentice Hall, [Galin2004] D. Galin, Software Quality Assurance: From theory to implementation. Harlow, England: Pearson Addison Wesley, [Horch1996] J.~W. Horch, Practical guide to software quality management. Boston, USA: Artech House, first ed., [Horch2003] J.~W. Horch, Practical guide to software quality management. Boston, USA: Artech House, second ed., I-121

122 I.7 Bibliography (2/4) [ISO9000] ISO 9000:2000, Quality management systems Fundamentals and vocabulary [ISO9001] ISO 9001:2000, Quality management systems Requirements [ISO 9004] ISO 9004:2000, Quality management systems Guidelines for performance improvements [ISO 9000addon] "ISO 9000 Introduction and Support Package" guidance documents from ISO/TC 176 SC2: N524 - Guidance on ISO 9001:2000 Sub-clause 1.2 'Application' N525 - Guidance on the Documentation Requirements of ISO 9001:2000 N526 - Guide to the Terminology used in ISO 9001:2000 and ISO 9004:2000 N544 - Guidance on the Concept and Use of the Process Approach for management systems N630 - Guidance on Outsourced Processes I-122

123 Bibliography (3/4) [ISO ] ISO Handbook:2002, ISO 9001:2000 for small businesses What to do, Advice from ISO/TC 176 [ISO/IEC 9126] Information technology - Software Product Evaluation - Quality characteristics and guidelines for their use ISO/IEC :2001: Software engineering -- Product quality -- Part 1: Quality model ISO/IEC TR :2003: Software engineering -- Product quality -- Part 2: External metrics ISO/IEC TR :2003: Software engineering -- Product quality -- Part 3: Internal metrics ISO/IEC TR :2004: Software engineering -- Product quality -- Part 4: Quality in use metrics [IEEE_Std_ ] Standards Coordinating Committee of the IEEE Computer Society, The Institute of Electrical and Electronics Engineers, Inc. 345 East 47th Street, New York, NY , USA. IEEE Std , IEEE Standard Glossary of Software Engineering Terminology. (Revision and redesignation of IEEE Std ). [IEEE_Std_ ] Standards Coordinating Committee of the IEEE Computer Society, The Institute of Electrical and Electronics Engineers, Inc. 345 East 47th Street, New York, NY , USA. IEEE Std , IEEE Standard fro Software Quality Assurance Plans. I-123

124 Bibliography (4/4) [Lyu1996] Michael R. Lyu, editor. Handbook of software reliability engineering. IEEE Computer Society Press, Los Alamitos, Calif., [McCall +1977] J.A. McCall, P.K. Richards, and G.F. Walters, Factors in Software Quality, Vol. 1, AD/A /015/055, Nat'l Tech. Information Service, Springfield, Va., [McConnell 1996] Steve McConnell. Software Quality at Top Speed. Software Development. August 1996 [Pressman2004] Roger Pressman. Software Engineering: A Practitioner's Approach. McGraw-Hill, 6 ed., [Sommerville2004] Ian Sommerville. Software Engineering. Addison Wesley. 7 ed., [Schulmeyer1992] G. G. Schulmeyer, ed., Handbook of software quality assurance. Van Nostrand Reinhold, I-124