Improving the quality of ISO 9001 audits in the field of software

Transcription

1 Information and Software Technology 40 (1998) Improving the quality of ISO 9001 audits in the field of software A.J. Walker* Software Engineering Applications Laboratory, Electrical Engineering, University of the Witwatersrand, Johannesburg, South Africa Abstract Internal quality system audits are a compliance requirement of ISO 900\2. Requirements of the internal quality audit clause define quality audit outputs in terms of audit planning and scheduling, recording of results and follow-up of audit activities. ISO provides general guidance on the conduct of audits. Where domain specific guidance is required on the audit, little support is available from ISO and national standards bodies. In particular, there is no freely available checklist support questions which probe the compliance requirements of ISO 9001\2, or provide industry specific guidance questions for probing the effectiveness of the implementation of the ISO 9001\2 process clauses. This paper reviews a national project to develop a checklist to probe ISO 9001 requirements for the field of software, and to offer guidance for examining the effectiveness of the implementation of the process clauses in the software domain. It is believed that this project is important for two reasons: the ISO 9001 compliance questions in the checklist are generically applicable, and secondly, the structure of the checklist has been devised to be tailorable to a wide range of application domains Elsevier Science B.V. All rights reserved. Keywords: Assessments; Audits; Checklists; ISO 9001: Quality management; Software engineering standards 1. Introduction There is little in common in the approach to auditing of a company against the compliance requirements of ISO 9001 [1] other than that which may be traced to ISO [2] which is the basis of international schemes for quality system auditor training and certification. Specific arrangements for auditor training and certification are available for the field of software, where the most well-known arrangement is the TickIT Scheme. It is noteworthy that even this well-known scheme is little in the way of specific guidance for compliance testing of the ISO 9001 system clauses and guidance on the implementation of the process clauses of ISO 9001 outside of ISO [3]. A commonly used tool in quality management is a checklist, which may contain a set of criteria against which the efficacy of a process is judged. Quality system auditors and certification bodies frequently assemble their own checklists as handy means for testing compliance, but there are no national or international commonly accepted checklists for testing or probing compliance to the shall requirements of ISO 9001 or which probes the implementation of the process clauses against international good software practices. Furthermore, where such checklists are know to be used by certification * Tel.: ; fax: ; walker@ odie.ee.wits.ac.za bodies, they do not make it a practice of releasing these checklists to their clients. This lack of uniformity in the framework for compliance testing crates a serious problem for companies which seek registration to ISO 9001 since there is a large gap between the ISO 9001 clauses and their interpretation for the field of software. So much so that this creates considerable problems for those who create and maintain software quality management systems against the ISO 9001 standard and those who interpret those requirements for compliance purposes (quality systems auditors). Against this background a project was initiated in November 1994 by the Software Engineering Applications Laboratory (SEAL) to develop a national standard (or, more accurately, recommended practice) under the auspices of the South African Bureau of Standards Technical Committee for Information Technology (TC71.1) for this purpose. At the May 1995 meeting of SABS National Committee for Information Technology (TC71.1) this project was tabled as a New Work Item to support more effective auditing of software quality management systems. The goal of the standard was to provide a common framework for assessments and follow-up audits for companies engaged in the development of software being developed for external or internal customers. Technical development was completed in November 1996, and formally published in August 1997 [12] /98/$ - see front matter 1998 Elsevier Science B.V. All rights reserved. PII: S (98)

2 866 A.J. Walker / Information and Software Technology 40 (1998) Fig. 1. A checklist sample from ISO 9001 clause 4.5. It was believed that such a tool would go a long way to reducing the all too frequent adversarial relationship between the quality assurance manager on the one part, the software developers on another part, and the external quality system auditors on the third part. Although there is a considerable need in this area, an investigation revealed that there were no suitable checklists available for this purpose, either locally or internationally. Current international practice is, however, moving towards encouraging the use of checklists for conducting assessments\audits of ISO 9000 quality management systems largely to harmonise audit and assessment practices. This checklist will be useful to: internal quality system auditors and quality managers; to individuals or bodies who conduct 2nd or 3rd party surveillance assessments/audits of quality management systems in the software industry. This paper presents an overview of this project in terms of: project requirements, process and product quality objectives and quality management practices; participation by industry locally and internationally; the product development and review process; the trialling (or validation) of the product; how this product could be used with good effect in other application domains. 2. Development of the checklist 2.1. Product requirements Functional and non-functional product requirements were identified. Functional requirements are tangible and can be measured, while non-functional requirements can be viewed as desirable goals or objectives for the project nice to have but impossible to measure Product functional requirements The following functional requirements were identified. 1. The requirements of ISO 9001 (as indicated by the shall in each clause) will be probed by a searching question (or series of questions) to test the extent of compliance of the system under review. 2. The checklist will address the domain of software. 3. The development of the checklist questions will be strongly guided by current and emerging international good practices, and will seek to use compliance indicators being used for this purpose elsewhere.

3 A.J. Walker / Information and Software Technology 40 (1998) The checklist shall be easily available in both hardcopy and electronic format. 5. The layout of the checklist will be guided by applicable SABS Recommended Practices, i.e. ARP 013: Drafting and Presentation of Standards [4]. 6. The header of the checklist table will support the conduct of the audit\assessment, and make provision for recording: Client Date Reference number for the assessment/audit Person(s) interviewed (Shown as a table, with the headings of name, initials, function) Auditor(s)\assessor(s) The columns of the checklist table will support the conduct of the audit\assessment by making provision for: auditor initials, assessee\auditee initials, checklist\- compliance result: (Categories A and B: C compliance, N non-compliance); (Categories C F: P present, A absent); X not applicable, implementation/observation/comments, implementation result, expressed in the (Categories A and B: C compliance, N non-compliance); (Categories C F: P present, A absent); X not applicable, reference to findings report. Note: These categories are provided to differentiate between what will be audited for compliance (A,B), and what may be investigated in terms of good practice (C F). (A sample fragment of the implemented checklist is shown in Fig. 1) Product non-functional requirements The following non-functional requirements were identified. 1. The use of the checklist will serve to: enhance customer confidence in the client quality management system improve the effectiveness and efficiency of audits\ assessments improve the objectivity of the assessment\audit. 2. That international recognition of the product will be promoted Project and product quality requirements Besides identifying technical requirements for the product, quality objectives were identified for the process applied to the development of the checklist and extent to which the product met the technical requirements project quality objectives The quality requirements for the process applied to developing the checklist were defined as the following. 1. To manage this product development in compliance with ISO 9001 requirements. 2. The Committee Draft stage will be used to apply the checklist in practical assessment/auditing situations to determine the utility of the questions and to elicit feedback for validation purposes Product quality objectives On the other hand, the quality objectives for the questions comprising the checklist were required to demonstrate the following [5]. 1. Objectivity: a question is objective if it is possible to provide the answer without the opinion of the checklist user. 2. Completeness: a question is complete if all the components needed to specify its meaning are present. 3. Repeatability: a question is repeatable if applied several times by the same checklist user always produces the same answer. 4. Reproducibility: a question is reproducible if applied by different users always produces the same answer. 5. Usefulness: a question is useful if its answer contributes to the evaluation process. 6. Measurability: a question is measurable if it is possible to determine the attributes and their measures. 7. Specific: a question is measurable if it is possible to determine the attributes and their measures. 8. All the shall requirements of ISO 9001 are addressed in the checklist. The extent to which these quality characteristics are exhibited by the checklist questions may be evaluated by field trialling and by the application of formal inspection and review techniques Local and international collaboration The project drew upon the following resources. 1. Core Group Members: (13) Individuals who are software quality system managers in local companies. 2. Extended Core Group Members: These include colleagues overseas experienced in software quality management (9) and a number of individuals locally who have shown interest in trialling the product in their companies (3 at the present time). 3. Organisational Representatives of the SABS TC71.1 Information Technology Committee: This group of individuals is responsible for approving the developed product Project communication Project communication depends heavily upon the use of the Internet for communication and document distribution.

4 868 A.J. Walker / Information and Software Technology 40 (1998) Table 1 Number of questions in each category for each ISO 9001 clause ISO 9001 Checklist category Clause A B C D E F Totals Total number of questions 718 The SEAL File Server is the repository of the project documents and records. All core group members have username and password access to the management products, technical products and records supported on the SEAL file server. The emerging checklist technical products are publicly available and are accessible using anonymous FTP access to the SEAL File Server thereby providing access to the checklist products for trialling and validation. (See Appendix 1) Product development management Developing national or international standards is an unavoidable resource intensive and time-consuming process. The goal is to achieve consensus amongst the various stakeholders on the technical attributes of a new product which may exert a considerable impact on prevailing practices, particularly if the standard affect contractual arrangements or legislation. In view of this management authorisation operates on a number of different levels and is governed by product progress through the Working Draft, Committee Draft and Draft SA Standard stages, which are formally governed by SABS Recommended Practice ARP 017 [6]. These steps are largely similar to those used for the development of ISO\IEC\JTC1 standards. The SEAL has a certified ISO 9001 quality management system and these requirements are applied to all documents and records emanating from this project Product trialling and validation The development path taken by this project is unusual in the emphasis placed on active testing of the product at each stage of the standardisation process. Feedback has been facilitated by providing a review form with the checklist product set. The net result of this review process is: 1. to meet the review requirements of the SABS and to move the product from Committee Draft to Draft National Standard stage; 2. to enhance confidence in the widespread of the checklist. 3. Technical features of the checklist The density of questions for each ISO 9001 clause and category is shown in Table 1 for Revision 0.30 of the checklist. An examination of the number of shalls in ISO 9001 shows a count of 139 instances. In many case the compliance requirement has multiple aspects, i.e. one shall may have number of sub-pars each of which represent a distinct compliance requirement. Not surprising, once a full count reveals 236 distinct shall requirements. An examination of Category B shows a total of 48 compliance requirements but it must be noted that these may overlap with the category A requirements and are not necessarily distinctive. A cursory overview of the Table 1 clearly indicates the areas demanding the heaviest attention, notably clause 4.4 (Design Control), 4.10 (Inspection and Test), and 4.9 (Process Control). Software guidance questions have drawn from: 1. ISO [7]: this standard is currently being revised by TC 176\SC2\WG17 to bring the document in line with ISO 9001 (1994) requirements. The current version of the International Standard (i.e. ISO (1991)) is compliant to ISO 9001 (1987). In view of this the DIS version of this standard was used as the normative reference. A secondary, but significant reason took account of the fact that the order of the clauses is ISO (1996) now follows the clause order in ISO The software guidance drawn from this document largely concerns the implementation of the product process clauses i.e. 4.3, 4.4, 4.7, 4.8, 4.9, 4.10, 4.11, 4.12, 4.13, 4.15 and ISO [8]: this draft international standard has been used as the normative reference on issues related to project management, which has provided some helpful guidance in interpreting the requirements of 4.4 (Design Control). 3. ISO\IEC Software Life-Cycle Processes [9]: this keystone international software engineering standard has provided valuable supplementary software guidance in the same categories as (1). 4. ISO 9127 [10]: this International Standard has provided

5 A.J. Walker / Information and Software Technology 40 (1998) useful guidance specifically on implementation requirements of 4.15 (handling, storage, packaging, preservation and delivery) of software products. 5. ISO\IEC 9126 [11]: this International Standard is used to probe ISO 9001 clause dealing with Design Input. 6. European Auditor s Guide (Rev 3.0) Sept 1995 [3]: this product has been used as an informative reference to the project, but it was found that guidance offered in this guide was almost invariably better supported in the standards listed above. The checklist does not attempt to be a cross reference between ISO 9001 and the Standards and support documents ISO 9001 and the Standards and support documents listed in (1) (6). Rather the approach has been to adopt a minimalist approach in developing the checklist. Questions offering software guidance are only supplied for the ISO 9001 category A and B questions are insufficiently specific to provide clear and unambiguous application in the field of software. Neither is the issue of providing questions forced artificially. In the case of most of the system clauses of ISO 9001 (i.e. 4.1, 4.2, 4.6, 4.14, ) no software related guidance is required. 4. Discussion External auditing of a quality system to determine compliance unavoidably contains elements which give rise to an adversarial relationship between auditor and auditee. Many of the negative aspects of this relationship can be avoided by establishing the framework of that relationship, managerially and technically in advance. Surprising, the quality profession has been slow to apply its own tools to managing a key dimension of its activity notably the testing of compliance to defined requirements. It is believed that the checklist described in this paper can be productively used to manage the auditor\auditee relationship by making the ground rules of scope and extent of compliance well-known in advance. The checklist may be readily applied to other areas by removing the sections with the existing software guidance and substituting guidance for each ISO 901 sub-clause as appropriate for the intended application domain. Since official publication in August 1997, local demand for the checklist has been strong several hundred copies have been sold locally. On the international front, formal steps have been taken to achieve recognition for the product as a recommended practice for quality audits in software engineering. The product is presently under review by the Software Engineering Standards Committee (ISO/IEC JTC1/SC7). Acknowledgements The author gladly acknowledges the strong support from the core and extended group members comprising the checklist development team, and in particular the project leader of SABS TC71.1 (Mr Wotjec Skowronski) for his constant support and encouragement in the development of this national standard. Appendix A. Information about the checklist Information about the checklist is available at URL: seal.ee.wits.ac.za/1995_47/ac1910.htm. This suite of pages provides background information about the project, the project team members, access to the checklist, and a feedback form. References [1] ISO 9001 Quality Systems Model for Quality Assurance in Design/ Development, Production, Installation and Servicing, International Organisation for Standardisation, [2] ISO : Guidelines for auditing quality systems. Part 1: Auditing, Internation Organisation for Standardisation, [3] The TickIT Guide, Appendix 1 European IT Quality System Auditor Guide, Issue 3.0, [4] SABS ARP 013: 1990 Drafting and presentation of standards, South African Bureau of Standards. [5] F. Fabbrini, N. Format, M. Fusani, S. Gnesi, Evaluating evaluation instruments, in: Proceedings Software Quality Management 95, South African Society for Quality and Software Engineering Applications Laboratory, Software Engineering Applications Laboratory, University of the Witwatersrand, Johannesburg, South Africa, [6] Procedures for the Technical Work in the preparation of South African Standards, ARP 017, 1993, GR 14. [7] ISO 9000 (DIS) Quality Management and Quality Assurance Standards, Part 3 Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software, [8] ISO (DIS) Quality Management Guidelines to quality in project management, Internation Organisation for Standardisation, [9] ISO\IEC Software Life-Cycle Processes, Internation Organisation for Standardisation, [10] ISO 9127 Information Processing Systems User Documentation and Cover Information for Consumer Software Packages, Internation Organisation for Standardisation, [11] ISO\IEC 9126 Information Technology Software Product Evaluation Quality Characteristics and Guidelines for their Use, Internation Organisation for Standardisation, [12] SABS ARP 042: 1997 ISO 9001 Audit Checklist for Software, 1st ed., South African Bureau of Standards, ISBN , Alastair Walker is presently an associate professor in the Department of Electrical Engineering, University of the Witwatersrand. He was responsible for establishing the Software Engineering Applications Laboratory in The SEAL received an ISO 9001 certification for software development in July He is a certified quality analyst and a certified software quality systems auditor.