How To Test Software For Safety

Transcription

1 Software Supplier Assessment Plan The life cycle for software parallels that for a computer system. It begins during the analysis of the requirements... by Robert W. Stotz, Ph.D. Manager of Validation Jacobs Engineering Group and Alan R. Bluhm Independent Consultant I t has long been recognized that two of the most critical steps in the development life cycle of a computer system are the definition of the system's functional requirements and the development of high quality software. Regarding quality, FDA stated in the Federal Register (Volume 61, Number 87, May, 1996) covering proposed changes to 21 CFR Parts 210 and 211 that "The GMP regulations are based on fundamental concepts of quality assurance: (1) Quality, safety, and effectiveness must be designed and built into a product; (2) quality cannot be inspected or tested into a finished product; and () each step of the manufacturing process must be controlled to maximize the likelihood that the finished product will be acceptable." These basic quality principles are directly applicable to the development and validation of computer systems, and associated software for GXP (Le., GLP, GMP, and GCP) applications in the pharmaceutical and health care industry. Background The life cycle for software parallels that for a computer system. It begins during the analysis of the requirements for the system and proceeds throughout all stages of the system development life cycle until the system is retired. The quality of a computer system must be built in during creation of the software. To assure that important aspects of quality are met, it is important for the software user to obtain assurance that the supplier has followed the appropriate software development and configuration management practices. The assessment should establish the following: Written development standards and procedures are being followed; Test plans and test methods adequately demonstrate that the software meets requirements; Good configuration management and documentation practices are utilized. Assessments of potential software suppliers should be made by the end-user prior to selection of the supplier. Once a supplier is selected, additional performance reviews should be conducted throughout the course of the project. Since the elements of an assessment and the requirements for documentation of the development process are heav- C o n d u c t i n g Au d i t s, G A P A s s e s s m e n t s, & C o r r e c t i ve A c t i o n s 221

2 Robert W. Stotz, Ph.D. & Alan R. Bluhm ily dependent on the type of software being developed, it would be appropriate at this point to define those classifications. Software can be classified into system, configurable and application-specific software. These classifications are not considered to be rigid or mutually exclusive, but they allow important distinctions to be made about vendorsupplied software and software testing. Operating system software provides basic operating instructions for the computer. Examples of system software include DOS and UX. Since operating system software (which is supplied with the computer) has a very large user base and is independent of the specific application, it does not require formal validation. Configurable software is a program which allows the user to define a number of conditions such as operating parameters, reporting parameters and alarm conditions. The user does not change the configurable program code itself, but can change the parameters to satisfy the requirements of a specific application. Lotus 1-2- of a computerized system, it requires the greatest degree of validation documentation. The documentation for both application-specific and configurable software are often scrutinized by the FDA during an inspection, since both have the potential for adversely affecting the operation of the computerrelated system. In the case of vendor-supplied software (both configurable and application-specific software), the end-user assumes the ultimate responsibility for the use of the software. Thus, the user must rely heavily on vendor-supplied information and test data to assess the quality of their software. As part of an assessment, the user should carefully scrutinize the development and testing practices of the software developer to ensure that the software developer meets the appropriate standards. The quality assurance discipline of the end-user firm (or their representative) should be familiar enough with software development and testing practices to effectively participate in the performance of the assessment. As part of the assessment, the end-user should assess the software programmer's application of software quality assurance standards, guidelines, and standard operating procedures relative to software development, verification, validation, and maintenance. Detailed studies of this documentation, as well as test cases, test reports, software anomaly and/or bug fix reports and change reports establish the integrity and worth of the software developer's product. Related factors to consider as part of an assessment are the software supplier's use of written quality assurance, testing and configuration management procedures, the experience level of their programmers, and the quality and depth of their documentation. For configurable software, the user base for the specific version of the configurable program being used, length of vendor support for prior versions, and their financial stability are also important considerations. An important part of the software development life cycle for both configurable and application-specific software, is how it is tested and documented. Thorough documentation is imperative throughout all stages of software development, including: requirements, design, programming, and testing. While demonstrating compliance with requirements and development standards, testing also shows Application-specific software is code which is written to the specific requirements of the end-user. in their application. and most Distributed Control System (DCS) software are examples of configurable software. Validation of configurable software is heavily dependent on assessing a supplier's practices, and the quality of documentation generated during software development and testing. Application-specific software is code which is written to the specific requirements of the end-user in their application. When a configurable program is used, the parameters and operations which are specified via the configurable software comprise the application-specific program. Examples would include designing a spread sheet in Lotus 1-2- to calculate yield of a particular mixing operation or setting up a Microsoft ACCESS database to track product complaints. (Note: If the program code of configurable software is altered for a specific enduser's application, the entire code is considered to be application-specific software.) Since applicationspecific software directly controls the functionality 222 I n s t i t u t e o f Va l i d a t i o n Te c h n o l o g y

3 Robert W. Stotz, Ph.D. & Alan R. Bluhm functionality, compatibility, and performance. Most important, testing identifies non-conformance with end-user requirements and accepted software development practices. Software testing is classified as structural or functional testing. Experience has shown that structural testing, and the detailed examination of the functional logic of code, may be the most important step in building quality into software. Structural testing includes peer review of the code to determine if it meets user requirements/specifications and structurally adheres to established standards/procedures. Structural testing also includes analysis of all logic paths, and inspection for "dead" code. Dead code is executable code that is extraneous to the normal operation of the system. On board diagnostics, utilities, configurable code, in-line code comments or other compiled code that is not accessed on a regular basis is not considered dead code. A listing of the source code (the human readable form of the code) must be available to perform structural testing. In its 1986 concept paper, PhRMA's (formerly PMA's) Computer Systems Validation Committee (CSVC) advised that source code for application-specific software should generally be available to the end-user. (In the same concept paper the CSVC also emphasized the importance of having written procedures that are followed when preparing or altering any software.) FDA Policy Guide 712a.1 published in May, 1987, emphasized three key points regarding source code: User-firms should maintain source code for vendor-supplied application programs. User-firms should review and approve such source codes. Dead code should be removed on the basis of such reviews. Documentation of the structural testing of configurable and application-specific software is the software developer's responsibility, but the responsibility for determining the adequacy of this structural testing documentation lies with the end-user. Therefore, a decision to purchase supplier software should be dependent upon their ability to provide appropriate documentation of the structural integrity of their software. A supplier assessment should be performed if there are any concerns about the adequacy of their documentation, or whether or not the supplier developed the software using software development standards and/or procedures. Functional testing of software evaluates the inputs or range of inputs compared to the expected output values. Functional testing does not require source code, but the source code can be helpful in designing tests to be performed. Functional testing does, however, require a comprehensive system specification that describes all functions of the platform in sufficient detail to define the tests required to assure proper performance. The supplier of configurable software may provide documented verification of functional testing of their software, but the end-user must also perform sufficient functional testing as part of operational qualification to verify the proper operation of the configurable software for their unique application and within their specific environment. Functional testing of application-specific software is generally the end-user's total responsibility. End-users who rely on vendors to perform functional testing of application-specific software need to obtain assurance of the vendor's education, training, computer validation experience and the adequacy of the testing documentation. Assessment Matrix An assessment matrix was developed by the authors to facilitate the performance and documentation of a supplier assessment, and to maintain consistency amongst the assessments performed. It is important to note that the matrix has aspects of both an assessment and an audit. An audit requires a pre-approved audit plan. An audit has a narrower, defined scope, and examines documentation in terms of compliance with traceable directives such as applicable regulatory requirements, standard operating procedures, accepted standards, manufacturing procedures, project and quality assurance plans, etc. The audit report includes observations for documents, and conclusions relative to the software developer's performance and compliance with appropriate directives. An assessment on the other hand is informal, has a much broader scope, and examines documentation in terms of current industry practice and regulatory agency expectations. An assessment report is subsequently generated that offers an opinion based on experience in the industry as to the level of compliance of reviewed documentation, and offers suggestions for improvement. Many of the items for review listed in the matrix C o n d u c t i n g Au d i t s, G A P A s s e s s m e n t s, & C o r r e c t i ve A c t i o n s 22

4 Robert W. Stotz, Ph.D. & Alan R. Bluhm are directly traceable, and therefore referenced, to ISO 9000, viz., ISO 9001, Quality Systems - Model for Quality Assurance in Design/Development, Production, Installation and Servicing, and ISO 9000-, Part, Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software. FDA made the following statement regarding the ISO 9000 series of standards in the May, 1996 Federal Register cited above: "Other organizations have developed standards to define quajityin the. manufeicturing process.one such organization is the International Organization of Standardization (ISO). Thepurpose of the ISO Standards is tof:jrovide generic guidance on.qucility in manufacturing processes to both. industry.and vef)dors supplying industry. Five stfjndards (9000-9()04) have been developed by thels0.counci/and are '. intendecl to be acceptedworldwide..these stan: danjs. areapplica61eto any industry:and are not specific othe. pharmeiceutical industi}t.compliance with thestandarrjs is vojuntary.the principles andpractices elucidatedin the ISO standards are.not ifl conflict:w;ththose provided : by the cgmp'regulations. Indef!Jd, voluntary ISO standards share corj1mon 'principles with FDA's' CGMp requ{rements." A key word, used twice, in the above statement is 'voluntary'. It is because of the voluntary nature of the ISO Standards that they are not, and never The rating/scoring system used in the performance of the assessment is as follows: a rating of Satisfactory (S) is equal to a score of, Needs Improvement () is equal to a score of, and Needs Substantial Improvement (NSI) is a zero score. The highest total score attainable is 20. No attempt has been made to prioritize the importance of the items listed in the matrix since their importance is dependent on specific project requirements, i.e., the ranking would be dependent upon the reviewer's end use of the product/service. Thus, the ranking of the items should occur, if at all, after the assessment has been completed, when items with 'NSI' or '' ratings can be prioritized relative to their potential impact on the validatability of the application. Example Application and Assessment Findings The following is an actual example of the application of the Assessment Matrix in the performance of an assessment of a software supplier. The company assessed, noted as "[CLIENT]," developed both configurable and application-specific software for their manufactured systems. At the time of the assessment [CLIENT] was developing new configurable software following a quality program that had been in place for less than a year. The Corporate Quality Manual and other documentation related to the software development and validation process at [CLIENT] were evaluated using the Matrix described above. [CLIENT] scored 227 out of 20 points, leaving an improvement potential of 9. This was a very good rating considering the youth of [CLIENT's] quality program, and relative to similar assessments that have been performed using this approach. It is important to bear in mind that the observations and ratings summarized in the following completed matrix concerning the [CLIENT's] Corporate Quality Program pertain to the development and release of software and documentation solely. No evaluation of quality systems as they pertain to the manufacturing facilities was made as a part of this assessment plan matrix. The findings of the assessment too lengthy for inclusion in the comments section of the matrix are presented following the matrix. Recommendations for improvement for [CLIENT] are presented in Example Assessment Recommendations section. A listing of the source code (the human readable form of the code) must be available to perform structural testing. will be, accepted by FDA as a substitute for GXP regulations. However, if one can provide a high level of assurance through a supplier assessment that ISO and/or ANSI/IEEE (also used in developing the assessment matrix) standards have indeed been followed in the development of whatever product and/or service the supplier is providing, our experience has shown that you will have demonstrated to FDA's satisfaction that the product/service is validatable. 224 I n s t i t u t e o f Va l i d a t i o n Te c h n o l o g y

5 Figure 1 See discus'sion be C o n d u c t i n g Au d i t s, G A P A s s e s s m e n t s, & C o r r e c t i ve A c t i o n s Has the company assigned personnel and resources to and validation activities independent from the code development process?. Has the company a means.of advising management that the requirerljents of the quality plan are implemented and mainta.ined? See discussion below ISO NSI See discussion below i 6. Does the company have a policy of conducting joint reviews between supplier and purchaser? 7. Doe)he company have a policy of establishing a uniqu quality plan for each project? ISO See discussion below ISO 900-A Does the company have a procedure for conducting internal audits to ensure compliance with the quality plan and adherence to standards? 9. If yes to Item 8, is there a policy that effectively notifies management of quality discrepancies? ISO NSI 0 ISO N1 0 No procedure was available for review See Item 8 and discussions regarding the lack of formal management endorsement a. Investigating the causes of non-conforming product and corrective action necessary to prevent recurrence? b. Analyzing all processes, work operations, concessions, quality records and customer complaints to detect and eliminate potential causes of non-conformance? c. Initiating preventative actions to deal with problems corresponding to the risks encountered? S d. Applying controls to ensure that corrective actions are taken and that they are effective? e. Implementing and recording changes in procedures resulting from corrective action? Refer to item 1Oc Refer to item 10c 11. Does the company have a policy in place to handle changes in the Purchasers requirements during development? 6J The required actions should be formalized via a procedure C/) 6' F' Qo I Item is addressed informally, but no procedure was available for review :b or:::s ;tj OJ :

6 226 g I n s t i t u t e o f Va l i d a t i o n Te c h n o l o g y (I) ::t ;e (I) 8' F'!=l QQ :b Qr ::l ;n tb a. Development phases to be carried out?f Iw I I The oolicv coverina this item e. Analysis of potential problems associated with the development phases and with the achievement of specified requirements? 14. Does the company have a formal development plan that includes the following issues: a. A schedule of development, implementation and associated deliverables? c. A listing of orqanizational responsibilities, 1t::UUI\.lt:: d. A listing of organizational and technical interfaces between different groups, specifically between the software development effort and testing group? 1. Does the development plan identify methods for ensuring that all activities are carried out correctly and include: d conventions for software development? opment? c. A configuration management plan? 8 8 Item presently addressed informally. Procedure should be implemented I 1: