Computer and Software Validation Volume II

Transcription

1 Table of Contents Maintaining the Validated State in Computer Systems Orlando Lopez Use Automated Testing Tools? Janis V. Olson Considerations for Validation of Manufacturing Execution Systems Chris Wubbolt and John Patterson John E. Lincoln Storm Clouds? Cloud Computing in a Regulated Environment Robert H. Smith Robert H. Smith Computer System Compliance and Quality Planning Bernard T. O Connor John E. Lincoln Requirements Management Orlando Lopez Integrating Risk Management into Computer System Validation Timothy Fields

2 Orlando Lopez Maintaining the Validated State in Computer Systems Orlando Lopez INTRODUCTION In the life science industries, plenty of attention is placed in the validation of computer systems during the development of new systems and associated infrastructure. After deploying the computer systems to support operations, the operational life of these systems initiates. This is the period of time that relevant processes are in place to run the computer system in its operational environment, monitored for satisfactory performance, and modified as part of corrective, adaptive, perfective, and preventive maintenance. Computer systems maintenance and operations create a threat to the validated state of the systems by altering the delivered validated computer systems to meet the changing environment and/or changing needs of users. The primary reasons a system development is not complete without the operational phase are the presence of latent defects in the computer system and changes that occur in its operational environment. Over 67% of the relative cost of the system will be accounted to the computer systems maintenance (1). A percentage of this budget will be assigned to the revalidation effort. It is very common in the life science industry to observe that a few years after deploying a computer system, deficient operational supporting processes and/or the incorrect implementation of such processes nullify the validated state of the computer system causing remediation activities. This situation is a typical situation across multiple computer systems and provokes a remediation project across the company and adds cost to the operational life of many computer systems. This paper provides a brief description of the typical operational life activities and processes in support to preserve the validated state of computer systems performing regulated operations. OPERATIONAL LIFE After the system has been released for operation, the computer system operational life activities take over(2). The operational life is governed by two key processes: operation and maintenance. The operation process defines the activities of the organization that provides the service of operating a computer system in its live environment for its users. The maintenance process defines the activities of the organization that provides the service of managing modifications to the software product to keep it current and in operational fitness. This process includes, as applicable, the migration and retirement of the computer system. To maintain uniformity during the maintenance period, the maintenance activities must be governed by the same procedural controls followed during the development period. Written procedural controls must be available for the operation and maintenance of computer systems (3). OPERATIONAL ACTIVITIES Routine use of computer systems during the operational life requires procedural controls that describe how to perform operational activities. These operational procedural controls must be in place and approved by the appropriate individuals. The execution of these procedural controls should be monitored by the regulated user to verify the accurate implementation and adherence. These procedural controls should be reviewed on a periodic basis as according with the local retention policy. In addition, it is vital that management should ensure 2

3 Orlando Lopez that the relevant associates are trained accordingly. Key operational procedural controls are: Archiving: In the context of the regulated user, archives consist of records that have been selected for permanent or long-term preservation based on their evidentiary value. All computer system baselines should be archived in an environmentally controlled facility, as applicable, that is suitable for the material being archived and that is both secure and, where possible, protected from environmental hazards. A record of all archived materials should be maintained. Backups: A backup process must be implemented to allow for recovery of the system following any failure that compromises its integrity (4). Integrity and accuracy of backed-up data and the ability to restore the data should be checked during validation and monitored periodically (5). The frequency of backups depends on data criticality, amount of stored data, and frequency of data generation. The procedural controls establishing the backup process must be in place to ensure the integrity of backups (secure storage location, adequately separated from the primary storage location, etc.). This may be part of a more general disaster recovery plan. Business Continuity: Business continuity procedural controls, including disaster recovery procedural controls, ensure minimal disruption in the case of loss of data or any part of the system. It is necessary to ensure that the integrity of the data is not compromised during the return to normal operation. At the lowest level, this may mean the accidental deletion of a single file in which case procedural controls should be in place for restoring the most recently backed-up copy. At the other extreme, a disaster such as a fire could result in loss of the entire system. The procedural control employed should be tested regularly and all relevant personnel should be made aware of its existence and trained to use it. A copy of the procedural controls should be maintained off-site. Infrastructure Maintenance: The procedural controls applicable for the preventative maintenance and repair of the infrastructure provide a mechanism for anticipating problems and, as a consequence, possible loss of data. In addition to the typical infrastructure elements such as system-level software, servers, wide-area network, local-area manager, and the associated components; the infrastructure includes uninterruptible power supplies (UPSs) and other emergency power generators. Modern infrastructure hardware usually requires minimum maintenance because electronic circuit boards, for example, are usually easily replaced and cleaning may be limited to dust removal. Diagnostic software is usually available from the supplier to check the performance of the computer system and isolate defective integrated circuits. Maintenance procedural controls should be included in the organization s procedural control. The availability of spare parts and access to qualified service personnel are important for the smooth operation of the maintenance program. Problem Reporting: The malfunction or failure of a computer system components, incorrect documentation, or improper operation that makes the proper use of the system impossible for an undetermined period are some characteristics of the incidents that can affect the correct operation of a computer system. These system incidents may become non-conformances. In order to remedy problems quickly, a procedural control must be established to record any computer system failures from the users of the system. These enable the reporting and registration of any problem encountered by the users of the system. Problem Management: Reported problems can be filtered according to whether their cause lies with the 3

4 Orlando Lopez user or with the system itself and then fed back into the appropriate part of the supplier s organization. In order to remedy problems quickly, a procedural control must be established if the system fails or breaks down. Any failures, the results of the analysis of the failure, and, as applicable, any remedial actions taken must be documented. Those problems that require a remedial action involving changes to any baseline are then managed through a change control process. Retirement: The retirement of computer systems performing regulated operations is a critical process. The purpose of the Retirement Period is to replace or eliminate the current computer system and, if applicable, ensure the availability of the data that it has generated for conversion, migration, or retirement. Restore: A procedural control for regular testing of restoring backup data to verify the proper integrity and accuracy of data must also be in place. Security: Computer systems security includes the authentication of users and access controls. Security is a key component for maintaining the trustworthiness of a computer system and associated records. Security is an ongoing element to consider and is subject to improvement. In particular, after a system has been released for use, it should be constantly monitored to uncover any security violations. Any security violation must be followed up and analyzed and proper action must be taken to avoid a recurrence. Training: All staff maintaining, operating, and using computer systems that perform regulated operations must have documented evidence of training in the area of expertise. For users, the training will concentrate on the correct use of the computer system, security, and how to report any failure or deviation from the normal operating condition. MAINTENANCE ACTIVITIES The validated status of computer systems performing regulated operations is subject to threat from changes in its operating environment, which may be either known or unknown. The January 2011 of the Eudralex Volume 4- Annex11 Computerised Systems establishes in Item 10 that any changes to a computerised system including system configurations should only be made in a controlled manner in accordance with a defined procedure. (5). The above statement is consistent with other regulations and guidelines such as: FDA Code of Federal Regulations Title 21 Part (6), , , (7), 11.10, (8); World Health Organization (WHO) - Technical Report Series, No. 937, Annex 4. Section 12 (9), and Pharmaceutical Inspection Convention Scheme (PIC/S) PI (Part Three) (10). Explicitly in Section 3.3 of the WHO Technical Report (9) stipulates three events to cover during the maintenance of computer systems: Verification and revalidation: After a suitable period of running a new system, it should be independently reviewed and compared with the system specification and functional specification. The periodic verification must include data checks, including any audit trail. Computer systems used to control, monitor, or record functions that may be critical to the safety of a product should be checked for accuracy at intervals of sufficient frequency to provide assurance that the system is under control. If part of a computerized system that controls a function critical to the safety of the product is found not to be accurate, then the safety of the product back to the last known date that the equipment was accurate must be determined. Change control: Modifications and adjustments to computer systems shall only be made in accordance with a defined procedural control that includes provisions for checking, approving and implementing the modification and/or adjustment. Checks: Data should be checked periodically to confirm that they have been accurately and reliably recorded and/or transferred. These checks include the following EU Annex 11 items: Data (11.5), Accuracy Checks (11.6), and Data Storage (11.7). 4

5 Orlando Lopez SUMMARY The regulated applications and infrastructure must be well maintained during the operational phase to assure that the systems remains in a validated state and provides costsavings to the maintenance of the computer systems. The operational phase comprises of: Ensuring that system changes are performed according to the approved change control procedure Maintaining training records for users and technical support personnel Controlling user access Conducting periodic reviews based on established criteria Ensuring that data is backed up and archived in accordance with established schedules Identifying and tracking system problems and related corrective actions Maintaining configuration items. The above activities must be outlined in written and approved procedural controls. About the Author Orlando López, currently working with Smith & Nephew, Memphis (Tennessee) as an IT Manager, Global Regulatory/ Compliance, has over 20 years of experience within Pharmaceutical and Medical Devices manufacturers. His role in Smith & Nephew includes subject matter expert for quality and compliance oversight on new computer systems implementation and maintenance. He is the author of two books: 21 CFR Part 11 - A Complete Guide to International Compliance, published by Sue Horwood Publishing Limited, and Computer Infrastructure Qualification for FDA Regulatory Industries published by Davis Healthcare International Publishing (www. suehorwoodpubltd.com).he may be contacted by at olopez6102@msn.com. References 1. Zelkowitz et al. [1979]. 2. O. Lόpez, 21 CFR Part 11: Complete Guide to International Computer Validation Compliance for the Pharmaceutical Industry, CRC Press: Boca Raton, FL. 2004, pp ICH Q7 Good Manufacturing Practice Guide for Active Pharmaceutical Ingredients. 4. OMCL Network/EDQM of the Council of Europe, Validation of Computerised Systems - Core Document (July 2009). 5. EU Volume 4 Good Manufacturing Practice Annex : Computerised Systems (Brussels, BE, June 2011). 6. Code of Federal Regulations, Title 21, Food and Drugs (Government Printing Office, Washington, DC), Part Code of Federal Regulations, Title 21, Food and Drugs (Government Printing Office, Washington, DC), Part Code of Federal Regulations, Title 21, Food and Drugs (Government Printing Office, Washington, DC), Part WHO, Validation of Computerized Systems, Technical Report 937, Annex 4 in Appendix 5, (2006). 10. PIC/S, Good Practices for Computerised Systems in Regulated GXP Environments (2007). Originally Published in Journal of GXP Compliance Volume 17 Number 2 5

6 Janis V. Olson Software Validation: Can an FDA-Regulated Company Use Automated Testing Tools? Janis V. Olson INTRODUCTION - WHAT ARE SOFTWARE TEST TOOLS? Software test tools help development and testing teams verify functionality, ensure both the reliability and security of the software they develop, and investigate software bugs. Off-the-shelf tools are available for all stages of software development. Examples include static code analyzers, record and replay, regression testing, and bug tracking. Some software testing tool vendors offer an integrated suite that starts with the gathering of requirements and continues through software development and testing throughout the life of a project, including supporting the live system. Other vendors concentrate on a single part of the application development life cycle, such as just testing. HOW DO YOU USE A TOOL SO THE US FOOD AND DRUG ADMINISTRATION WILL ACCEPT IT? FDA will accept use of a software testing tool if the following are done: Know, understand, and document the intended uses of the tool Test and verify the tool in the context of the intended uses Document the validation of the tool Maintain the validated state Know, Understand, and Document the Intended Uses of the Tool Companies, project teams and individuals only get what they want if they know what their requirements are for how they are going to use the tool. So the first task is to understand the intended use requirements and get agreement that this is how the tool will be used. Understand the native functionality of the tool when used as is and what needs to be configured or customized to get the performance needed. Plan what to do in a validation plan or a project plan. Identify what will be done by whom when. Then, go to work on setting up the tool. Documentation of the process is the key to success. Document what can be done with the tool s native configurations and any customizations needed to assure that the tool will work as expected. Review the native functions and the configurations and map them back to the intended use requirements to assure that all the needs of the tool as stated in the intended use requirements have been covered. Document the intended uses of the tool in the context of the processes that will use the tool in written procedures. Procedures need to be written at the level needed by the people in the organization who will use them. Test and Verify the Tool in the Context of the Intended Uses One should use code already tested or verified you have through other means. Determine the amount of testing/verification and the extent of testing that needs to be done based on the risk of the failure of the tool. Low-risk tools still require testing and verification that the tool will meet one s intended uses. Document your testing with test cases or scripts. Document the results; including the objective evidence of the actual results that can be compared to the expected results. Do not just record that the tool worked as expected or passed. Use good and poor code to test the tool. Keep those pieces of code as part of the test documentation. Do fault insertion to assure that the tool behaves properly in all the expected scenarios. Do testing in the context of the intended use. Understand that risk is based on the intended use of the tool. Know the tool s limitations. 6

7 Janis V. Olson Each tool will solve problems but also create potential issues. Understand what these potential issues are and their consequences. Mitigate the possible risks possibly as part of the procedures for the use of the tool. Document the Validation of the Tool Validation is more than just the testing. It is the linking of all the activities and documentation to one another. The intended use requirements are mapped to the tool s native functions and any special configurations and/ or customizations. They are also mapped to the testing scripts. The executed scripts have the documentation of the objective evidence of the results that can be compared to the expected results. Once all this documentation is together and linked, this is the assembled evidence that the tool is validated for its intended uses. Write a summary report of the effort. Train the users and use the tool. Maintain the Validated State Once the tool is validated, monitor the use and the results of the tool. If the users find that they have additional uses that were not validated, those will need to be validated prior to use. Likewise, if users find new expected scenarios that were unknown at the time of the original validation, additional documentation of the behavior of the tool under these new scenarios must be placed in the validation file and tested. Monitor for additional issues and their consequences and evaluate the risks. Document new risks as part of the risk analysis. TWO EXAMPLES OF COMMONLY USED TESTING TOOLS Static Code Analyzer A static code analyzer is a commonly used code review tool. FDA recommends the use of static code analyzers in high-risk applications, especially for medical device software. Static code analyzers will find issues that software engineers know are not associated with the basic code. Many false positive issues are found that software engineers then have to evaluate and investigate. Static code analyzers have coding rules that they enforce. If the code does not conform to their rules, the software engineers will get many issues that need to review and resolve. Once the software engineers have resolved them, this information on the false positives routinely identified by the tool can be used so that the next time the false error is found, in the same way by the static code analyzer there is no need to do another investigation. Some companies change their coding standards to conform to the tool for all future development. However, because they often have built code on old code bases, they maintain a list of false errors they will ignore or not investigate fully. Naturally, this is based on the risk of not investigating these errors. Static code analyzers also allow software engineers or the tool vendor to do custom configurations. These customizations tell the analyzer to ignore some of the false errors that the tool finds but that the developers know are not issues with the code and can be ignored. A company should do extensive research to determine the best static code analyzer for their needs. Code modules with common errors, complexities, and poor coding practices as well as modules with good code should be developed if not already available to the software engineers. Several different manufacturers static code analyzers should then be evaluated. The tool that found the most types of errors and issues should be chosen. Limitations of the static code analyzer should be documented so other verification methods can be used to find errors the static code analyzer missed or was not intended to find. A list of common false errors the tool found should be compiled. The false code errors, the investigations into each one, the results of the investigations, and what the software engineers would do with each false error when found in their code modules was prepared. These became part of their standard operating procedures in the use of the tool. Record and Replay Another commonly used tool is an automated record and replay testing tool. These tools allow testers to record each keystroke as they are running a test manually. The testing team then can schedule testing to be performed any time in the future and the tool will replay each keystroke as if a tester was doing it manually. The advantage of this approach is that the documentation accumulated while recording, such as screen shots or other documentation, is used not only to show that the code passed the test when run manually but also used to compare the tool output the 7

8 Janis V. Olson very first time it is run and demonstrates, in fact, that the tool is running the manual tests and running them correctly. To optimize testing, many of these tools allow testers to modify the keystrokes rather than rekeying them, insert negative testing, or change if the code is changed and requires changes to the testing steps run by the tool. Modifications to the keystrokes outside the record feature of the tool should be documented and independently reviewed to assure that the changes are done appropriately and correctly. The documentation of the keystrokes is the code for the tool. After validation of the tool, testers can have the tool continue to provide the documented evidence of the test results, or chose after to only have the tool record that the tests passed or how they failed. All testing failures will need to be investigated and resolved. About the Author Janis V. Olson (Halvorsen) is Vice President of Regulatory and Quality Services, a global team of FDA compliance experts. Janis worked for FDA for over twenty-two years in various positions. She currently works with EduQuest assisting companies with FDA regulation compliance and understanding computer system development and validation. SOFTWARE TESTING TOOLS ARE BUSINESS DRIVERS AND HAVE BENEFITS The use of software testing tools will increase the quality of software applications by efficiently detecting functional, performance, and security issues. Compared to manual testing, which may be inconsistent because of human error, testing tools also are more consistent and repeatable in following the documented testing procedures/processes. Many of the tools can measure software performance throughout development, testing, and maintenance of the software product. Other tools can assist in conducting and documenting risk analysis and benchmarking against other software. The use of these tools can improve a company s time-to-market or time-to-implementation because of more frequent and consistent testing, finding, and fixing of issues and bugs. This improves product lifecycle management processes and reduces your total cost of ownership. Once the tool is in a validated state, use it over and over. Companies can be confident that the tool is giving them the information they need to demonstrate, to FDA, to their customers, and to all their stakeholders, that the software is being tested consistently and effectively. Originally Published in Journal of Validation Technology Volume 18 Number 3 8

9 Chris Wubbolt and John T. Patterson Considerations for Validation of Manufacturing Execution Systems Chris Wubbolt and John T. Patterson INTRODUCTION Improvements in computer-based control systems through technological breakthroughs in integrated circuit manufacturing and software design have been essential to the development of the modern day pharmaceutical manufacturing facility. Automation of processes have improved efficiency, reduced costs, and increased compliance for the manufacture of pharmaceutical products. The MES represent one of the more recent offspring of these technological improvements and are capable of a high degree of integration with different types of equipment and systems including both shopfloor automation equipment (e.g., DCS, PLC) and the business management systems such as enterprise resource planning (ERP) systems. The MES also enables critical functionality including electronic batch records (EBR) that help ensure the highest level of consistency and reproducibility, thereby ensuring the highest level of product quality and corresponding assurance of patient safety. Because of the favorable business case provided by the MES for pharmaceutical and medical device products, their use within these types of manufacturing environments is increasing. A common architecture for an MES system is shown in Figure 1. VALIDATION STRATEGY AND APPROOACH To ensure the appropriate validation of an MES design, it is essential to have a well-defined validation strategy that is based on company validation and quality standard operating procedures (SOPs) and further detailed in an approved quality assurance plan (QAP) or equivalent validation or quality planning document. GAMP5 provides an excellent basis for development of a company-wide quality management system that comprehensively addresses requirements for validation, implementation, and use of computerized systems, including explanation of key principles as well as templates and additional guidance and instruction (see Reference). For MES, several elements that would normally be considered in a validation or quality assurance plan are as follows: System scope including interfaces with other IT/ automation systems System description and intended use Roles and responsibilities during the validation effort, including required approvals on deliverables Governing quality management system and system development lifecycle (SDLC) policies and SOPs, including: Computer system validation policy and procedures Risk management Configuration management Change control Training MES vendor involvement (additional discussion pertaining to vendor involvement is provided below) Testing and qualification strategy SDLC deliverables, including: Quality assurance plan 9

10 Chris Wubbolt and John T. Patterson Figure 1: Common architecture for an MES system. MES Overview Specifications Manufacturing Execution System Test Documents Reports Hardware Processor Speed RAM Disk Space Work Station #1 Software Application Software Operating System (Windows XP) Workstation Network Software Hardware IBM Compatible Laptop computer User/Technical Manuals SOPs Instrumentation and Equipment Requirements Functional specifications System design specifications Risk assessments Design qualification Configuration management Qualification protocols Qualification and test summary reports Electronic batch record (EBR) development SOPs Part 11 compliance documentation, including electronic record and electronic signature applicability assessments Security and access controls, including user groups and roles Definition and description of batch end reports Development phase issue management (e.g., test incident reports) Documentation management Validation summary reporting Incident, change, and problem management SOPs (operational phase) Back-up and recovery processes EBR operational phase change management SOPs Periodic review reports Decommissioning (if required). LEVERAGING MES VENDOR QUALITY PROCESSES Implementation and validation of MES is a considerable undertaking and requires significant organizational resources. 10

11 Chris Wubbolt and John T. Patterson The ability to leverage the MES vendor s design, development, testing and verification, and quality processes and documentation may help to streamline the validation and implementation processes while maintaining a high level of quality and compliance. The MES vendor s quality processes and documentation must be formally assessed to allow leveraging of existing documentation and testing that the vendor may have conducted. Typically, for critical systems such as an MES, an on-site audit must be conducted to adequately assess the vendor s quality systems and system development lifecycle methodology. A vendor audit normally includes assessment of the following vendor processes: Overall quality system Organizational structure System development lifecycle methodology, including System functional and detailed design specifications Coding standards and code review Unit testing Integration and system testing Configuration management Incident, problem, and change management Release management processes, including major, minor, and patch releases Training program Customer support. The ability to leverage a vendor s previous efforts is dependent upon the quality of the vendor s documentation and processes, particularly those related to their SDLC processes. Although software vendors are typically not regulated by the US Food and Drug Administration (unless the vendor produces software used in medical devices), if vendor documentation, including testing, can be leveraged it must be at a GMP quality level. GMP quality documentation processes typically include adherence to applicable SOPs, version control, approved documents, and an appropriate level of quality review and approval. Testing and incident resolution, including review and approval of test results, should also be carefully evaluated during the vendor assessment and audit process. The results of the vendor assessment are typically summarized within a vendor self-assessment or audit report depending on the criticality of the vendor supplied software. If vendor documentation and processes are utilized during the validation process, the vendor self-assessment, audit report, or supporting documentation should clearly indicate the quality processes and documentation that were assessed. If the vendor s processes and documentation are deemed adequate, it may be possible to leverage the vendor s documentation and previous SDLC activities during the MES validation effort. If quality issues are identified during the vendor assessment, the ability to leverage vendor activities and documentation may be limited. The amount of vendor documentation to be used during the validation effort should be documented within the project validation or quality plan, or risk assessment documentation, along with appropriate justifications. GLOBAL VS. LOCAL IMPLEMENTATIONS For multi-site organizations, another consideration is whether to develop individual (i.e., stand alone) MES instances versus a global instance, which could better facilitate the highest level of standardization within the organization, as well as allowing leveraging of resources. Although this may not be a priority, particularly for smaller organizations, significant efficiencies can be gained through the use of a single (or minimum) number of global instances from which all other local MES instances can be more efficiently configured. If such a global and local approach is pursued, it is important to consider how their respective quality and validation planning processes will work and coordinate with each other, including roles and responsibilities within the local and global organizations. For example, to ensure the ability to update the global instance independent of the local instances, it is considered 11

12 Chris Wubbolt and John T. Patterson Figure 2: MES validation process. System Identification MES Validation Quality / Validation Plan User Requirements Specifications System Design Design Specification Configuration and Build - Change Mgt. - Incident/Problem Mgt. - Periodic Review Testing Operation and Maintenance Functional Specification Unit and Integration Testing Installation, Functional, User Acceptance Validation Report SOPs beneficial to have a separate quality or validation plan in place for any global instances. The local MES quality or validation plan will also need to appropriately reference and align with the global quality and validation plan because it would be assumed that the local instances would leverage significant testing and qualification (e.g., operational qualification [OQ]) from the global, both of which may be needed as documented evidence during regulatory inspections. Finally, one final consideration is whether the global instance is for testing purposes only, or whether it is also used for manufacture of product. Either approach can be used and ultimately depends on what makes sense for the organization, although the global approach will require more rigorous project management, communication, and configuration management controls and processes. RISK MANAGEMENT One other important consideration is the use of the risk management techniques in ensuring appropriate risk failure scenarios are identified and appropriate mitigations are put in place prior to use of the MES in pharmaceutical or medical device manufacturing. Such risk assessments should be used wherever possible, particularly when there may be limited knowledge of the possible failure scenarios and should be focused on failure scenarios that can have impact on product quality or patient safety. For example, risk assessments of interfaces between MES and business ERP systems can be effective in identifying possible failure points that can be used to ensure appropriate monitoring is put in place prior to the commencement of manufacturing. Mitigation strategies identified as a result of the risk assessments may include, but are not limited to, identification of the need for additional testing (or, in some cases, reduced testing or testing that leverages vendorsupplied documentation), system re-design, procedural controls, or increased monitoring of system operations. After a mitigation strategy is implemented, it is important to verify the effectiveness of the mitigation strategy. One way to assess the effectiveness of a mitigation strategy is to re-rank the risk scenario to determine if the mitigation strategy has reduced the risk to a lower priority. 12

13 Chris Wubbolt and John T. Patterson 13 Risk Assessment Example The use of a risk assessment to focus validation efforts to critical activities may include, for example, the ability to leverage unit testing or development testing in place of operational qualification or system testing during the validation effort. The risk assessment effort should be completed following an approved SOP and documented accordingly, typically within a risk assessment report. In addition, the unit or development testing must be of GMP quality and completed in a controlled testing environment. A risk assessment that identifies low risk functionality (as defined by the risk assessment SOP) may allow unit or development testing to be used in place of operational qualification or integrated system testing. Low risk functions typically refer to those functions that have minimal impact on product quality, patient safety, or data integrity. SYSTEM DEVELOPMENT LIFECYCLE One final consideration for the MES validation and quality planning process is the type of software development process (e.g., traditional waterfall, rapid prototyping) that will be used. In general, if only very limited configuration changes are planned then a traditional waterfall method is probably more appropriate. However, if more sophisticated software configuration or some customization is needed, a more efficient iterative or rapid prototyping method should be used and appropriately managed using pre-approved SOPs and work instructions to ensure appropriate control of the software development process. OPERATIONAL CONTROLS Once the initial MES validation is completed, it is equally important that operational phase processes including change, incident and problem management, and periodic review along with appropriate business and quality governance be established to ensure that the MES maintains a state of on-going validation. It is critical that such operational phase processes exhibit the following characteristics: Managed by defined processes including approved SOPs and work instructions IT-based incident and problem management systems aligned with any business corrective action preventive action (CAPA) systems Change management processes that include both business and regulatory quality involvement Periodic review is performed at appropriate time intervals Risk assessments be revisited (as appropriate based on incident or problems) Ongoing auditing of activities by independent quality group. When properly implemented, the MES validation process will include both initial development and operational phase processes as described in Figure 2. SUMMARY In summary, the considerations for validation of an MES system are similar to any other type of IT or automation system used in pharmaceutical manufacturing. However, it is important to understand that in many cases the technological complexity of the MES versus many other control technologies, including the interfaces with other systems, may increase the need for more rigorous review, testing, and oversight of the MES validation process. This increase in technical complexity and the prevalent use of paperless batch records afforded by MES will demand the highest level of inspection readiness during regulatory (e.g., EMA, FDA) reviews. REFERENCE ISPE, GAMP 5: A Risk-Based Approach to Compliant GxP Computerized Systems, GAMP 5, February Originally Published in Journal of Validation Technology Volume 18 Number 1

14 Chris Wubbolt and John T. Patterson About the Author Chris Wubbolt is a principal consultant at QACV Consulting specializing in computer systems compliance, including validation, as well as quality assurance activities such as auditing, training, and six sigma quality improvement processes. He may be reached by at chris.wubbolt@qacv.net. John T. Patterson is the Senior Director of IT Compliance and is responsible for the overall regulatory compliance and inspection readiness of IT and Automation capabilities within the Manufacturing and Supply Chain function of Merck & Co. He may be reached by at john_patterson@merck.com. 14

15 John E. Lincoln Lifecycle Considerations for Device Software John E. Lincoln INTRODUCTION This discussion addresses methodology and documentation recommended by the US Food and Drug Administration to effectively address lifecycle considerations in software development. This includes steps to verify (or test) and validate software or firmware in a medical device to assist in performance, provide a display, accept input, or similar functions. It also applies to software that is the actual product such as imaging software. This discussion is based on the model provided in the FDA guidance document on the General Principles of Software Validation of January 11, Much of the following information is taken from this document (1). The basic principles discussed can also be applied to software used in processes, production equipment, facilities, test equipment, quality management system (QMS) under the current good manufacturing practices (CGMPs), and in other regulated industries other than devices. Note that QMS issues also require addressing the issues outlined in 21 CFR Part 11, Electronic Records/ Electronic Signatures (2). While most software books in the software industry address lifecycle as the development lifecycle up to the product release, FDA and the International Organization on Standardization (ISO) expect lifecycle to go far beyond development and initial release and to include updates, distribution and control, and other activities through final retirement and de-commissioning. FDAregulated companies frequently neglect these latter areas. SOFTWARE IS DIFFERENT FROM HARDWARE Software is different from hardware. This seems obvious, but it bears emphasis because it often is not given the consideration required in a company s CGMP systems. The conscious recognition of this fact is crucial to all product and software lifecycle activities. While software shares many of the same engineering tasks as hardware, it has some important differences. Some of these are listed in the guidance as follows: The vast majority of software problems are traceable to errors made during the design and development process. While the quality of a hardware product is highly dependent on design, development, and manufacture, the quality of a software product is dependent primarily on design and development with a minimum concern for software manufacture. Software manufacturing consists of reproduction that can be easily verified. One of the most significant features of software is branching (i.e., the ability to execute alternative series of commands based on differing inputs). This feature is a major contributing factor for another characteristic of software its complexity. Even short programs can be complex and difficult to fully understand. Testing alone cannot fully verify that software is complete and correct in most cases. In addition to testing, other verification techniques and a structured and documented development process should be combined to ensure a comprehensive validation approach. Unlike hardware, software is not a physical entity and does not wear out. In fact, software may improve with age, as latent defects are discovered and removed. However, as software is constantly updated and changed, such improvements are sometimes countered by new defects introduced 15

16 John E. Lincoln into the software during the change. Unlike some hardware failures, software failures occur without advanced warning. The software branching that allows it to follow differing paths during execution may hide some latent defects until long after a software product has been introduced into the marketplace. Another related characteristic of software is the speed and ease with which it can be changed. This factor can cause both software and non-software professionals to believe that software problems can be corrected easily. Combined with a lack of understanding of software, it can lead managers to believe that tightly controlled engineering is not needed as much for software as it is for hardware. In fact, the opposite is true. Because of its complexity, the development process for software should be even more tightly controlled than for hardware in order to prevent problems that cannot be easily detected later in the development process. Seemingly insignificant changes in software code can create unexpected and significant problems elsewhere in the software program. The software development process should be sufficiently well planned, controlled, and documented to detect and correct unexpected results from software changes. Given the high demand for software professionals and the highly mobile workforce, the software personnel who make maintenance changes to software may not have been involved in the original software development. Therefore, accurate and thorough documentation is essential (1). For these and other reasons, software engineering needs an even greater level of managerial scrutiny and control than does hardware engineering throughout their respective lifecycles. And the software lifecycle takes on even greater importance (1). SOFTWARE LIFECYCLE ACTIVITIES The FDA guidance does not recommend the use of any specific software lifecycle model. Software developers are expected to establish a software lifecycle model that is appropriate for their product with consideration of product risk to the end user, clinician or patient, and in harmony with their company standard operating procedures (SOPs) and structure. The software lifecycle model should cover the software from its birth to its retirement. Activities in a typical software lifecycle model as defined by the FDA guidance include the following: Quality planning System requirements definition Detailed software requirements specification Software design specification Construction or coding Testing or verification Installation Operation and support Maintenance Retirement Systems, validation, and controlled documentation to support the above (1). Verification, testing, and other tasks that support software validation occur during many of these activities. A lifecycle model organizes these activities in various ways and provides a framework for monitoring and controlling the software development project. Several software lifecycle models (e.g., waterfall, spiral, rapid prototyping, and incremental development) are defined in FDA s Glossary of Computerized System and Software Development Terminology (3). If a product is sold outside the US and carries a CE-mark, then a company s notifiedbody may have their own suggestions as well. Developing and executing a software lifecycle program requires that the company write an SOP on the subject that includes either of the following: Pointers to other documents that address each of the bullet points defined as part of the software s lifecycle Excerpts from appropriate documents in the SOP or a separate document (the approach favored by the author) that would still require that either some pointing or referencing is still performed, or that a tight change control or where used system exists. Change control must ensure that when one of those linked documents is updated, the update is reviewed for possible impact on the SOP or 16

17 John E. Lincoln separate lifecycle document. A company may opt for some variant that results in a self-contained SOP with either an attachment or a separate lifecycle document tied to specific software app, revision or release, device master record (DMR) document, or similar controlled document defined by the SOP. One of the FDA guidance documents on devices having commercial-off-the-shelf (COTS) software illustrates how a company should consider product hazards throughout the product s lifecycle by means of a flow chart (4). Whatever method a company initially selects, it can be developed, tried and refined, continuously improved, or ultimately completely revised as use and change history dictate. Any discussion of software lifecycle as expected by FDA is broader than the scope of just software validation in the strictest definition of that term. Planning, verification, testing, traceability, configuration management, and many other aspects of good software engineering discussed under lifecycle management are important activities that together help to support a final conclusion that software is validated at any point in its lifecycle. In any such activity there must be an integration of software lifecycle management and risk management activities. Based on the intended use and the safety risk associated with the software to be developed, the software developer should determine the specific approach, the combination of techniques to be used, and the level of effort to be applied. Software validation and verification activities must be conducted throughout the entire software lifecycle. When the software is developed by someone other than the device manufacturer (e.g., off-the-shelf software, or contracted software development), the software developer may not be directly responsible for compliance with FDA regulations. In that case, the party with regulatory responsibility (i.e., the device manufacturer) needs to assess the adequacy of the off-the-shelf software developer s activities and determine what additional efforts by the company or the contractor are needed to establish that the software is validated for the device manufacturer s intended use and to regulatory requirements. CHANGE CONTROL THROUGHOUT THE SOFTWARE LIFECYCLE An FDA analysis of medical device recalls revealed that approximately 8% of recalls are attributable to software failures. Of those software-related recalls, the majority (~79%) were caused by software defects that were introduced when changes were made to the software after its initial production and distribution (1). Software validation and other related good software engineering practices over the product or software s lifecycle are viewed by FDA as a principal means of avoiding such defects and resultant recalls. Change control, including verification and regression testing or validation of such software changes needs to be a prominent part of the company software lifecycle management program. Software may be developed in-house or under contract. However, software is frequently purchased offthe-shelf for a particular intended use. All production or quality system software, even if purchased off-the-shelf, must have documented requirements that fully define its intended use and information against which testing results and other evidence can be compared to show that the software is validated for its intended use. FDA requirements in software lifecycle issues such as development, changes, verification, and validation are not new. Validation of software using the principles and tasks discussed and expected have been conducted in many segments of the software industry for more than 30 years. REQUIREMENTS AND SPECIFICATIONS The quality system regulation states that design input requirements must be documented and that specified requirements must be verified. A documented software requirements specification provides a baseline for both validation and verification. The software validation process cannot be completed without an established software requirements specification (see 21 CFR 820.3(z) and (aa) and (f) and (g)) (5). FDA will carefully review requirements and specifications lists against lifecycle activities and validation test cases during audits to ensure basic points are understood, addressed, and validated by test cases, and fully documented. The company should do no less under a formal documented system. VERIFICATION AND VALIDATION (V&V) 17

18 John E. Lincoln The quality system regulation is harmonized with ISO 8402:1994, which treats verification and validation as separate and distinct terms. On the other hand, many software engineering journal articles and textbooks use the terms verification and validation interchangeably, or in some cases refer to software verification, validation, and testing (VV&T) as if it is a single concept with no distinction among the three terms. The author generally defines verification as specific test, inspection, checklist, or similar activities. The author generally defines validation as a collection of verification activities, destruction or non-destructive, that shows scientifically that specified requirements that comprise the system s objectives have been consistently met. A conclusion that software is validated is highly dependent upon comprehensive software testing, inspections, analyses, and other verification tasks performed at each stage of the software development lifecycle. Testing of device software functionality in a simulated use environment (being increasingly stressed by FDA), and user site testing are typically included as components of an overall design validation program for a software automated device. RISK-BASED Software V&V are difficult because a developer cannot test forever, and it is hard to know how much evidence is enough. In large measure, software validation is a matter of developing a level of confidence that the device meets all requirements and user expectations for the software automated functions and features of the device. The level of confidence, and therefore the level of software validation, verification, and testing effort needed, will vary depending upon the safety risk (hazard) posed by the automated functions of the device. Expected risk analysis tools are outlined in ISO (which addresses devices) and ICH Q9 (which addresses pharmaceutical and is useful for basic principles for devices). Additional guidance regarding safety risk management for software may be found in Section 4 of FDA Guidance for the Content of Pre-market Submissions for Software Contained in Medical Devices, and in ANSI/AAMI/ISO and IEC (6-8). DEFECT PREVENTION Software quality assurance needs to focus on preventing the introduction of defects into the software development and lifecycle processes and not on trying to test quality into the software code after it is written or changed. Software testing is limited in its ability to surface all latent defects in software code. The complexity of most software prevents it from being exhaustively tested. Software testing is necessary, but in most cases, software testing by itself is not sufficient to establish confidence that the software is fit for its intended use. In order to establish that confidence, software developers should use a mixture of methods and techniques to prevent software errors and to detect software errors that do occur. This best mix of methods throughout the software s life depends on many factors including the development environment, application and intended use, project scope, programming language, and product end-use risk. Software validation is a major element of software maintenance throughout its lifecycle, and thus takes place within the environment of an established software lifecycle. The software lifecycle is usually defined generally by SOP, and specifically by a dedicated plan or plans. Both the systemic lifecycle SOP and the product or process-specific plan lists software engineering tasks and documentation necessary to support the software validation efforts and associated activities from cradle to grave. In addition, the software lifecycle plan and resultant activities contain specific verification and validation tasks that are appropriate for the intended use of the software. The final conclusion that the software is validated at any point in time should be based on evidence collected from planned efforts conducted throughout the software lifecycle up to that point in time. SOFTWARE VALIDATION AFTER CHANGES The guidance states, due to the complexity of software, a seemingly small local change may have a significant global system impact. When any change is made to the software, the validation status of the software needs to be re-established. Whenever software is changed, a validation analysis should be conducted not just for validation of the individual change, but also to determine the extent and impact of that change on the entire software system (1). Then the software developer conducts an appropriate level of software regression testing to show that unchanged but 18

19 John E. Lincoln vulnerable portions of the system have not been adversely affected. Design controls and appropriate regression testing provide the confidence that the software is validated after a software change (1). Validation documentation should be sufficient to demonstrate that all software validation plans and procedures have been completed successfully. DESIGN REVIEWS Design reviews should be held throughout the software s lifecycle, especially at important junctures. Reviews are a requirement during the development process to verify the completion of one key milestone prior to moving on to the next. The review team must have at least one independent member (i.e., one who has no vested interest in the outcome reached during the review process). Documented design reviews should also be held for regression testing of proposed or implemented changes, new revisions or releases, major hardware platform changes, and decommissioning. Validation activities should be conducted using the basic quality assurance precept of independence of review...assign or contract review members that are not involved in a particular design or its implementation, but who have sufficient knowledge to evaluate the project and conduct the verification and validation activities (1). The guidance states, answers to some key questions should be documented during formal [documented] design reviews. These include: Have the appropriate tasks and expected results, outputs, or products been established for each software lifecycle activity? Do the tasks and expected results, outputs, or products of each software life cycle activity: Comply with the requirements of other software life cycle activities in terms of correctness, completeness, consistency, and accuracy? Satisfy the standards, practices and conventions of that activity? Establish a proper basis for initiating tasks for the next software lifecycle activity? (1) ACTIVITIES AND TASKS Software validation and other lifecycle activities are accomplished through a series of milestones and tasks that are planned and executed at various stages of the software development lifecycle, including maintenance and change. These tasks may be one-time occurrences or may be iterated many times, depending on the lifecycle model used and the scope of changes made as the software project progresses (1). For each of the software lifecycle activities, there are certain typical tasks that support a conclusion that the software is validated and works as intended. However, the specific tasks to be performed, their order of performance, and the iteration and timing of their performance will be dictated by the specific software lifecycle model that is selected and defined by SOP and the safety risk associated with the software application. For low-risk applications, certain tasks may not be needed at all. However, the software developer should at least consider each of these tasks and should define and document which tasks are or are not appropriate for their specific application. This discussion (and the FDA guidance document [1]) is meant to be generic and is not intended to prescribe any particular software lifecycle model or any particular order in which tasks are to be performed. Once a software product has been baselined (i.e., approved), any change to that product should have its own mini life cycle plan including testing. Testing of a changed software product requires additional effort. Not only should testing demonstrate that the change was implemented correctly, testing should also demonstrate that the change did not adversely impact other parts of the software product. MAINTENANCE AND SOFTWARE CHANGES As applied to software, the term maintenance does not mean the same as when applied to hardware. Software maintenance includes corrective, perfective, and adaptive maintenance but does not include preventive maintenance actions or software component replacement (1). The software validation guidance states, changes made to correct errors and faults in the software are corrective maintenance. Changes made to the software to improve the performance, maintainability, or other attributes of the software system are perfective maintenance. Software changes to make the software system usable in a changed environment are adaptive maintenance. When changes are made to a software system, 19

20 John E. Lincoln either during initial development or during post-release maintenance, sufficient regression analysis and testing should be conducted to demonstrate that portions of the software not involved in the change were not adversely impacted. This is in addition to testing that evaluates the correctness of the implemented change. The specific validation effort necessary for each software change is determined by the type of change, the development products affected, and the impact of those products on the operation of the software (1). Whether production or quality system software is developed in-house by the device manufacturer, developed by a contractor, or purchased off-the-shelf, it should be developed using the basic principles outlined elsewhere in this guidance. The device manufacturer has latitude and flexibility in defining how validation of that software will be accomplished. The software developer defines a lifecycle model. Validation is typically supported by verifications of the outputs from each stage of that software development lifecycle and checking for proper operation of the finished software in the device manufacturer s intended use environment. REGRESSION TESTING Regression analysis and testing are employed to provide assurance that a change has not created problems elsewhere in the software product. Regression analysis is the determination of the impact of a change based on review of the relevant documentation (e.g., software requirements specification, software design specification, source code, test plans, test cases, and test scripts) in order to identify the necessary regression tests to be run. Regression testing is the rerunning of test cases that a program has previously executed correctly and comparing the current result to the previous result in order to detect unintended effects of a software change. Regression analysis and regression testing should also be employed when using integration methods to build a software product to ensure that newly integrated modules do not adversely impact the operation of previously integrated modules. HOW MUCH VALIDATION EVIDENCE IS NEEDED? The level of validation effort should be commensurate with the risk posed by the product or automated operation and the lifecycle stage of the software. In addition to risk, other factors influence the level of validation testing. These include complexity of the product or process software, the degree to which the user or device manufacturer is dependent upon that product or automated process to produce a safe and effective outcome, and its use and reliability history up to that point. The company then determines the nature and extent of testing needed as part of the next validation effort. Documented requirements and product risk analysis (tied to the end user of the resultant device) of the product or process help to define the scope of the evidence needed to show that the software is validated for its intended use. These decisions should be recorded in the software documentation, preferably tied to specific product risk document sections, paragraphs, or line items. BENEFITS OF SOFTWARE LIFECYCLE ACTIVITIES Software lifecycle activities and their validation are a major part of the development and maintenance of software throughout its lifecycle. They play a critical role in assuring the quality of device software and software automated operations throughout the lifecycle. Software validation can increase the useability and reliability of the device, resulting in decreased failure rates, fewer recalls and corrective actions, less risk to patients and users, and reduced liability to device manufacturers. Well-documented software validation can also reduce long-term costs by making it easier and less costly to reliably modify software and revalidate software changes. Software maintenance can represent a large percentage of the total cost of software over its entire lifecycle. An established comprehensive software validation process helps to reduce the long-term cost of software by reducing the cost of validation for each subsequent release of the software. REFERENCES 1. FDA, Guidance Document, General Principles of Software Validation; Final Guidance for Industry and FDA Staff, January 11, FDA, 21 CFR Part 11, Electronic Records; Electronic Signatures Final Rule, 62 Federal Register 13430, March 20, FDA, Glossary of Computerized System and Software 20