Risk-Based Validation of Commercial Off-the-Shelf Computer Systems

Transcription

1 Risk-Based Validation of Commercial Off-the-Shelf Computer Systems Published by Advanstar Communications in Journal of Validation Technology May 2005, Vol. 11, No. 3 Supplied by (*) Global on-line resource for validation and compliance (*) With permission of Advanstar Communication Copy right by Advanstar Communication. Additional copies and other publications on validation and compliance can be ordered from

2 Risk-Based Validation of Commercial Off-the-Shelf Computer Systems BY LUDWIG HUBER, PH.D. ABSTRACT This article describes how to adopt risk-based approaches for the validation of commercial computer systems used in the regulated pharmaceutical industry. This paper will help to guide readers through a logical, riskbased approach for computer system validation. It offers recommendations on how to define risks for different system and validation tasks and for risk categories along the entire life of a computer system. The scope of this paper is limited to Commercial Off-the-Shelf (COTS) systems and does not include risks typically involved during software development. The article contains two parts. Part one deals with risk assessment, in which we discuss approaches to categorizing computer systems into high, medium, and low-risk levels. (These levels serve as an example. Any ranking of levels of risk that is relevant to the product and the manufacturer may be substituted. The thought process of ranking is the same.) Part two offers recommendations for validation steps for the different categories as defined in part one. INTRODUCTION Computer systems are widely used in pharmaceutical industry for instrument control and data evaluation in laboratories and manufacturing. They are also widely used for data transmission, documentation, and archiving. When used in regulated environments they should be formally validated. The main compliance-related purpose of their validation is to ensure accuracy and integrity of data created, modified, maintained, archived, retrieved, or transmitted by the computer system. In addition, a computer validation, typically, is pre-requisite to obtaining reliable system operation and the highest system uptime, which are business requirements of the industry. Depending on the complexity and functionality, validation of computer systems can be a huge task. The efforts for validation should be balanced against the benefits, which means the amount of work should be in line with the problems that can occur if the system is not fully validated. The mechanism to balance benefits against investments is risk assessment in which we define the extent of validation according to the risk a specific computer can have on data integrity, and ultimately, product quality and safety. The risk-based approach should enhance industry s ability to focus on identifying and controlling critical functions that affect product quality and data integrity. Industry task forces have recommended risk-based approaches for validation for a long time. For example, Good Automated Manufacturing Practice (GAMP) has a chapter in its Guide for Validation of Automated Systems in Pharmaceutical Manufacture. 1 Also, the United States (U.S.) Food and Drug Administration (FDA) has recognized the importance of risk-based compliance. This became most obvious when the FDA announced its science and riskbased approaches as part of the 21st Century drug Good Manufacturing Practice (GMP) initiative in We will focus our attention and resources on the areas of greatest risk with the goal of encouraging innovation that maximizes the public health protection, said FDA Commissioner Marc MacClellan at an FDA/Industry training session. 8 David Horowitz added, there are two elements to a risk-based approach to inspections: We need to go to the right places and we need to look at the right things. 8 One reason for this risk-based approach is FDA s limited resources to inspect all manufacturing sites every two years. 182 Journal of Validation Technology

3 We have over 6,000 domestic drug facilities and the number of GMP inspections that we have been able to inspect has declined by about two thirds in the last 20 years. So we can't take the chance that we are squandering our limited resources on lower risk facilities. That would prevent us from doing a minimum level of scrutiny and oversight and working with the higher risk facilities, Horowitz said. 8 In the meantime, FDA has begun to allocate its resources based on risk. For example, beginning in the fall of 2004, FDA began using a risk-based approach for prioritizing domestic manufacturing site inspections for certain human pharmaceuticals. This approach should help the Agency predict where its inspections are likely to achieve the greatest public health impact. 2 The FDA is not only taking advantage of the risk-based approaches, but also encourages the industry to do so, for instance, in software and computer validation. The industry guidance on General Principles of Software Validation states: The selection of validation activities, tasks, and work items should be commensurate with the complexity of the software design and the risk associated with the use of the software for the specified intended use. 3 The same guide has also specific recommendations on what is expected for lower risk systems: For lower risk devices, only baseline validation activities may be conducted. As the risk increases, additional validation activities should be added to cover the additional risk. The FDA Part 11 Guidance on Scope and Application states: We recommend that you base your approach (to implement Part 11 controls, e.g., validation) on a justified and documented risk assessment and a determination of the potential of the system to affect product quality and safety, and record integrity. The most specific advice for risk-based compliance of computer systems came from the Pharmaceutical Inspection Convention s, Good Practices for Computerized Systems Used in Regulated Environments. 5 It has several recommendations related to risks: For critical GXP applications, it is essential for the regulated user to define a requirement specification prior to selection and to carry out a properly documented risk analysis for the various system options. This risk-based approach is one way for a firm to demonstrate that it has applied a controlled methodology, to determine the degree of assurance that a computerised system is fit for its intended purpose. The inspector will consider the potential risks, from the automated system to product/material quality or data integrity, as identified and documented by the regulated user, in order to assess the fitness for purpose of the particular system(s). The business/gxp criticality and risks relating to the application will determine the nature and extent of any assessment of suppliers and software products. Basically, this means the FDA and other agencies expect a risk assessment for each computer system, otherwise full validation is required. Companies without justified risk assessments will not be able to defend their selected level of validation. The real value in a comprehensive risk-based validation approach is in doing exactly the right amount and detail of validation for each system. The principle is quite clearly illustrated in Figure 1. Costs for validation increase when going from no validation to 100% validation. Full validation for a COTS system would mean, for example, the testing of each function of the software under normal and high load, across and beyond the expected application range, and this for each possible system configuration. In addition, whenever the system is changed, may it be computer hardware, operating system, or application software, full revalidation would require that the same tests be rerun. In today s rapidly changing computer environment, this could possibly mean that the system would be used 100% for testing. At the same time that testing increases, the risk May 2005 Volume 11, Number 3 183

4 of unexpected system failure decreases, because errors found during testing can be corrected or work around solutions can be found and implemented. The optimum testing is, obviously, somewhere between zero and 100%. The range depends on the impact the software or system has on (drug) product quality. For example, a system used in early drug development stages will have a lower impact and require less validation than a system used in pharmaceutical quality control. In the past, companies frequently have applied the principles of such risk-based validation, but the rationale behind it was not documented and the approach was not implemented consistently within a company. The extent of validation depended more on individual validation professionals than on a structured rationale. As explained earlier, in new guidance, the FDA suggests that industry base the extent of its validations on a justified and documented risk assessment. Most confusing to the industry has been finding a structured way to prioritize risks. The FDA has been asked frequently to prepare a matrix of regulated processes indicating the level of risk associated with each. The FDA has made it very clear that this will not happen, because each situation is different. However, they have released criteria to be used in making these determinations. These are defined as: impact on product quality and patient safety. General advice came from FDA s John Murray when he answered questions concerning FDA s expectations at the Institute of Validation Technology (IVT) Computer System conference in May 2004: 15 Really, I don t recommend you do a detailed risk assessment on every record in the building. I think you need to set up a systematic way of doing it and you are going to put certain records in certain categories from the very beginning. If a record is used to release product and this record is incorrect and you release an unsafe product I would make that your highest category, direct impact to public health. Figure 1 Risk vs. Validation Costs Risk Risk Impact on Product Quality Low Optimum High Costs 0% Validation 100% Risk-based validation takes two steps: Define the risk category - for example - high, medium, and low, and define the extent of validation for each category according to guidelines as laid out by the company. One final comment before we start with risk-based approaches, the model proposed in this paper has two objectives: 1. Get started quickly to take immediate benefit of the risk-based approach. Start with a qualitative risk assessment based on experience with the same or similar systems and gain further experience for full risk management for later implementation. 2. Fulfill FDA requirement of basing the extent of validation for each level on justified and documented risk assessment. It is quite obvious that there are no generally accepted models to copy, and there is no universal solution. Each company must figure out the answers for itself because success really does depend on the unique situation of a company. The model suggested in this article is just one example for implementation. The FDA would allow many others. For example, this model suggests three risk categories: high, 184 Journal of Validation Technology

5 medium, and low. It also would be acceptable to have only two: high and low, or five and more. All models would be accepted as long as the approach is justified and documented. APPROACHES for RISK ASSESSMENT and MANAGEMENT The National Institute for Standards and Technology (NIST) has defined the term, risk, as: The probability that a particular threatsource will exercise (accidentally trigger or intentionally exploit) a particular information system vulnerability and the resulting impact if this should occur. 12 The types of risks a pharmaceutical company deals with include patient risk (safety and efficacy of drugs), regulatory risks [FDA 483 s, Warning Letters (WLs), product recalls, etc.], and financial risk due to, for example: inability to get products approved for marketing, inability to ship finished products, or consequences of unauthorized disclosure of trade secrets and private information. Risk management is the entire process from identifying and evaluating the risk to defining risk categories, and taking steps to reduce risk to acceptable levels. Risk assessment includes the first two parts: analysis and risk evaluation. There are a number of standard risk assessment techniques available and widely used in the industry. The most important ones include the Failure Mode and Effects Analysis (FMEA) approach, Fault Tree Analysis (FTA), and the application of Hazard Analysis and Critical Control Point (HACCP) methodology. All three methods have been described in brief by H. Mollah. 9 An approach widely used in medical device industry is based on the International Organization for Standards (ISO) While FMEA and FTA are based more on quantitative, statistical data, the ISO approach is more qualitative in nature. The concept is to determine risk factors based upon their likelihood and severity, the mitigation of those risks, and monitoring and updating the process as necessary. The model, as described by GAMP, 1 is similar but adds detectability as another criterion: the more likely the problem will be detected, the lower the risk. Labcompliance has developed an extensive risk management master plan using the concept as described in the ISO standard. 10 For the scope of this publication, we follow the approach as described in the ISO standard. The model presented in this paper is more qualitative than quantitative and is very much based on the experience of users, validation groups, and auditors either with the same or with similar systems. For the scope of this paper, we introduce readers to the concept of full risk management, but then only focus on risk assessment. However, bear in mind that some of the current validation tasks, such as vendor assessment and even testing, are already steps towards the mitigation of risks involving computer systems. RISK MANAGEMENT OVERVIEW Risk management of a commercial computer system starts when the system is specified and purchased, continues with installation and operation, and ends when the system is taken out of service and all critical data have been successfully migrated to a new system. The approach we take is to divide risk management into four phases as illustrated in Figure 2. The phases include risk analysis, risk evaluation and assessment, and on going evaluation and control. Risk Analysis Define computer system components and software functions. Identify potential hazards and harms using inputs from system specifications, system administrators, system users, and audit reports. Risk Evaluation and Assessment Define the severity, probability, and risk of each hazard, for example, by using past experience from the same or similar systems. Determine acceptable levels of risk and identify the hazards that would need mitigation to reach those levels. Identify and implement steps to mitigate risks. On-Going (re)evaluation and Control On an on-going basis, evaluate the system for new hazards and changes in risk levels. Adjust risk and mitigation strategy as necessary. These activities should follow a risk management plan and the results should be documented in a risk management report. May 2005 Volume 11, Number 3 185

6 Risk Analysis The first step in the risk management process is the risk analysis, sometimes called risk identification or Preliminary Hazard Analysis (PHA). The output of this phase is the input for risk evaluation. Inputs for risk analysis include: Specifications of equipment including hardware and software User experience with the same system already installed User experience with similar systems IT staff experience with the same or similar network equipment Experience with the vendor of the system Failure rates of the same or similar system (mean time between failures) and resulting system downtime Trends of failures Service records and trends Internal and external audit results Inputs, for example, can come from operators, the validation group, Information Technology (IT) administrators, or from Quality Assurance (QA) personnel as the result of findings from internal or external audits. The project manager collects input on potential hazards including possible harm. For consistent and complete documentation, forms should be used. The forms should have entry fields to include relevant data on the individual who made the entry, risk description, possible hazards and harms, probability of occurrence, and possible methods of mitigation. An example is shown in Figure 3. Occasional problems and harms with computer systems include, but are not limited to the following: Hard drive failure on local Personal Computers (PCs) or on the server computer can cause severe system downtime and loss of data. Loss of network connectivity due to hardware failure, for example, the network interface card, can cause system downtime. System overloads can cause slow-down of operations and system downtime. Inadequate vendor qualification or absent specifications on vendor support purchasing agreements can result in reduced uptime because of missing support - in the case of hardware, firmware, or software problems. Inadequate or absent documentation of installation can make it difficult to diagnose a problem. Inadequate or absent verification of security access functions can result in unauthorized access to the system. An insufficient or absent plan for system backup can result in data loss in case of system failure. Figure 2 Risk Management (with permission, see Reference 10) Risk Management Plan Risk Analysis Risk Evaluation Risk Mitigation/Control On-going Evaluation Identify the system Identify hazards and possible harms Estimate, justify and document risk level (probability and severity) Estimate costs of mitigation vs. non-mitigation Define and take actions for mitigation Monitor for new harms Monitor risk levels Update plan and take actions }Risk Assessment Documentation 186 Journal of Validation Technology

7 Poor or absent documentation of hardware and software changes can make it difficult to diagnose a problem. Inadequate quality assurance policies and procedures or inadequate reviews can lead to poor system quality. Risk Evaluation Process This phase is used to categorize and prioritize the risk from a business and compliance, or health risk standpoint. Data should be entered into a form with entry fields for risk descriptions, business (continuity) impact, product quality, safety, and compliance impact, as well as probability of occurrence. An example is shown in Figure 4 and the various impacts are described below. Impact on Business Continuity This is related to a company s ability to market a new product and its reliance on system uptime for continuous shipment of product. Evaluating these issues will answer these questions: in currency, how big would be the losses due to delays of new product approval and shipment stoppages? Impact on Product Quality The question here is whether the system has an impact on product quality. This question asks whether the system impacts the identity, strength, safety or efficacy of a drug. A direct impact on product quality means that any failure cannot be corrected before a new drug is approved for marketing or before a batch is released for shipment. For a high-risk classification, the probability of detecting the problem would be low or zero. An example is an analysis system used in quality control where analysis results are used as criteria for the release of product. Impact on Human Health and Safety Includes consumer safety and environmental hazards. An example of high severity would include circumstances whereby poor product quality could cause adverse effect to the health of patients or users. Note: Because an impact on health and safety can only occur when there is also an impact on product quality, we combine both factors. Impact on Compliance This is related to the risk of failing regulatory inspections and receiving single or multiple WLs or inspectional observation reports. A typical compliance issue is the insufficient integrity of regulated data. There are other indirect affects wherein the health of a patient or a worker is affected, such as claims against the company, product recalls, a negative Figure 3 Template: Identification of Risks Name/Organization: System ID: Location: Date: Risk Description Impact of Possible Harm Probability of Occurrence Method of Mitigation May 2005 Volume 11, Number 3 187

8 reputation for the company, etc. Information from this category will be used to calculate an overall risk factor. In our example, the risk categories are converted into numeric values such that: high=3, medium=2, and low=1. (See Figure 5.) Risk factors are calculated using the following formula: (Business Impact + Safety + Compliance Impact) x Probability of Occurrence = Risk Factor FACTORS CONTRIBUTING TO RISK High-Risk Factors Examples of factors contributing to high-risk levels include those related to product quality and health and safety, business continuity, and regulatory compliance. Product Quality and Health and Safety Systems used to monitor, control, or supervise a drug manufacturing or packaging process Systems used in a production environment for testing, release, labeling, or distribution of products Figure 4 Template: Risk Evaluation Name/Organization: Infrastructure ID: Location: Date: Business Continuity Impact Product Quality and Safety Compliance and Data Integrity Probability of Occurrence High (3) = >$$$ per day No replacement system High (3) Failure will most likely have an adverse effect on product quality and consumer safety. High (3) Loss or change of GXP critical data,* resulting in product recall High (3) Medium (2) = > $$$ per day Replacement after one day Medium (3) Failure is not likely to have an adverse effect on product quality or on consumer safety. Medium (2) FDA Warning Letter likely Medium (2) Low (1) = > $$$$ per day Low (3) Failure will not affect product quality or data integrity Low (1) FDA 483 inspectional observation likely Low (1) *GXP critical data are records required by GXP regulations. The assignment of risk levels, the cost per day that will identify the level of risk, as well as the specific recommendations on calculating the probability of occurrence, should be defined and added for individual projects. 188 Journal of Validation Technology

9 Users interact manually with the system and data having the ability to manipulate data Failure of the system can have direct impact on product quality No or low probability that the problem will be detected or can be corrected Product quality problems may lead to death or serious and permanent injury Business Continuity System must run 24 hours a day, 7 days a week Highly complex hardware, software, and system configuration Highly customized Unskilled operators No work around solutions Vendor unrecognized in the pharmaceutical industry; no support from vendor, e.g., no documented evidence on validation during development, or no phone or on-site support in case of problems Compliance Used for GXP regulated applications System failure can impact data integrity or can cause loss of data Low-Risk Factors Factors contributing to low severity risk levels include those related to product quality, health and safety, business continuity, regulatory compliance, and probability. Product Quality System is used in early product development stage System is fully automated and relies on well-validated processes High probability that problem will be detected and can be corrected Health and Safety System failures or lack of data integrity do not have any impact on human health Figure 5 Template to Determine the Overall Risk Factor Risk Description Business Continuity Impact Product Quality and Safety Compliance and Integrity Probability of Occurence Risk Factor May 2005 Volume 11, Number 3 189

10 Business Continuity Used occasionally Highly skilled operators Widely used commercial systems No customization Work around solutions available Full support from recognized vendor, e.g., documented evidence on validation during development, local language phone support or on-site support in case of problems Compliance Not used in regulated applications Failure of the system does not have an impact on data integrity and cannot cause loss of data Probability Probability should answer the questions: what is the likelihood that the system will fail, generate wrong data, or that data are lost? Probability should be expressed in occurrence within a set time period. We recommend using five categories: Frequent (e.g.: once every month) Probable (e.g.: once in one to three months) Improbable (e.g.: once in three to twelve months) Occasional (e.g.: once in one to three years) Impossible We use past experiences from the same or similar systems to estimate the probability. Importance of a Risk Management Master Plan The most significant task during the risk assessment process is to define criteria for criticality, which determines the final risk level. For example, this question frequently comes up: what if an inspector questions my decision? There are no absolute measures, so a dispute may occur. Discussing this question today is similar to an industry discussion of 10 to 15 years ago concerning computer validation when the frequently asked question was how much validation is enough? Answering this question about validation was nicely solved with the development of the Validation Master Plan (VMP). Companies developed such master plans on a fairly high level to guide validation specialists through the validation process by explaining the procedure for easy understanding, offering templates for convenient implementation, and giving examples on what to validate for different systems. In the meantime, such VMPs have become a legal requirement in Europe through Annex 15 of the European GMP directive. 16 The U.S. FDA may not ask for a VMP; the inquiry may be for the company s approach to validation. The VMP, and the examples it contains, has become the perfect document to help answer any question about the level of validation. An equivalent document in the area of risk assessment is a risk management master plan. Such a document should be developed at a fairly high level within the company. It should describe the company approach to risk management and assessment and should include templates for risk identification, evaluation, mitigation, and control. It also should include criteria and examples for severity and probability. The master plan can be used to derive risk management plans for individual projects. The main advantages are increased efficiency, and, even more importantly, consistent implementation. A risk management master plan should also include examples of factors that impact risk categories. This is important to ensure a consistent approach in the company risk assessment. An example with some recommendations is shown in Figure 4. Examples Examples are quite useful for getting an idea of what type of systems fall into the different categories. Another type of question that is frequently posed is whether, for example, a laboratory management system or a documentation system falls in the high, medium, or low-risk category. Sometimes even systems from specific vendors are mentioned. This is the wrong question. The risk is not dependent mainly on the system but more on the records created, evaluated, transmitted, or archived by the system. A Laboratory Information Management System (LIMS) in a non-regulated research department is not a high-risk system, at least not from a compliance view. On the other hand, a LIMS in a pharmaceutical quality control laboratory is most likely a high-risk system because the records have a high impact on product quality. Both the International Society for Pharmaceutical Engineering (ISPE) and the Pharmaceutical Research and Manufacturing Association (PhRMA) have given examples for what may qualify as high-risk. The PhRMA wrote a letter to 190 Journal of Validation Technology

11 the FDA on Nov 29, 2001 related to the Proposed FDA Guidance on the Scope and Implementation of 21 Code of Federal Regulations (CFR) Part 11. The letter included a ranking of five systems related to their risk on product quality. Those with the highest risk were manufacturing batch records and manufacturing LIMS and Quality Assurance (QA) systems. 13 The ISPE wrote a White Paper on the Risk-Based Approach to 21 CFR Part 11 with the recommendation that the focus of efforts should be on records that have a high impact, i.e.: those records upon which quality decisions are based. Examples of high impact records include batch records and laboratory test results. 14 Examples of records with low impact include environmental monitoring records not affecting product quality, training records, and internal computerized system information such as setup and configuration parameters. Other examples are planning documents and Standard Operating Procedures (SOPs) for non-critical operations. GAMP has published a Good Practices Guide: A Risk Based Approach to Compliant Electronic Records. 16 This document illustrates examples of records that have high, medium, and low impact on risk. In general, systems fall into the high-risk category when they have a direct impact on product quality and patient safety. Examples are systems used in pharmaceutical manufacturing and quality control such as electronic batch record systems, analytical control systems, also document management systems and data bases with high-risk records. For example, wrong analytical test results that are used as a criterion to release a batch are highly critical, because there is no further testing and the product is released to the market immediately. An example of a system with high impact on patient safety is a distribution record system. If a product must be recalled because adverse effects on patients have been identified and some of the distribution records are lost, incorrect, etc., the product cannot be completely removed from the market, thereby having a high impact on patients. Examples of systems in the medium-risk category include systems that are used to qualify and monitor the systems defined as high-risk. These would also include configuration management software. Examples of low-risk systems include word processing systems that are used, for example, to generate validation records. The reasons for relegating these systems to the low-risk category include the relatively low likelihood that they would have errors, the likelihood that errors would be detected by proofreading, and, in this case, the likelihood that such errors would have no direct impact on product quality or patient safety. Validation Tasks Once the risk level is identified, validation steps can be defined. Risk level information is used for considerations such as: In what detail do we specify the system? For example, for a low-risk system, we only prepare a high-level system description and for high-risk systems we develop detailed system requirement specifications. How extensively do we test the computer system? For example, high-risk systems will be tested under normal AND high load conditions. Test cases should be linked to the requirement specifications. How much equipment redundancy do we need? For example, for high-risk systems we should have validated, redundant hardware for all components. For medium-risk systems, redundancy of the most critical components is enough, and for low-risk systems, there is no need for redundancy. How frequently must we back-up data generated by the system? While a daily back-up is a must for high-risk systems, weekly incremental back-up is sufficient for low-risk systems. What type of vendor assessment is required? For example, high-risk systems will require vendor audits while, for medium and low-risk systems, an audit checklist and documented experience from the vendor should be enough. What requirements of Part 11 should be implemented in the computer system? For example, high-risk systems computer generated audit trails should be implemented, while for low-risk systems, a paper-based, manual audit trail is enough. Validation tasks should be defined for each phase starting from planning through specification settings, vendor qualification, installation, testing, and on-going system control. The tasks should be consistent within an organization for each risk category. They should be well documented and be included either in the risk management master plan or in the validation master plan. Figure 6 summarizes examples with validation activities for each validation phase and task. May 2005 Volume 11, Number 3 191

12 Figure 6 Examples for Validation Tasks Validation Phase High Medium Low Planning Specifications Vendor Assessment Installation Functional Testing Ongoing Control Change Control Computer System Audits Detailed validation plan with all activities, deliverables, owners, and timetables Document all Uniquely number all Define critical vs. non-critical requirements Direct vendor audit or third party audit Verify correct software installation Document system and all components and configurations Document software versions Test all critical standard functions Test over entire applications range. Include high load and stress tests. Test correct functioning of user specific configurations. Link tests to requirements Regular virus check Regular revalidation Regular regression testing All changes approved by system owner and QA Regular audit of system and subsystems High-level plan with key activities Document all requirements Uniquely number all requirements Review of vendor documentation Mail audit Document system, all components, and configurations Document software versions Test all critical standard functions under normal and high loads Test correct functioning of user specific configurations Regular virus check Regular regression testing All changes approved by system owner "For cause" audits in case of problems No specific project validation plan Follow template for low-risk applications No specifications Refer to vendor documentation Document vendor Document software and version No testing Regular virus check All changes documented by user No audits Contingency planning Back-up Regular review of the audit plan Redundancy for all hardware Daily incremental backup Weekly complete back-up Redundancy for key hardware, e.g., server Daily incremental back-up No redundancy Weekly incremental back-up 192 Journal of Validation Technology

13 For some validation tasks other factors should be considered besides the impact of the system on product quality. One such factor is vendor qualification. The question of how much to invest depends on two factors: product risk and vendor risk. Factors that impact vendor risk include: Experience with the vendor (software quality, responsiveness and quality of support) Size of the company Company history Represented and recognized in industry, e.g.: Bio/Pharma Expertise with (FDA) regulations Future outlook How likely is the company to stay in business? Figure 6 is recommended as a starting point for a commercial, networked Off The Shelf (OTS) system with user specific configurations, i.e., network configurations. Such an automated system would fall into category four, as defined by GAMP. 1 This table can be extended to a third dimension to include systems or software that have been developed for a specific user (GAMP category five) and to include systems that do not require any user specific configurations (GAMP category three). GAMP categories indicate the level of system customization. The extent of validation is lower for GAMP category three and higher for systems in GAMP category five. The principle is shown for early validation phases in Figure 7. Of course, to generate such information is more time consuming and only makes sense if several systems in GAMP category three or five should be validated. Conclusion Risk analysis and evaluation of software and computer systems is a good tool to optimize validation costs by focusing on systems with high impact on both the business and compliance. Substantial cost savings are possible for medium and low-risk systems. Validation activities of a low-risk system can be limited to documenting which systems have been used. The risk is less dependent on the type of system than on the type of records generated by the system. For example, a LIMS system used in a research envi- Figure 7 Validation Tasks for Risk and GAMP Categories (The figure shows only the first four validation phases). Complexity Level of Customization GAMP Category 5 GAMP Category 4 GAMP Category 3 Increasing Controls Activity Low Risk Medium Risk High Risk Planning Specifications Vendor Assessment Installation Increasing Controls May 2005 Volume 11, Number 3 193

14 ronment has a lower compliance risk than the same system used in pharmaceutical quality control. Regulatory agencies require companies to base the extent of validation they complete on a justified and documented risk assessment. To do this efficiently, we recommend the following steps: 1. Develop a risk management master plan. This describes the company s approach to risk assessment and has templates and examples for easy and consistent implementation. This plan should also include validation tasks for each risk category. 2. Develop a risk management project plan for each computer system validation project. Use the risk management master plan approach as a source to define steps, owners, and deliverables. 3. Identify risks, possible hazards and harms and define the risk category, for example: high, medium, and low. This should be based on likelihood and severity. To estimate the severity, look at the records handled by the system and at their impact on product quality and consumer safety. 4. Determine validation tasks for each lifecycle phase. Use the approach, templates, and examples from the risk management master plan 5. Develop a risk management plan with a sound justification and the documentation of your results. For the long term, we recommend that risk assessment be extended to full risk management with an action plan for risk mitigation and on-going review and control. ABOUT THE AUTHOR Ludwig Huber, Ph.D., is Compliance Program Manager at Agilent Technologies. He is the editor of the global on-line resource for validation and compliance issues for laboratories. He is the author of the books, Validation and Qualification in Analytical Laboratories, and Validation of Computerized Analytical and Networked Systems, published by Interpharm Press. Dr Huber was a team and review member of PDA s task forces 21 CFR Part 11 and Validation of Laboratory Data Acquisition Systems, and of the GAMP special interest group on Laboratory Equipment. He is currently on the Scientific Advisory boards of the Academy of European Compliance of Pharmaceutical Technology and of the Institute of Validation Technology s Journal of GXP Compliance. He may be reached by at ludwig_huber@agilent.com or at REFERENCES 1. GAMP Good Automated Manufacturing Practice, Guide for Validation of Automated Systems in Pharmaceutical Manufacture, Version 3, March 1998, Version 4, December Pharmaceutical cgmps for the 21st Century: A Risk-Based Approach, and FDA Issues Final Report on its 21st Century Initiative on the Regulation of Pharmaceutical Manufacturing Article Acronym Listing CFR Code of Federal Regulations COTS Commercial Off The Shelf FDA Food and Drug Administration FMEA Failure Mode and Effects Analysis FTA Fault Tree Analysis GAMP Good Automated Manufacturing Practice GMP Good Manufacturing Practice GXP Good Manufacturing, Clinical, and Laboratory Practice HACCP Hazard Analysis and Critical Control Point ISO International Organization for Standardization ISPE International Society for Pharmaceutical Engineering IT Information Technology IVT Institute of Validation Technology LIMS Laboratory Information Management System NIST National Institute for Standards and Technology OTS Off The Shelf PC Personal Computer PHA Preliminary Hazard Analysis PhRMA Pharmaceutical Research and Manufacturing Association QA Quality Assurance SOP Standard Operating Procedure U.S. United States VMP Validation Master Plan 194 Journal of Validation Technology

15 3. United States Food and Drug Administration, General Principles of Software Validation: Final Guidance for Industry and FDA Staff, United States Food and Drug Administration, Industry Guidance: 21 CFR Part 11Electronic Records, Electronic Signatures, Scope and Applications. 5. Pharmaceutical Inspection Convention: Good Practices for Computerized Systems Used in Regulated Environments (5). 6. United States Food and Drug Administration, Code of Federal Regulations, Title 21, Food and Drugs, Part 11 Electronic Records; Electronic Signatures; Final Rule; Federal Register 62 (54), Pharmaceutical Inspection Convention, January 2002, (DRAFT), Good Practices for Computerised Systems in Regulated GXP Environments, DIA/FDA Industry Training Session, May H. Mollah, Risk Analysis and Process Validation, Bio- Process International, Vol. 2, No 9, Oct ISO 14971:2000, Medical Devices Application of Risk Management to Medical Devices, Geneva, Switzerland, Labcompliance, Risk Management Master Plan, National Institute of Information Technology. Risk Management Guide for Information Technology System, NIST Special Publication PhRMA, Letter to the FDA, Related to Proposed FDA Guidance on the Scope and Implementation of 21 CFR Part 11, on October 29, ISPE, White Paper, Risk-Based Approach to 21 CFR Part 11, John Murray at the Institute of Validation Technology Computer System Validation conference, May Qualification and Validation, Annex 15 to the EU Guide to Good Manufacturing Practice, GAMP Good Automated Manufacturing Practice, Good Practice Guide: A Risk-Based Approach to Compliant Electronic Records and Signatures, ISPE, Tampa, FL, February 2005 The Institute of Validation Technology announces the availability of its Special Edition: Utilities Qualification (Code SEUQ), containing over 150 pages of practical advice and vital information to help you qualify your facility s utility systems. These articles were provided to IVT by leading validation professionals, and are now compiled in this single, easy-to-use resource. Order Your Copy Today! Only $199 Order Online and get a 3% Discount! For more information, call: PI2405 May 2005 Volume 11, Number 3 195

16 Standard Operating Procedure Risk-Based Validation of Computer Systems This is an example of a Standard Operating Procedure. It is a proposal and starting point only. The type and extent of documentation depends on the process environment. The proposed documentation should be adapted accordingly and should be based on individual risk assessments. There is no guarantee that this document will pass a regulatory inspection. Company Name: Controls: Superseded Document N/A, new Reason for Revision N/A Effective Date Jan 1, 2004 Signatures: Author I indicate that I have authored or updated this SOP according to applicable business requirements and our company procedure: Preparing and Updating Standard Operating Procedures. Name: Signature: Date: Approver I indicate that I have reviewed this SOP, and find it meets all applicable business requirements and that it reflects the procedure described. I approve it for use. Name: Signature: Date: Reviewer I indicate that I have reviewed this SOP and find that it meets all applicable quality requirements and company standards. I approve it for use. Name: Signature: Date: 196 Journal of Validation Technology

17 1. PURPOSE Software and computer systems should be validated for compliance and business reasons. The extent of validation depends on the risk the system can have on product quality and safety and on the complexity. This SOP gives guidelines on what the extent of validation should be for risk categories as defined by the SOP in Reference SCOPE Risk-based validation of software, computer systems, and other computer related GXP requirements such as limited access, system audits, and change control. 3. GLOSSARY/DEFINITIONS Item GAMP Explanation Good Automated Manufacturing Practice (Forum). The GAMP Forum exists to promote the understanding of the regulation and use of computer and control systems within the pharmaceutical manufacturing industry GAMP Category 3 GAMP Category 4 GAMP Category 5 Standard software package. All applications problems are solved with standard functions. However, typically not all available functions are exercised by the user s application. Configurable software package. Provides standard interfaces and functions that enable configuration of user specific applications. Custom software package. Developed to meet specific needs of an application. Custom software may be a complete system or add on to a standard package. Custom software may be developed and supported in-house or by an external supplier. Standard Function Function that comes with software GAMP Category 3. Critical Requirement Requirement that the user determines to be critical for the effective use of the system. Note: For other definitions, see May 2005 Volume 11, Number 3 197

18 4. REFERENCE DOCUMENTS 4.1. GAMP 4 Guide: "Validation of Automated Systems," ISPE, Brussels, 2001 (order from SOP S-134 "Risk Assessment for Systems Used in GXP Environments." Available through "Risk Management Master Plan." Available through 5. RESPONSIBILITIES 5.1. System Owner Owns the process to define and document extent of validation for a specific system Drafts documentation Operation s Manager Reviews and approves documentation Quality Assurance Advises on regulations and guidelines related to GXP and 21 CFR Part Reviews documentation for compliance with internal policies and guidelines Approves documentation. 6. FREQUENCY OF USE 6.1. Initially whenever equipment is qualified and software and computer systems are validated After system updates or other changes and the change indicates that the extent of qualification or validation may need to be changed Whenever system reviews indicate that the extent of qualification or validation may need to be changed. 198 Journal of Validation Technology

19 7. PROCEDURE 7.1. System owner defines validation steps for life cycle phases and tasks using Tables to as guidelines Planning Validation Steps Planning System GAMP Cat. 3 GAMP Cat. 4 GAMP Cat. 5 High Risk Detailed validation plan with all activities, deliverables, owners, and timetables. Detailed validation plan with all activities, deliverables, owners, and timetables. Detailed validation plan with all activities, deliverables, owners, and timetables. Medium Risk High-level plan with key activities. High-level plan with key activities. High-level plan with key activities. Low Risk No specific plan. High-level plan with key activities. High-level plan with key activities Setting Specifications Validation Steps Setting Specifications System GAMP Cat. 3 GAMP Cat. 4 GAMP Cat. 5 High Risk Medium Risk Low Risk Document all Uniquely number all Define critical vs. non-critical. Document all High-level system descriptions. Document all Uniquely number all Define critical vs. non-critical. Document all Uniquely number all Define critical vs. non-critical. Define and document all non-standard Document all Uniquely number all Define critical vs. non-critical. Document all Uniquely number all Define critical vs. non-critical. Define and document all non-standard May 2005 Volume 11, Number 3 199

20 Vendor Assessment Validation Steps Vendor Assessment System GAMP Cat. 3 GAMP Cat. 4 GAMP Cat. 5 High Risk Review of vendor documentation. Vendor audit. Vendor audit. Medium Risk Document experience with vendor and system. Review of vendor documentation. Vendor audit. Low Risk None. None. None Installation Validation Steps Installation System GAMP Cat. 3 GAMP Cat. 4 GAMP Cat. 5 High Risk Verify correct software installation. Document system and all components and configurations. Document software versions. Verify correct software installation. Document system and all components and configurations. Document software versions. Verify correct software installation. Document system and all components and configurations. Document software versions. Medium Risk Document system and all components and configurations. Document software versions. Document system and all components and configurations. Document software versions. Document system and all components and configurations. Document software versions. Low Risk Document system and all components and configurations. Document software versions. Document system and all components and configurations. Document software versions. Document system and all components and configurations. Document software versions. 200 Journal of Validation Technology

21 Functional Testing Validation Steps Functional Testing System GAMP Cat. 3 GAMP Cat. 4 GAMP Cat. 5 High Risk Test critical functions. Link tests to Test critical standard functions. Test all nonstandard functions. Link tests to Test all functions. Link tests to r e q u i r e m e n t s. Medium Risk Test critical functions. Test all critical standard and nonstandard functions. Link tests to T e s t a l l fu n c t i o n s. Link tests to r e qu i r e m e n t s. Low Risk No testing. Test critical nonstandard functions. Test critical functions Ongoing Maintenance and Performance Control Validation Steps Ongoing Control System GAMP Cat. 3 GAMP Cat. 4 GAMP Cat. 5 High Risk Regular virus check. Regular regression testing. Regular virus check. Regular regression testing. Regular virus check. Regular regression testing. Medium Risk Regular virus check. Regular virus check. Regular regression testing. Regular virus check. Regular regression testing. Low Risk Regular virus check. Regular virus check. Regular virus check. May 2005 Volume 11, Number 3 201