Managing quality in clinical trials is the focus of daily activities that,

This article describes a targeted review of a quality management system that provides the organization with the inherent ability to identify, analyze, and address possible performance issues through the appropriate use of metrics. Liz Wool, RN, BSN, CCRA, CMT Intertwining Quality Systems with Metrics to Improve Trial Quality Managing quality in clinical trials is the focus of daily activities that, per the Clinical Trials Transformation Initiative (CTTI), substantiate the clinical research community s ability to effectively and efficiently answer the intended question about the benefits and risks of a medical product (therapeutic or diagnostic) or procedure while assuring protection of human subjects. 1 At the Drug Information Association s 2011 Annual Meeting, a presenter from the European Medicines Agency (EMA) provided a description of quality as sufficient to support the decision-making process on medicines throughout the clinical development and postmarketing authorization. 2 With the advent of increasing protocol complexities, technological capabilities, and globalization of research, a prospective, systematic, and methodical approach to ensuring quality in clinical trials is needed and is being advocated by regulators. Additionally, both the Food and Drug Administration (FDA) and EMA are collaborating with all stakeholders in clinical research to define and describe the elements of quality by design (QbD) in the context of clinical trial conduct. QbD incorporates the elements of a quality management system with benchmarks to the International Conference on Harmonization (ICH) in the ICH Q7 through Q10 documents and the International Organization for Standardization s ISO 9000 standards for quality management systems. In May 2012, the EMA hosted a workshop that focused on a reflection paper by the agency s Good Clinical Practice (GCP) International Working Group about risk-based quality management in clinical trials. The group requested input from various stakeholders (including ACRP) on what the QbD elements, context, and framework are for clinical trials. Similarly, the FDA and CTTI are launching the QbD workstream in 2012. As clinical researchers in the 21st century, stating that we have standard operating procedures (SOPs) and training is not defining a quality management system. This article describes a targeted review of a quality management system that, when adhered to in its entirety, provides the organization with steps for defining, planning, monitoring, measuring, and continuously improving the quality of its work, leading to the inherent ability to identify, analyze, and address possible performance issues through the appropriate use of metrics. Performance Metrics in Clinical trials Quality System Rather than focus solely on GCP compliance, SOPs, processes, forms, and training, there needs to be a renewed focus on building quality within an organization that inherently possesses the culture of quality. 3 A quality x 29

management system sets out the standards to be achieved and the method to meet them. The system should define what people, actions, and documents should be employed to conduct the work in a consistent manner, leaving evidence of what has happened. It may include manuals, handbooks, procedures, policies, records, and templates. 4 In 1946, the International Organization for Standardization (ISO) was founded, and in 1987, it published the first ISO 9000 standard for quality management systems. 5 Many people believe that ISO 9000 focuses only on manufacturing of products; however, ISO 9001:2008 was written such that small businesses (e.g., consulting firms) can implement ISO 9000 for their organizations. The ISO definition for quality states that the quality of something can be determined by comparing a set of inherent characteristics with a set of requirements. With this in mind, this article will discuss the quality management system principles espoused in ISO 9000-9001, extrapolating its use in the global clinical research arena. Additional terms that organizations may use to describe their quality management system include clinical quality system, integrated quality management, quality management, or total quality management. From a practical standpoint, it is important to understand how the elements and components of an organization s quality management system relate to this article s description of a quality management system in order to perform a comprehensive gap analysis. Due to space limitations, alternative theories and methods will not be discussed here. Kleppinger and Ball, in their article Building Quality into Clinical Trials with the Use of a Quality Systems Approach, describe the utility and application of ISO 9000-9001 quality management systems principles for clinical trial planning, execution, ongoing monitoring, and continuous improvement during the clinical trial lifecycle. 6 The authors assert that, even though a quality system does not impose something totally new on clinical research, a systematic approach will produce a more reliable and useful end product that is, high- quality data obtained without compromising the protection of human subjects rights and welfare. After implementing a quality management system, assessing, monitoring, and measuring how well the organization is performing to the established standards and methods is required. Using visual inspection and confirmation, document review, data analytics and review, and metrics, the organization implements the foundational cornerstones for assessing its performance. Elements of a Quality System The overarching framework for a quality management system is illustrated in Figure 1, which visualizes the critical plan, do, check, act approach of a committed organization to quality through the development, implementation, and maintenance of a quality management system. Figure 1 Framework for a Quality System Responsibility, Review of QMS Issue Escalation Metrics System Corrective and Preventive Action Program Performance Dashboards GCP Quality Assurance Unit, Annual Audit Plan Risk Quality System (QMS) Framework Process Improvement A quality management system provides the prospective, systematic, methodical, scientifically based framework to plan, manage, monitor, and measure the quality of the organization and its performance throughout the clinical trial and product development lifecycles. Specifically, the quality management system establishes the standards under which work will be performed and how the organization and personnel perform and document their assigned clinical trial activities, duties, and functions. An additional illustration is noted in Figure 2, which outlines the phases of product development for clinical research benchmarking to similar illustrations for the ICH Q10 Pharmaceutical Quality System Guideline. 7 This network system of interrelated processes provides uniformity and consistency for people and actions, describing the work performed and how the work is documented, as outlined in Figure 3. In the clinical trials context, this Quality Policy, Quality Manual Deviation Procedural Documents, Document Control Process Monitoring and Analyses Change Control Training 30 x Monitor August 2012

framework establishes a reliable network of commitments throughout the organization and business enterprise, which each employee of and contributor to the research site knows and understands and to which everyone performs. Thereby, the organization or business establishes, maintains, and manages this network of commitments, which supports building quality into the clinical trial practices. 8 Figure 2 An Outline of Clinical Research Phases Within a Quality System (QMS) Structure Protocol Design & Operational Design Investigational Product-Device Planning GCP Quality System Building Quality Into the Clinical Program Start-Up Conduct Close-Out Good Clinical Practices Data Lock & Analysis, CSR* Marketing Application A quality management systems approach in clinical research is a further extension of delivering quality care for those patients who volunteer to participate in clinical research. QMS Elements Enabler Enabler Responsibilities Quality Culture, Policy, Objectives Resources Quality Commitment: All Staff Process Performance Systems, Processes, Documentation Quality-Performance Monitoring System (Quality Control/Quality Assurance) Corrective & Preventive Action (CAPA) System Change System-CQI, Document Control Review: Ongoing Acceptability of the System Knowledge Risk Inherent in a quality management system is documenting the work performed, evaluating deviations from the established quality standards and controls, and taking the necessary actions for immediate and continuous process improvement. This framework for quality is not so different from that used in hospitals and medical institutions, which must obtain and maintain accreditation of their facility, per country/state requirements. Therefore, a quality management systems approach in clinical research is a further extension of delivering quality care for those patients who volunteer to participate in clinical research. Table 1 presents a targeted description and application (examples) of critical aspects of both the quality management system framework and the plan, do, check, act principles. *CSR=clinical study report. execution of the clinical trial to the established quality standards. Specifically, quality metrics provide the ability to measure progress to the standards and goals defined by the organization. Quality metrics are reported as the error rate, and require predefined tolerance limits for effective monitoring, measurement, and reporting of quality and compliance signals to the organization. Note that what most organizations refer to as key quality indicators are a subset of performance indicators. For example, a site monitoring report may be completed on time; however, is the report completed Figure 3 The Network of Interrelated Processes correctly per the standards outlined in the monitoring visit report completion guidelines, monitoring visit report template, monitoring plan, and monitoring SOP? Did the CRA/monitor capture critical issues, protocol deviations, or violations in the monitoring report as identified by either database review of deviation listings or during an onsite quality assessment visit that was performed by his or her supervisor or the sponsor? Additional examples are described in Table 2. Upon identification of a metric that has either met or is out of range of the predetermined tolerance limits, the next Metrics: Reporting Performance of the Quality System When defined and used appropriately, metrics are an effective tool for monitoring and measuring the performance of an organization s quality system and People Network of interrelated processes with each process made up of: Work Activities, Tasks Records, Documents, Forms Resources, Rules, Regulations Reports, Materials Supplies, Tools, Equipment x 31

Table 1 Descriptions and Examples of Quality System Elements Quality System Element Quality Culture An organizational value system that results in an environment that is conducive to the establishment and continual improvement of quality Maintain an awareness of quality as a key cultural issue Make sure that there is plenty of evidence of management s leadership Empower employees and encourage self-development and self-initiative Recognize and reward the behaviors that tend to nurture and maintain quality culture Commitment Required to establish authority and commitment to the provision of resources to develop, maintain, and sustain the quality system Quality Policy Describes how an organization approaches quality and the requirements for meeting expectations. Risk Supportive framework (enabler) for the quality management system A continuous, formal process involving the systematic application of documented management policies, procedures, and practices to the tasks of analyzing, evaluating, controlling, and communicating risk Knowledge Supportive framework (enabler) for the quality management system Prospective, controlled, and methodical approach for identifying/capturing and disseminating collective organizational expertise (past, present) for staff to perform their roles, duties, activities, and functions in clinical research to improve organizational performance The process usually involves several of the following stages or subprocesses in the use of knowledge: create, identify, collect, organize, share, adapt, and use 9 Communication A quality system s requirements, standards, and effectiveness need to be communicated to everyone involved in a particular activity in order for the system to work 32 x Monitor August 2012 Example Incorporated into mission statement, employee on-boarding training program, and employee handbook Executed at a division/department level in a manner that provides a direct, concrete, and succinct link between quality and the specialty areas of employees Quality policy document for the organization/business that is referred to and adhered to by the organization at all levels Video on organization/business intranet communicating this commitment to all employees Assignment of resources (people, funds, facilities, equipment) as required to develop, maintain, and continuously manage the quality system Guy s and St. Thomas National Health Service (NHS) Foundation Trust Quality Policy The Quality Policy describes the quality system that is required to ensure that clinical research conducted in NHS clinical research facilities in the U.K. fulfills statutory requirements laid down in current and any future regulations, as well as research governance guidelines. The aim of the policy is to maintain a quality management process that not only meets the requirements of applicable regulations and guidelines, but also adds value to the reputation of the trust as a location where high-quality, robust research is conducted. Risk management principles may be applied to clinical trials by prospectively identifying those aspects that are critical to ensure the reliability of results (data quality, data integrity) and protection of study subjects Protocol Risk Plan for global clinical trial execution Recruitment Plan risk assessment and associated Risk Plan to ensure on-time subject enrollment Conduct lessons learned during the clinical trial and review and identify information that is important to other teams to improve performance in their job, on their protocol; includes real-time analysis and review of systems, processes, SOPs, and methods and implementation of changes in a timely manner with rapid communication to staff ACRP s Online Community, which includes sharing of best practices, lessons learned, SOPs; the end-user reviews the information and utilizes it as stated or modifies it for use (e.g., site SOPs and forms posted) Policies, procedures, study requirements, and responsibilities should be communicated prospectively to affected staff, contract research organization (CRO) and service provider personnel, and clinical investigators; adequate training should be provided to all study staff, per their job functions (sites, sponsors, CROs, vendors, contractors, consultants) (continues)

Table 1 Descriptions and Examples of Quality System Elements (continued) Quality System Element Job Responsibilities and Assessment of Personnel Competencies Job descriptions are present and current, and there is a methodology for continuously reviewing and updating them according to changes in the regulatory landscape and other job responsibilities Do personnel possess competencies, knowledge, experience, skills, and training to execute their assigned responsibilities, duties, functions, and activities? Document Control Document control is a consistent method of controlling documents that includes version numbering, dating, issuing, and withdrawing as controlled procedures Ensures that only the current version of the procedural document is used by all personnel Quality Plans Documents specifying which procedures and associated resources shall be applied, by whom, and when to a specific project, product, process, or contract; describes how the quality system is applied to a specific deliverable Quality Standards Organizational standards/requirements for conducting business Regulations, guidances, guidelines, regulatory authority inspection manuals Quality Control A set of activities intended to ensure that quality requirements are actually being met Monitoring and Measurement of the Quality System Organization monitors customers perceptions of whether it has met their requirements Suitable methods are utilized to monitor and measure the organization s performance Facilities and Equipment Controlled environments required to execute the protocol Quality Assurance The aspect of quality management that focuses on the confidence that quality requirements are fulfilled Self-inspection audits of the quality system, processes, activities, and documents to independently assure that the defined requirements/standards for the protocol/system/ process/procedure have been adhered to Conducted by defined, qualified personnel independent of the activity; performed in a systematic manner Example Each position has a current and documented job description, employee/contractor training plan, and training file Procedural documents include SOPs, informed consent templates, and investigational product accountability logs, etc. Protocol-Specific Quality Plan Monitoring Plan, Data Plan, Project Plan, Recruitment Plan, Quality Oversight Plan of Third Parties Plan, Data Monitoring Committee Charter, Data-Safety Monitoring Plan, Annual Audit Plan Protocol-specific requirements for endpoint assessments (i.e., by an MD or certified assessor), and other study-related procedures/ activities Each informed consent is obtained prior to any study-specific procedures performed on the subject Protocol (e.g., subject enrollment criteria) Trial-specific procedures and expectations (e.g., correct data entry per the source documentation into electronic data capture) Predefined requirements (e.g., temperature at which investigational product must be stored) Organizational controlled, procedural documents (e.g., SOPs [unblinding, randomization], templates, forms, job aids [flow charts, reference cards, business operations manuals]) Procedural documents include: SOPs, informed consent templates, investigational product accountability logs Sponsor s clinical research associate (CRA)/monitor perform onsite monitoring visits to a clinical investigator Refrigerator and freezer temperature is routinely checked/monitored to ensure the temperature is within specified limits (requirements); this routine check/monitoring is documented in the temperature log Data analyses of performance (predefined/predetermined metrics and tolerance limits) Customer report of product quality complaints Internal audits of the quality system (see quality assurance section) Clinical site s refrigerators/freezers possess the protocolrequired temperature range with documented evidence of ongoing monitoring, calibration, and maintenance per manufacturer specifications Organization s routine audit of the quality system Clinical study report audit Clinical investigator site audit Quality systems audit of vendor/third party Trial Master File audit (continues) x 33

Table 1 Descriptions and Examples of Quality System Elements (continued) Quality System Element Example Deviation Protocol deviation log maintained by the site and by CRAs/ Supports learning in the organization through the identification, monitors recording, and investigation of activities that are not performed correctly or as planned, and provides the framework on how to investigate, plan, and change the way activities are performed Corrective and Preventive Action (CAPA) Program Incorrect version of the informed consent used for a study Corrective action aims to address and manage identified areas subject (corrective action: current version of consent in the file) of noncompliance and nonconformity (an issue or problem) by Subject s investigational product dose not adjusted per the investigating the root cause in order to accurately eliminate it results of their liver or renal values, as required by the protocol Preventive actions aim to establish proactive methods/steps/ (preventive action: checklist, per patient visit, outlining actions to foresee any issues and to prevent them from occurring requirements) Continuous Improvement Use of audit findings, audit conclusions, analysis of data, The organization shall continually improve the effectiveness management reviews, and deviation management to improve the of the quality system through the use of the quality policy, quality system/organization quality objectives, audit results, analysis of data, CAPA CAPA plans implemented as a result of audit conclusions, audit program, and management review findings, internal monitoring of systems/processes and quality Build on the knowledge known and learned to make system by staff, CRA/site monitor feedback proactive improvements in individual trials and across all trials, and in the business enterprise (close correlation to knowledge management for the organization) Issue Escalation Cases of suspected scientific/ethical misconduct and/or The issue escalation process describes how the project fraud are escalated within 24 hours to the compliance/quality identifies, tracks, and manages issues and action items that assurance department and senior management for investigation are generated throughout the project life cycle; it also defines how to escalate an issue to a higher level of management for resolution and how resolutions are documented Unanticipated issues and action items are assigned to a specific person for action and are tracked to resolution Review of the Quality System s Performance Biannual meeting and review of the quality system utilizing Senior management s overarching responsibility for review metric reports and performance dashboards and analysis, at predetermined intervals, of the functioning/ adequacy of the quality system utilizing key indicators of performance, quality, and revenue Has the quality system provided management with the information to reassure them of compliance to the organization s quality system? Through the assessment of metrics (performance, quality indicators), does the organization need to add anything or change anything in the quality system to meet organizational quality objectives? step is the evaluation of the metric and any relevant companion metrics. This analysis supports a robust and comprehensive root cause analysis as to what the issue is, the issue s impact, and what corresponding directed and focused CAPA steps and continuous improvement measures should be taken. Further, metrics need to be evaluated with the associated risk management plan and actions analyzed: Did our mitigation plans work? Should we implement our predefined contingency plans? Do we need to reevaluate the assumptions reference in the development of the risk management plan and revise that plan accordingly? Is this an issue we did not expect, thereby necessitating the need for the development of a new risk management plan? Quality systems metrics analysis focuses on the following critical questions: 10 1. How is the system performing? 2. Which aspects of the system are performing poorly or need improvement? 3. Which aspects of the system are performing adequately? 4. What is ideal performance? 5. Are the improvements having the desired effect? Figure 4 presents some useful guidelines for successfully using and evaluating quality systems metrics. 10 Summary Effective execution of clinical trials featuring quality built within requires 34 x Monitor August 2012

Table 2 Examples of How Quality Indicators are Subsets of Performance Indicators Performance Indicator Quality Indicator Case report forms (CRFs) completed and CRF query rate < 5% submitted on time > 90% Imaging study completed on time >95% Image readability > 98% Blood samples collected on time > 95% Quantity not sufficient < 1% Staff trained on SOPs prior to performing SOP deviation rate < 3% responsibilities/tasks in SOP > 95% Staff trained to the protocol-investigational Protocol deviations-violations < 2% plan prior to performing delegated study tasks > 95% Site staff delegated tasks prior to study-start > 95% Subjects enrolled on time > 85% Informed consent obtained 100% Informed consent obtained 100% a prospective, systematic approach to quality through the implementation of a robust and comprehensive quality management system whereby metrics provide the ability to measure an organization s performance. The qual- Staff delegated responsibilities correctly per licensure and/or certification requirements per state/country 100% Subjects enrolled meet enrollment criteria 100% Subjects consented with the correct ICF version 100% Subjects consented prior to study-specific procedures 100% ity management system principles described in this article represent the framework and components of progressive 21st century practices for use by all stakeholders in the clinical research enterprise. Figure 4 Guidelines for Using and Evaluating Quality Systems Metrics Successfully Using Quality Systems Metrics Identify the stakeholders and the metrics for their activities Determine the metrics required for periodic management review of the quality system Determine how the metrics will change the way you perform your business Select the right metrics and rationalize the calculation rules (do you have this requisite expertise inhouse?) Determine how you will use the results Define the reporting mechanisms (scorecard, dashboard, real-time reports) and issue escalation pathways both internally and externally with other parties Determine the right source of the data Collect the data and validate the results Continuously evaluate the metrics (obtain feedback and study the utility of your measurement) Continuously improve the process Communicate the results Successfully Evaluating Quality Systems Metrics What does this metric measurement mean? What will I do with this information? How will this communicate a quality performance threshold or tolerance limit requiring investigation or evaluation? Do I need another metric to get the whole picture? Identify any companion metrics to assist with the evaluation References 1. Clinical Trials Transformation Initiative. Available at https://www.trialstransforma tion.org/scope. 2. Sweeney F. 2011. Defining Quality in Clinical Trials. DIA Annual Meeting, 2011. 3. Cameron K, Since W. 1999. A framework for organizational quality culture. Quality Journal, 1999, pp. 7 25. 4. BARQA. 2010. Quality Systems Workbook. Available at www.barqa.org (free download available). 5. International Organization for Standardization; www.iso.org. 6. Kleppinger C, Ball L. 2010. Building quality into clinical trials with use of a quality systems approach. Clinical Infectious Disease Journal 51(Supp. 1): S111-S116. 7. ICH Q10 Pharmaceutical Quality System presentation, www.ich.org/products/guidelines /quality/quality-single/article/pharmaceutical -quality-system.html. 8. Burrow D. CDER BIMO Warning Letters as Case Studies-Building Quality in Clinical Trials. Presentation, ACRP Global Conference, 2012. 9. American Productivity and Quality Center Publication. 2000. Stages of Implementation: A Guide for Your Journey to Knowledge Best Practices. Houston, Texas. 10. Zuckerman D. 2006. Pharmaceutical R & D Metrics. Gower Publishing Ltd., England. Additional Sources Cianfrani C, Tsikais J, West J. 2009. ISO 9001: 2008, Explained. Milwaukee, Wis.: ASQ Quality Press. Clinical Trials Transformation Initiative. Developing Effective Quality Systems in Clinical Trials: An Enlightened Approach; www.ctti.org. ISO 9001:2008. Quality Systems Requirements; www.iso.org. Ribière V, Khorramshahgol R. 2004. Integrating total quality management and knowledge management. Journal of Systems 16(1). Toth-Allen J. 2012. Building Quality into Clinical Trials: An FDA Perspective. FDA Webinar, 04 May 2012. Tricker R. 2010. ISO 9001:2008 for Small Businesses. Burlington, Mass.: Elsevier. Liz Wool, RN, BSN, CCRA, CMT, has 22 years of experience in the clinical research industry. She is president and CEO of QD-Quality and Training Solutions, Inc. (QD-QTS), a clinical quality systems, training, auditing, and CRO-vendor oversight consulting firm providing services to institutions, investigators, sponsors, and CROs. QD-QTS has offices in San Bruno, Calif., and Franklin, Tenn. A certified Master Trainer and instructional designer, she is also a member of ACRP s Association Board of Trustees and Editorial Advisory Board. She can be reached at lizwool@qd-qts.com. x 35