Adopting Site Quality Management to Optimize Risk-Based Monitoring Medidata and other marks used herein are trademarks of Medidata Solutions, Inc. All other trademarks are the property of their respective owners. Copyright 2013 Medidata Solutions, Inc.
2 Introduction As it is now well documented, clinical R&D organizations have been actively embarking on a paradigm shift in their approach to ensuring quality in the conduct of clinical trials. In particular, a more targeted, risk-based approach to site monitoring and data cleaning is being actively pursued by organizations both large and small. The primary goals of this new paradigm include significant improvements in resource efficiency associated with clinical R&D, increased quality in trial conduct and corresponding data, and the maintenance or even reduction of clinical trial timelines. While the interest in this paradigm has increased steadily in recent years, progress had been significantly impeded by the following factors: Lack of full electronic data capture (EDC) adoption, which severely hampers an organization s ability to proactively assess quality across all sites in a clinical trial. Concern over the potential of increased regulatory scrutiny, especially in the absence of clear guidance or endorsement for risk-based approaches from the FDA or EMA. Potential for actual decreased trial and data quality, due to less intensive on-site monitoring during trial conduct. The first of these issues has been largely overcome, as EDC and similar clinical technologies have become business as usual for most clinical R&D organizations over the past six or more years. The second issue was also finally addressed last August in the form of the FDA s draft guidance on risk-based monitoring, followed closely by EMA s reflection paper on risk-based quality management in clinical trials. These documents not only provide the industry guidance on how to approach a risk-based monitoring program, they firmly endorse the concept. The following from the FDA guidance clearly demonstrates this: FDA encourages greater reliance on centralized monitoring practices than has been the case historically, with correspondingly less emphasis on on-site monitoring. 1 The following will help you understand the challenges faced when implementing risk-based monitoring and how developing a site quality management program especially one leveraging real-time site quality analytics can simplify the adoption of this strategy.
3 The Role of Data Quality and Metrics in the Adoption of For many organizations, the final concern regarding potential impact on data quality has remained. Medidata has asked one question in this regard: to what extent is patient electronic case report form (ecrf) data actually corrected during trials, as a result of all of the intense scrutiny traditionally applied to that data by study teams in the form of 100 percent source document verification (SDV), extensive data management reviews and safety/medical reviews? In order to assess this, Medidata computed the following two measures using the Medidata Insights metrics warehouse, comprising over 2,500 studies from 65 contributing sponsor organizations: Total Data Correction Rate: The total percentage of ecrf data found to have one or more updates ( corrections ) following initial submission of the ecrf form. Post-Capture Data Correction Rate: The total percentage of ecrf data found to have one or more updates following the initial capture session at the site, defined as a four-hour window following initial submission of each ecrf form. The surprisingly low industry median for these two measures is illustrated in Figure 1. The total data correction rate comes in at just 4.3 percent, while the post-capture data correction rate is only 2.7 percent. Putting this in context, the post-capture rate indicates that greater than 97 percent of ecrf data is already in its final form ready for formal reporting, analysis and submission before any site monitor, data manager or other sponsor representative has had a chance to scrutinize the data! Figure 1: ecrf data correction rates 5.0% 4.32% 4.0% 3.0% 2.68% 2.0% 1.0% Total Post-Capture Source: Medidata Insights metrics warehouse
4 It is relevant to note that these metrics currently account for any updates to data values in the EDC audit trail, which includes natural patient event-based updates, such as adverse event (AE) resolution/outcome information that do not represent actual corrections. While Medidata has not yet filtered out these type of updates from the metric, initial analysis of AE and concomitant medication forms on various studies reveals a very high correction rate on these resolution data fields (e.g., AE resolution date, ConMed Stop Date, etc.), typically over 20 percent. Removal of these fields from the calculation will likely yield a significantly lower post-capture data correction rate, perhaps as low as two percent overall. Whatever the final figure, it s clear that our industry is spending a tremendous amount of resource effort for a very modest return, in terms of impact on the final clinical trial dataset. Figure 2 illustrates that a very typical breakdown of clinical trial costs and site monitoring in particular is one of the two largest cost drivers at 30 percent or more. If nothing else, the very modest ROI with traditional site monitoring methods should compel us to seek a more effective, efficient approach to ensuring clinical data quality. Figure 2: Clinical trial cost breakdown Site Payments Other IVRS and Drug Distribution Project and Clinical Leadership Data Management and Statistics Monitoring Source: Medidata internal analysis
5 The Impact of Regulatory Guidance on Site Management The FDA draft guidance identifies two key monitoring approaches that comprise an effective risk-based approach: on-site monitoring and centralized monitoring. On-site monitoring is the current de-facto practice, so the introduction of centralized monitoring now becomes a key enabler to realizing greater overall quality and resource efficiency in the new paradigm. But what exactly is centralized monitoring or more to the point what should it be? The FDA draft guidance outlines the primary goals of centralized monitoring that should guide a solution. Among them are the following: Target on-site monitoring by identifying higher risk clinical sites (e.g., sites with data anomalies or a higher frequency of errors, protocol violations or dropouts relative to other sites) Augment on-site monitoring by performing monitoring activities that can only be accomplished using centralized processes (e.g., statistical analyses to identify data trends not easily detected by on-site monitoring) Conduct aggregate statistical analyses of study data to identify sites that are outliers relative to others and to evaluate individual subject data for plausibility and completeness Conduct analyses of site characteristics, performance metrics (e.g., high screen failure rates, high frequency of eligibility violations, and delays in reporting data), and clinical data to identify trial sites with characteristics correlated with poor performance or noncompliance 2 The items above all reflect a similar theme that calls for leveraging the clinical and operational data captured through EDC and other electronic systems. Together, these can help sponsors identify sites with emerging quality-related risks, relative to other sites in the trial. This in turn enables the attention of a sponsor team s finite resources to be directed to those sites with emerging risks as proactively as possible to remediate any issues, thereby ensuring optimal quality across the trial. Any implementation of centralized monitoring should therefore include an effective signal detection system that emphasizes earliest possible detection of emerging quality-related risks at one or more trial sites.
6 Identification of the quality measures to use for signal detection should of course be driven by the key risk areas identified in the monitoring plan for each study. However, it is very important to note that the vast majority of trials will share similar key risk categories including the following: Patient safety monitoring and reporting, for example AEs and serious AEs (SAEs), Patient data quality, and Protocol compliance, including eligibility criteria, and visit and dosing schedules. Key Metrics to Consider for Adopting Centralized Monitoring As such, the industry can and should identify a relatively small set of measures that serve as effective surrogates for the quality of trial conduct by sites in each of these categories. As an example, one might consider the following measures related to the quality of patient ecrf data capture at each site, all of which should be readily computable using available EDC-based information: Data correction rate (percent of data corrected after initial entry) Rate of ecrf queries from clinical data management Rate of ecrf queries and/or SDV findings from site monitors Rate of auto queries (i.e., queries resulting from programmed data checks within the EDC system) Cycle time from patient visit to entry of ecrf data by the site Cycle time for sites to respond to queries from the sponsor team All of the above measures relate to processes that may have a real impact on the quality of ecrf data. One approach might be to compute all of these measures in parallel to help detect sites with emerging issues. This may have some degree of success; however, it will more likely lead to a proliferation of signal-detection noise that will actually detract from the goal of finding the true high-risk sites. Rather, we should ask which groups of measures are descriptive of the same site process and then choose the one or two that enable the most proactive and effective assessment of site quality around that process. With respect to data quality, for example, it is fair to say that ecrf query rates (e.g., auto queries, data management queries, site monitor queries, etc.) and data correction rates should all be good indicators of the level of quality with which sites are capturing patient data into the sponsor s EDC system.
7 In order to confirm this, Medidata conducted an analysis of these measures across over 2,000 studies in the Medidata Insights metrics warehouse, which reveals a very strong correlation between each of these query rates and data correction rates. So it holds that any one of these measures should be a sufficient surrogate to detect data quality issues. However, one of them auto query rate is clearly the most effective when viewed from the perspective of early signal detection. This is because, unlike manually generated queries and subsequent data corrections, auto queries are generated and observable immediately upon entry of the ecrf data at the site. Therefore, both the numerator and denominator comprising an auto query rate are always current, whereas the numerator in the other rates always lags behind the denominator by weeks or months (and yes, even years of delay in query generation have been observed!). Thus auto query rate is likely to be not only sufficient to detect potential data quality issues at sites, but also the most effective among the obvious choices. Note that the Visit-to-eCRF Entry cycle time metric while it does overlap with auto query rate as a surrogate for data quality also measures an additional aspect of site behavior that is worth observing. In particular, lack of timely attention to capture of patient data in the sponsor s EDC system and slowness in responding to sponsor queries may portend broader issues with site s attentiveness to and engagement with the trial overall. Additionally, various sponsor reviews of incoming EDC data begin losing their value the longer the delay in getting access to that data. So these cycle times provide an important complement and addition to the auto query rate. Understanding a Typical Site Quality Management Scenario It is indeed important to assess all potential site quality measures in the context of not only how effectively, but how proactively, they are able to support signal detection. Getting to a relatively small, focused set of standard quality measures is critical to a successful centralized monitoring implementation. From a tool perspective, organizations and study teams should be looking for a centralized dashboard in which these quality measures are pre-computed across all sites and that provides for each site a measure-by-measure and overall assessment of risk-level with respect to quality. The tool should be configured in a way that directs study team members quickly and effectively to sites with the highest risk levels requiring follow-up. Figure 3 depicts what the tool might display for a given site and measure. In the conceptual example, the AE rate for the study overall is 8.7 AEs per subject-year, and for site 105 it is 5.6, which puts that site at an elevated (yellow) risk level for potentially under-reporting AEs.
8 Figure 3: Example site and study AE rate scorecard 3.7 5.6 6.2 8.7 11.2 13.5 Site 105 AE Rate Study AE Rate Note that this would not mean conclusively that the site is under-reporting AEs only that it is at an elevated risk of such. The study team would be responsible for deciding what, if anything, is the appropriate course of action to address this observation. It could be as simple as having the site monitor take some extra time at the next site visit to scrutinize the site s AE reporting process and perhaps to review AE reporting requirements with the site staff. It could also include increasing the amount of SDV targeted for patients enrolled at that site particularly for the AE forms. Or it could be a combination of the above. Either way, with a site quality management program, sponsors could better anticipate such risks and identify the root causes more quickly. Implementing a Site Quality Management Program About Medidata Medidata Solutions is the leading global provider of cloud-based solutions for clinical research in life sciences, transforming clinical development through its advanced applications and intelligent data analytics. The Medidata Clinical Cloud brings new levels of productivity and quality to the clinical testing of promising medical treatments, from study design and planning through execution, management and reporting. We are committed to advancing the competitive and scientific goals of global customers, which include over 90% of the top 25 global pharmaceutical companies; innovative biotech, diagnostic and device firms; leading academic medical centers; and contract research organizations. info@mdsol.com mdsol.com +1 866 515 6044 Using a standard set of measures, this type of centralized dashboard could be deployed very quickly for each new study a critical requirement given the importance of early issue detection during trial conduct. Organizations should also plan to employ a dashboard tool that is turnkey: not only easy to deploy for each study but also with site quality metrics automatically assessed and presented in a near real-time fashion. Such a centralized monitoring tool is not only ideal but quite feasible, and will help your organization revolutionize the way total quality is managed in clinical research. For more information on how Medidata Solutions can help you fully realize the benefits of centralized and risk-based monitoring by leveraging the site quality management concepts in this paper, please contact your Medidata representative. Medidata Clinical Cloud Cloud-based clinical research solutions Innovative technology Data-driven analytics Reduced costs Improved time to market Faster decisions Minimized risk
9 Endnotes 1. AUG-2011 FDA Draft Guidance: Guidance for Industry: Oversight of Clinical Investigations A Risk-Based Approach to Monitoring 2. Ibid