We are used to the idea of measurement in the general practice of

Transcription

1 This article describes some key considerations for organizations as they review their approach to metrics or as they begin developing a key set of metrics. Keith Dorricott, BSc Using Metrics to Direct Performance Improvement Efforts in Clinical Trial Management We are used to the idea of measurement in the general practice of medicine, such as the vital signs taken after a baby is born or during the course of therapy for an illness in an adult. We would wonder what a medical practitioner was doing if he or she failed to take measurements such as blood pressure, heart rate, cholesterol, etc., when providing medical care and then compare those measurements to established norms. Measuring is fundamental to our ability to understand and control the world we live in, and this is particularly true for scientific disciplines; it is part of the scientific method. In clinical trial management, if you want to know how enrollment is going for a particular trial, you might look at the enrollment rate or the number of subjects enrolled. You might want to compare these to your initial expectations to see if you are on track and take remedial action if necessary. Without a defined process for measurement a method, a way to capture data for review what would be the point of taking the measurements in the first place? A metric has been defined as a standard of measurement. 1 Metrics are essentially the definitions of how we collect data on measurement and the value of those measurements once they are made. Many organizations recognize the need to measure (to use metrics), but their measurement systems have typically built up over time and have not been put together from a strategic perspective. Hammer 2 claims that across all the organizations with which he works, there is a wide consensus that they measure too much or too little, they measure the wrong things, and they do not use the metrics effectively. 2 This article describes some key considerations for organizations as they review their approach to metrics or as they begin developing a key set of metrics. The overall approach is shown in Figure 1. Performance Metrics in Clinical trials Measurement Needs a Purpose In the general practice of medicine, there are myriad things you could measure. However, if you attempted to measure everything, there would be a substantial cost and the medical practitioner would be overloaded with all the data. The particular measurements that are useful will depend on the circumstances; measurements of the health of a newborn baby, for example, will be very different from those of someone who has high cholesterol. Similarly, in clinical trial management, there are many things you could measure. Often, companies attempt to measure and track large numbers of metrics simply because they can. 3 If you measured and reported all possible metrics across a set of clinical trials, the cost would be significant. You x 9

2 Figure 1 Selecting and Implementing a Metric Select/Define your metrics and targets Determine the purpose of measurement Think Big Picture Drive value in the metrics use the so what? test Have a mix of metric types Measure process not people performance Start small Metrics definitions use of industry standards Determine how you will collect, display and use the metrics Program and validate would be completely confused about how to interpret all the data and would not have the resources to tackle all the questions that would arise, resulting in inaction. You would have the cost of data collection and reporting, but no outcome. Considering these factors, a typical flow for how a metric might be used is shown in Figure 2. To get through the steps of selecting and implementing a metric (Figure 1) and the steps involved in using a metric (Figure 2) involves many resources and their associated costs. Every metric is a balance of that cost versus the benefit you can get out of the metric itself. There are only a relatively small number of key things that are really useful to measure in a given circumstance (perhaps up to a dozen); these are often termed as the key performance indicators. So how do you go about determining those vital metrics? A key consideration that will help you focus on the important metrics is Figure 2 Using a Metric Review and interpret Are data on track? Yes Continue to monitor determining the purpose of measurement. 4 For example, the purpose of measurement in clinical trial management might be: For a contract research organization (CRO) to be able to demonstrate oversight for the trials in its control to ensure timely, accurate, actionable data. To reduce the time to conduct clinical trials. To maximize the success of applications of new drugs to regulatory authorities. As described in the following sections, once you have determined the purpose of your measurement, there are a number of other key considerations. Think Big Picture As with a masterpiece painting, in the general practice of medicine, the little No Root Cause Analysis Agree and take actions to get the metric on track details are important, but so is the overall composition. Treating symptoms individually without considering them together and whether there is a common underlying cause would not be in the best interests of the patient. Thinking big picture from the perspective of those who are going to use the metrics can help to narrow down the metrics that you plan to collect. Thinking big picture from the perspective of those who are going to use the metrics can help to narrow down the metrics that you plan to collect. Part of thinking about the big picture is to select a mix of different metrics types. Having different types of metrics in your measurement system helps to minimize the chance of suboptimization. 4,5 For example, focusing only on speed could make an activity faster; but if it adversely affects quality, then subsequent activities can be undermined and the overall effect might be to increase the length of the trial. Generating a protocol quickly might be desirable, but not if there are underlying quality issues that mean costly, time-consuming protocol amendments are needed later. 10 x Monitor August 2012

3 Some different types of metrics you should consider are shown in Table 1, along with examples and the risk of sub-optimizing by focusing only on a specific metric type. Note that a metric is typically either a lagging or leading indicator, and an indicator of one or more of the factors of cycle-time, timeliness, efficiency, or quality. The bad news is that some of the most important things are not always measurable. 6 For example, many measurements are possible for a newborn baby, but can you measure the instinct of the midwife who looks at the baby and says he looks good and healthy? The right metrics can certainly help you manage the business, but they will never tell the whole story. Keeping the big picture view helps you to realize when to be cautious in using particular metrics without others, or in relying too much on metrics that might be leading to suboptimization. Some Metrics Contain More Value Some metrics are inherently more useful than others; they can tell the story that would otherwise need several lesser metrics. A good way to determine if you have selected one of these more powerful metrics is to use the so what? test. 4 If you were to gather data on that metric, what would you do with it? What action might it drive? If you cannot think of actions that would result from collection of the data on a particular metric, it may not be of value to use that metric. For example, you might be interested in the quality of work performed at investigator sites in relation to the attention the sites have received from monitors. So you might select the number of investigator site audits in the last three months as a metric (see Table 2). Imagine you now have the data: There were two audits. So what? You don t know how many audits there should have been; you don t know how many sites could have been audited; and you don t know the result of the audits. Of course, you could collect a variety of metrics that would capture those other details, but perhaps there is a single metric of more value, such as the number of critical observations? Imagine you now have the data: There were four critical observations. So what? Maybe there were 50 audits? If you cannot think of actions that would result from collection of the data on a particular metric, it may not be of value to use that metric. Perhaps you could measure the mean number of critical observations per site Table 1 Metric Types Metric Type Description Example Metric Leading Indicator Provides information that you can The proportion of sites activated act on immediately to get the trial/ versus expected would be a leading process back on track. indicator for whether subject enrollment is likely to be on track. Lagging Indicator Cycle-time Timeliness Efficiency Quality Provides information that you can use for future trials or for baselining for process improvement efforts. Measures the time taken to complete a task. Measures whether a particular milestone has been met. Measures the amount of resource required to complete a task or set of tasks versus that expected. Measures how well an output from a process meets the requirements of the customer of that process. The time taken from last subject last visit to database lock. The median time from subject visit to data entry into an electronic data capture system. The number of days between planned and actual dates of the first site activated. The difference between the actual final total contract value and the initial baseline contract value for a CRO running a clinical trial. The proportion of expedited safety reports that are received by regulatory authorities within the required timelines gives an indication of the quality of the pharmacovigilance reporting process. Risk of Focusing on This Metric Type Only Lack of data to help with process understanding and improvement Lack of data to affect current work and act before negative consequences occur Faster cycle-time with poor quality leading to a process needing to be repeated unnecessarily; longer overall cycle times Meeting the timelines, but using excessive resources and not at the required quality level Process using minimal resources, but not meeting timelines High quality outputs, but missing timelines and with high cost x 11

4 Table 2 Building Value into Your Metrics Possible Quality Metric Data So What? Number of investigator site audits in the last three months Number of critical observations in the last three months Mean number of critical observations per site audit in the last three months audit? Imagine you now have the data: There were two. So what? Here you have some actionable data; having an average of two critical observations per site audit would be a real cause for concern. You would want to understand the root cause, finding out which sites were audited and looking for systemic issues, such as a confusing protocol or poor training of site staff. This one metric has high value, as it gives an indication of quality of work at investigator sites and by monitors. It best matches the purpose of interest to you. In a similar way, if you are looking for a metric to indicate whether a clinical trial is on track, measuring the number of sites that have been activated does not pass the so what? test. Measuring the proportion of sites activated out of the total expected has more value; even better, however, would be to measure the proportion of sites that have been activated out of those expected to be activated at that particular time. This gives an immediate indicator of whether you are on track for activating sites, and there are clear actions you could take. It passes the so what? test. Considering the value inherent in different metrics also brings out the Knowing this tells us nothing about the quality. Possibly a cause for concern, but we do not know how many audits there were. Definitely sounds like a cause for concern. We would want to take action and investigate further. importance of the definition of a metric. The definition should be written down so that there is no ambiguity; this definition can be used when trying to understand why the metric is at a particular value. There are industry organizations that have developed standardized, defined metrics for use in clinical trial management, 7 and the potential metrics they provide can Using a metric to measure the performance of people may induce people to spend their time trying to meet the target often by any means rather than to focus their efforts on trying to improve the process itself. be used as a starting point for metrics selection. Using standard metrics also makes it easier to benchmark to compare performance across organizations and help to drive continuous improvement. Metrics Should Measure Process Performance, Not People Performance Increasing Value As you select your metrics, you should focus on the process rather than using the metrics to measure people. By using a metric to measure people, there is a high risk of sub-optimization, as individuals focus on the metric to the exclusion of everything else and gaming or cheating can result. 8 Seddon describes the impact of implementing a metric by looking at the percentage of times ambulance staff reach Category A incidents within eight minutes of an emergency call. 9 Using the metric to assess manager performance led to misreporting, such as Category A calls being reclassified as Category B when the eight-minute goal was not met and Category B calls classified as Category A when crews arrived within target. Also, varying the definition of the start and end times meant that eight minutes to one authority could be 10 minutes to another. As seen in the case just described, using a metric to measure the performance of people may induce people to spend their time trying to meet the target often by any means rather than to focus their efforts on trying to improve the process itself. It is one of the best ways to make people lose the big picture and sub-optimize. It would be better to involve the staff in determining appropriate metrics more clearly related to the purpose of the process the health outcome for the patient and then to get them to focus their efforts on using the metrics to understand their process better and to improve the process for everyone. In Conclusion Finally, you need to consider how you will display, validate, review, and act on the metrics. There are many systems that can be used for display, from basic Excel spreadsheets to specific software designed for the purpose. Ensuring the data are accurate (i.e., validating) is an important step to give confidence to those who are going to use the metrics. All these efforts will be of little value, however, without a process like that shown in Figure 2 to review the metrics so that they can be used to drive decisions and actions. In the general practice of medicine, measurements are fundamental. Similarly, measuring the clinical trial process can give valuable information 12 x Monitor August 2012

5 for use in tracking, understanding, and improving process performance. The number of metrics you track should be kept small (around eight to 12), to allow the organization to focus on what really matters and to minimize cost. There should be an overall purpose of measurement that will help you when selecting appropriate metrics to track and review. Ideally, metrics should represent more than a simple counting of items, and should cover a range of different types, such as lagging, leading, timeliness, cycle-time, quality, and efficiency. Having different metric types from across the process ensures that they complement each other and provide a better overall picture of performance. However, watch out for metrics that focus on improving one area to the detriment of another. The most crucial consideration is that the metrics are actually used not for managing people performance, but to understand and improve processes. If particular metrics fail the so what? test, then consider removing or replacing them. Ideally, metrics measure the performance of systems and processes, and analysis of them should help direct your efforts in process improvement. By careful use of measurement in clinical trial management, we extend the scientific method beyond the science of the trials themselves, and that science reminds us of the really big picture that the fundamental purpose of our efforts as clinical researchers is to improve patients lives. References 1. Merriam-Webster Online Dictionary, www. merriam-webster.com/dictionary/metric. 2. Hammer M The seven deadly sins of performance measurement and how to avoid them. MIT Sloan Management Review 7(43). 3. Nelson G Implementing metrics management for improving clinical trials perfor- mance. BeyeNETWORK. Available at www. b-eye-network.com/view/ Zuckerman DS Pharmaceutical Metrics. Gower. 5. Sullivan L Defining quality that matters in clinical trial startup activities. The Monitor 25(7): Nelson LS, quoted by Deming WE Out of the Crisis. MIT Press, p Metrics Champion Consortium, champion.org/default.aspx. 8. Pyzdek T Gaming the metrics use metrics to guide improvement, not measure the performance of people. Quality Digest. Available at 9. Seddon J Freedom from Command and Control. Vanguard Education, p Keith Dorricott, BSc, is director for operations management, process improvement, and metrics at INC Research in the United Kingdom. He is an active member of the Metrics Champion Consortium, and worked with the organization to launch the Process Improvement Work Group in Prior to his seven years working on improving the clinical trial process at various contract research organizations, he was technical manager at Eastman Kodak manufacturing. It was at Kodak that he honed his skills in process improvement techniques, including Six Sigma and Lean. He is a Lean Sigma Master Black Belt. He can be reached at keith.dorricott@incresearch.com. x 13