Standard of measurement by which efficiency, performance, progress, or quality of a plan, process, or product can be assessed 1.

Transcription

1 Meaningful Metrics

2 What is a Metric? Standard of measurement by which efficiency, performance, progress, or quality of a plan, process, or product can be assessed 1. A quality metric is a measurement of quality. It can be either a leading or lagging indicator. A leading indicator provides information proactively in order to take action before contravening a target objective or goal. Can be used to predict an outcome. A lagging indicator measures performance after the fact and can be used to determine if goals and objectives have been achieved Confidential

3 Regulatory Guidance Limited guidance in the GMP s on what constitutes quality metrics ICH Q10 (2007) - Performance indicators Measurable values used to quantify quality objectives to reflect the performance of an organization, process or system, also known as performance metrics in some regions. C Canadian GMP s refers to APR s 21CFR180(e) - US GMP s, refers to APR s 21CFR US GMP s for Medical Devices refers to quality data EU GMP s Quality Management refers to Product Quality Reviews 3 Confidential

4 Why do we collect Metrics? Gauge to measure performance Provides feedback and information on your state of control Indicator for trends to act proactively not reactively.leading Allows you to make informed decisions Drives improvement (minimize cost and maximize performance) Good business.good motivator 4 Confidential

5 What do we Measure What Gets Measured gets Done True, but sometimes we lose sight of why we collect metrics or question the value of what we are measuring..collecting metrics for the sake of collecting metrics is non value added. Your CMO is not in the metrics business Not everything that counts can be counted, and not everything that can be counted counts. Albert Einstein 5 Confidential

6 What Metrics do you Measure at your CMO? Only those that you need! 129 line items?? You re kidding right?? Service? Quality? Leading or Lagging? Consider from a product quality perspective and a GMP perspective 1. Define objective 2. Define metric 3. Set targets 4. Collect data 5. Assess and communicate 6. Modify or change What you measure at your site may not be an important measure at you CMO s site 6 Confidential

7 CMO Organizations Metrics are an integral part of our business. Like most companies we use them to drive performance, identify areas for improvement and strengthening Have defined set of metrics based on our assessment of our performance Metrics are aligned to our corporate objectives Have capabilities to electronically track and trend most data Have resources to track and trend our data 7 Confidential

8 Common Quality Metrics Number of Deviations (trending root cause) Also consider deviation per batch Number of OOS s (trending root cause) Batches Right first time If the site is running generally >90% then consider dropping metric. Lower numbers or downwards trend should be closely monitored. Deviations Past Due Completing an investigation (RCA, CAPA Plan and Approvals) within a timely manner is important. However ensuring that the content of the investigation in complete and thorough is critical. Although there are those that may go past the due date with QA approval, recommend that it is the exception rather than the rule. Consider Aging? CAPA Past due Keeping track of your CAPA s is imperative. Make sure timelines are realistic. Although there are those that may go past the due date with QA approval, recommend that it is the exception rather than the rule. Consider Aging? 8 Confidential

9 Other Considerations Stability Past Due - compliance to pull dates APR past Due compliance to schedule (ensure CMO Portion is defined) SOP Training Compliance EM results and trends Batch Rejections Documentation Errors Documentation creation, review and approvals Audit observation trends others On time delivery, Adherence to Schedule, EH&S, Filings, Launches 9 Confidential

10 Metrics Formatting Consider a 12 month rolling Identify what constitutes a positive or negative trend Select reasonable scales and maintain consistency Communicate and reinforce definitions and associated calculations Automate if possible eliminate subjectivity Include colors to emphasize the good, the bad and the ugly Include commentary to clarify anomalies or negative trends Consider measuring the negative not the positive 10 Confidential

11 # Past Target Approval Date # Past Target Approval Date Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 # Past Target Approval Date Examples 4 3 EU Regional Trend 9 Commercial Operations Comparison F G H US Regional Trend A B C D E 11 Confidential A B C D E F G H Within 10 days Past Target Approval Date (On-Time < X 10 Days) Within 30 days Past Target Approval Date (10 Days < X 30 Days) Within 60 days Past Target Approval Date (30 Days < X 60 Days)) Beyond 60 days Past Target Approval Date (60 Days < X) a F G H B C D A E 1

12 Percentage (%) Past Due Number (#) Past Due # of Dev. Examples 7 6 North American Region LINCON OPERATIONS DEVIATIONS ROOT CAUSE TREND FY2011 Procedure External Process Other Material Human 1 2 Month Equip/Mech A B C D E F 0 CAPA Past Due % Target CAPA Past Due # 12 Confidential

13 Data Management What do you do with the Data? How is it reviewed, analyzed and assessed internally? If it stopped coming would anyone know? Who see s the data? Is the visibility restricted to Quality management? Is your data shared with the CMO? Is their open dialogue and discussion? Is the data defined and understood? Does the internal management team understand what is being measured? Does the CMO have the same level of understanding? How effectively is it communicated Does the data get disseminated down to those that have an opportunity to influence or impact the outcome? 13 Confidential

14 Communication and Ownership It is critical that results are effectively communicated and understood throughout the organization Ensure that there is departmental ownership for their own performance metrics. Quality metrics do not just belong to Quality A well run department.will be actively involved with the collection and dissemination of metrics pertaining to their area of operation and have detailed plans in place for remediation they will not wait to be asked. Individual contribution and ownership is key to success If you are not keeping score, your only practicing, not playing. Vince Lombardi 14 Confidential

15 Awareness and Ownership Department Meetings Are metrics discussed? Are employees engaged? Do employees contribute to CAPA plan? Are employees rewarded for success? Site Management What goes on in management meetings? Are metrics presented? Who is presenting the data? What questions are being asked? Does each concern have a defined CAPA plan? Are we holding our managers accountable? Are we being accountable? Are we seeing positive change? Ownership? Senior Executive What goes on in a executive review? Who is presenting the data? What questions are being asked? Does each problem have a defined CAPA plan? Are we holding our sites accountable? Are we being accountable? Are we providing resources and capital to rectify issues? Are we seeing a positive benefit for the meeting? A metric review meeting without action plans and decisions is of no value 15 Confidential

16 Ignorance is NOT Bliss!! The importance of senior management awareness, understanding and engagement cannot be overstated. Recent regulatory issues suggest that this awareness, understanding and engagement is either lacking or not as strong as it should be. There is a lot of attention at the moment on how FDA is approaching outof-compliance firm and it s a very positive development that the media is attuned to the types of failures that are occurring in the pharmaceutical industry, which often can be traced back to inattention by senior management. Rick Friedman September Confidential

17 CMO Relationship A CMO is an extension of your operation. If this relationship is not strong in your existing operation what does that mean about your understanding of what is occurring at your CMO. Regulators are considering holding drug owners more accountable for problem at their CMO s. Transfer of information and knowledge is an important first step followed by open and effective channels of communication This relationship needs to be actively and responsibly managed. This does not mean to measure everything and anything or pitch your tent. It is a partnership, make it work 17 Confidential