Implementation of a clinical metrics Dashboard Case study from a large pharma

Transcription

1 Implementation of a clinical metrics Dashboard Case study from a large pharma ICR Symposia 28 Feb 2012 Janssen R&D, Global Clinical Operations Laurent Morlet Pharm.D, Program Manager Clinical IT

2 Setting the scene (I) the Organisation: Janssen R&D Global Clinical Operations (GCO) About 1500 dedicated staff Staff located in around 50 countries Around 200 studies ongoing at anyone time Up to 4000 sites active at the same time Senior Management Geographically spread, different cultural background Very motivated to have a performance dashboard Worried about how their region or function will look Suspicious on data being able to reflect how well they are doing Cinical Metrics Dashboard by Laurent Morlet 25 Jan

3 Setting the scene (II) the IT Landscape: Large CTMS system in place Change management under a governing board Multiple Owners/stakeholders Cumbersome to customise/modify Large SAP customised finance system MasterControl QAAD system for QA auditing Planisware customised for Project Management Office Department maintained Excel files Cinical Metrics Dashboard by Laurent Morlet 25 Jan

4 Setting the scene (III) Legacy performance tools: Operational reporting system extracting from CTMS Set of predefined reports Custom reports for specific individuals Operating on image copy of the CTMS CTMS built-in reporting capability Excel tweaking of the above Excel or Access wizardry by various groups or individuals in different departments or locations. Cinical Metrics Dashboard by Laurent Morlet 25 Jan

5 Key requirements Intention to Balance the scorecard Several categories established: speed, quality, capacity etc Review of external benchmarking (KMR) Doability of each metric within the organization constraint Ability to compare metrics Between different regions Between different therapeutic areas (TA) Between different study phases Roll up and drill down capability: On a geographical basis On a clinical phase basis On a TA basis Cinical Metrics Dashboard by Laurent Morlet 25 Jan

6 Key requirements Intention to Balance the scorecard Several categories established Review of external benchmarking (KMR) Doability of each metric within the organization constraint Ability to compare metrics Between different regions Between different therapeutic areas(ta) Between different study phases Roll up and drill down capability: On a geographical basis On a clinical phase basis On a TA basis Cinical Metrics Dashboard by Laurent Morlet 25 Jan

7 Key requirements 6 Categories defined: Speed Quality Cost and Revenues Customer satisfaction Capacity People Open ended list Limit influencing parameters so remediation can be determined Cinical Metrics Dashboard by Laurent Morlet 25 Jan

8 Key requirements Intention to Balance the scorecard Several categories established Review of external benchmarking (KMR) Doability of each metric within the organization constraint Ability to compare metrics Between different regions Between different therapeutic areas(ta) Between different study phases Roll up and drill down capability: On a geographical basis On a clinical phase basis On a TA basis Cinical Metrics Dashboard by Laurent Morlet 25 Jan

9 Speed BSC item Selected by GCO SLT? Specs defined in core team Is a KPI Is captured in a system Is easy to measure with Unit FPI on target YES X % of sites initiated YES % enrollment YES X LPI on target YES DB lock on target YES X RFS to Proposal signed (days) Proposal signed to Trial Plan approved (days) Protocol to First site open First site open to First Patient In FPI to Last patient In (sub selection reports) Last patient last visit to Database Lock NO NO YES YES YES YES Time taken to resolve DB queries YES X Protocol to DB ready/edc ready NO

10 Key requirements Intention to Balance the scorecard Several categories established Review of external benchmarking (KMR) Doability of each metric within the organization constraint Ability to compare metrics Between different regions Between different therapeutic areas(ta) Between different study phases Roll up and drill down capability: On a geographical basis On a clinical phase basis On a TA basis Cinical Metrics Dashboard by Laurent Morlet 25 Jan

11 Key requirements Made the use of absolute measurement nearly impossible Speed category: no absolute time valid Quality: #findings are relative to #audits Cost: direct $ amount irrelevant Capacity: Simple FTE counting not valid Necessary to switch to relative measurements Use of % chosen in many cases Difference between actual and set target Cinical Metrics Dashboard by Laurent Morlet 25 Jan

12 Key requirements Intention to Balance the scorecard Several categories established Review of external benchmarking (KMR) Doability of each metric within the organization constraint Ability to compare metrics Between different regions Between different therapeutic areas(ta) Between different study phases Roll up and drill down capability: On a geographical basis On a clinical phase basis On a TA basis Cinical Metrics Dashboard by Laurent Morlet 25 Jan

13 Key requirements Metrics to be valid and meaningful at any level: Site Country Oncology Phase II Calculation to be doable at any level Choice of recalculation vs averaging Cinical Metrics Dashboard by Laurent Morlet 25 Jan

14 Key requirements Cinical Metrics Dashboard by Laurent Morlet 25 Jan

15 Review of the task ahead, a set of metrics that: adapt to the organisation and IT landscape No major alteration of business systems to achieve reasonable timelines Would be comparable across many dimensions Meaningful at any level Calculated at any level Cinical Metrics Dashboard by Laurent Morlet 25 Jan

16 Choices had to be made Project split in 3 phases Number of metrics trimmed down to <10 Prioritise metrics where data is already captured Spend time designing a new but powerful datamart Datamodel intended to live beyond this project Informatica ETL (Extraction-Transformation-Load) layer Enterprise Cognos package Cinical Metrics Dashboard by Laurent Morlet 25 Jan

17 Oracle Cognos Tablespace - GCO Shared Dimensions Framework (AGORA) Reports (AGORA) AGORA User Source System 1 Source System 2 ETL Tablespace - Facts (AGORA) Tablespace - AGORA Dimensions Tablespace - Facts (SMART) Tablespace - Dimensions (SMART) Framework (Balanced Scorecard) Framework (SMART) Reports (Balanced Scorecard) Metrics (Balanced Scorecard) Reports (SMART) Access Manager BSC User SMART User Source System n Tablespace - Facts (???) Tablespace - Dimensions (???) Framework (???) Reports (???)??? User Business Area IT Team (e.g. IDS) Joint Business Area IT Team and CBS2 Business Area IT Team with CBS2 assistance CBS2 ITS Cinical Metrics Dashboard by Laurent Morlet 25 Jan

18 Speed metric : GCO wide recruitment % KPI objective: follow-up on recruitment status against plan # of enrolled subjects over # of subjects expected to be enrolled at this moment in time Start to measure when First Subject Enrolled Expected is reached until Last Subject Enrolled Roll up mechanism is recalculation not averaging Metrics based on all ongoing studies green >90%, 80%< orange <90%; red<80% Additional reports available with the following dimensions: R&D company; Therapeutic area; Development program; region; country; below target; in House vs CRO Recruitment speed % Cinical Metrics Dashboard by Laurent Morlet 25 Jan

19 Speed metric : GCO wide recruitment % Cinical Metrics Dashboard by Laurent Morlet 25 Jan

20 Speed metric 3: First Subject consented on target KPI objective: evaluate trial startup processes First Subject Consented at Protocol level First Subject Consented Expected vs First Subject Consented Actual at protocol level At protocol level the value is YES on or before target date; NO past target date Metric is expressed as a percentage of trials with first subject consented on target Metric based on the last 12 months (rolling), historical quarterly data available green >90%, 80%< orange <90%; red<80% One metric for Ph.I Healthy volunteers and another for combined Ph.I, II, III in patients FPconsented on Target Cinical Metrics Dashboard by Laurent Morlet 25 Jan

21 New Metric: Ongoing subjects Views/Reports available Dimensions available: In House/CRO Therapeutic Area Clinical Phase Emerging/Non emerging R&D Company Year Cinical Metrics Dashboard by Laurent Morlet 25 Jan

22 Metric Name % (Sample size) Cinical Metrics Dashboard by Laurent Morlet 25 Jan

23 Cinical Metrics Dashboard by Laurent Morlet 25 Jan

24 Cinical Metrics Dashboard by Laurent Morlet 25 Jan

25 Cinical Metrics Dashboard by Laurent Morlet 25 Jan

26 What came to light Quality of data deep down in business systems not good enough Often the real data meant good performance but it was not recorded or not on time (re-writing history?). Disconnect between local/ta and management knowledge of performance Supporting data to establish more realistic objectives. Some cultural backgrounds more inclined to see this as tool to blame rather than increasing visibility and performance Cinical Metrics Dashboard by Laurent Morlet 25 Jan

27 Additional collateral initiatives Identify a set of compulsory and critical fields in source systems Communicate and train on this set Create some quality check reports on source system Extensive training on the Dashboard about contributing fields and calculation methods Complexity of calculations Cinical Metrics Dashboard by Laurent Morlet 25 Jan

28 Net Result Ability to combine information from many source systems Historical trending Visibility of information for anybody anywhere anytime Justification to timely collect good data in so many systems Cinical Metrics Dashboard by Laurent Morlet 25 Jan

29 Successful result meant more Dashboards Quality Risk Management Operational reporting Quality Assurance etmf compliance Trial Master Source Cinical Metrics Dashboard by Laurent Morlet 25 Jan

30 Thank you and let s hope our organizations Are not as cynical as Dogbert. Cinical Metrics Dashboard by Laurent Morlet 25 Jan

31 Implementation of a clinical metrics Dashboard Case study from a large pharma ICR Symposia 28 Feb 2012 Janssen R&D, Global Clinical Operations Laurent Morlet Pharm.D, Program Manager Clinical IT