Crea%ng Value from Data Governance, Ethics, and Appropriate Use

Crea%ng Value from Data Governance, Ethics, and Appropriate Use Thomas P. Scarnecchia VP & CTO Digital Aurora & Execu<ve Director Observa<onal Medical Outcomes Partnership at the Founda<on for the Na<onal Ins<tutes of Health

Conflict of Interest Disclosure Thomas Scarnecchia, MS Has no real or apparent conflicts of interest to report. 2/20/12 Scarnecchia - OMOP 2

Learning Objectives Describe the Observational Medical Outcomes Partnership's research program and the governance and technology that enable it Identify lessons learned from establishing the partnership's research laboratory and network Describe insights from the Observational Medical Outcomes Partnership's research into the use of observational data 2/20/12 Scarnecchia - OMOP 3

Observational Medical Outcomes Partnership (OMOP) Public-Private Research Partnership established to inform the appropriate use of observational healthcare databases for studying the effects of medical products: Conducting methodological research to empirically evaluate the performance of alternative methods on their ability to identify true associations Developing tools and capabilities for transforming, characterizing, and analyzing disparate data sources across the health care delivery spectrum Establishing a shared resource so that the broader research community can collaboratively advance the science 2/20/12 Scarnecchia - OMOP 4

The Partnership Established in 2007 as public-private research partnership between FDA, PhRMA, and FNIH Robust governance model with broad stakeholder representation across two advisory boards and an executive board Diverse research community Open and transparent research culture Growing open source community around the OMOP framework, technology, and methods 2/20/12 Scarnecchia - OMOP 5

The Observational Medical Outcomes Partnership is funded by the Foundation for the National Institutes of Health through generous contributions from the following: AbboO Amgen Inc. AstraZeneca Bayer Healthcare Pharmaceu<cals, Inc. Bristol- Myers Squibb Eli Lilly & Company GlaxoSmithKline Johnson & Johnson Lundbeck, Inc. Merck & Co., Inc. Novar<s Pharmaceu<cals Corpora<on Pfizer Inc. Pharmaceu<cal Research Manufacturers of America (PhRMA) Roche sanofi- aven<s Schering- Plough Corpora<on Takeda. 2/20/12 Scarnecchia - OMOP 6

Partnership Structure Governance Provided by an Executive Board Scientific and Informatics advisory boards to inform decisions FNIH Board Scien%fic Advisory Board Healthcare Informa%cs Advisory Board OMOP Execu%ve Board Execu%ve Director Program Management Office Key Design Elements: 1. Governance and Oversight 2. Research Leadership 3. Program Management 4. Partners & Collaborators Key Oversight Principal Inves%gators Research Team Research Compu%ng Team Dedicated Industry, academic and other external 2/20/12 resources Scarnecchia - OMOP 7

Executive Board Oversees Partnership Operations Janet Woodcock, MD - FDA Rebecca Burkholder -The National Consumers League Sherine Gabriel, MD, MSc - The Mayo Clinic Jesse L. Goodman, MD, MPH FDA Stephen Jacobsen, MD, PhD - Southern California Permanente Medical Group Ronald L. Krall, MD - Retired GSK Richard Platt, MD, MSc - Harvard Medical School and Harvard Pilgrim Health Care Brian Strom, MD, MPH - Pennsylvania School of Medicine David Wheadon, MD - PhRMA Marcus Wilson, Pharm.D. - HealthCore 2/20/12 Scarnecchia - OMOP 8

Scientific Advisory Board Independent review of and expert input into the scientific aspects of OMOP s activities Healthcare Informatics Advisory Board Independent review and expert input into the technology, privacy, data model, and terminology Elizabeth Andrews, RTI Health Solutions Andrew Bate, Pfizer Jesse Berlin, Johnson & Johnson Robert Davis, Kaiser Permanente Sean Hennessy, University of Pennsylvania Mike Katz, FDA patient representative Allen Mitchell, Boston University David Page, University of Wisconsin Judy Racoosin, FDA Ken Rothman, RTI Health Solutions Judy Staffa, FDA Jeff Brown, Harvard Medical School Stan Huff, Intermountain Healthcare Diane MacKinnon, IBM (retired) Ken Mandl, Harvard University Clem McDonald, National Library of Medicine Mitra Rocca, FDA Rob Thwaites, United BioSource Corporation 2/20/12 Scarnecchia - OMOP 9

Research Investigators The lead scientists for the OMOP project who guide and participate in the research across all project phases Management Team FNIH provides program management, grants management, and operational support. William DuMouchel Chief Statistical Scientist, Oracle Health Sciences Abraham G. Hartzema, PharmD, MSPH, PhD, FISPE Professor and Eminent Scholar Pharmaceutical Outcomes & Policy, Perry A. Foote Chair in Health Outcomes Research, University of Florida College of Pharmacy Patrick Ryan, PhD Associate Director, Analytical Epidemiology, Johnson & Johnson Pharmaceutical Research and Development Martijn Schuemie, PhD Assistant Professor, Erasmus University Medical Center of Rotterdam Visiting Research Scientist, Columbia University Executive Director Thomas Scarnecchia, MS VP & CTO, Digital Aurora Program Managers Emily Welebob, RN, MS Christian Reich, MD, PhD David Madigan, PhD Professor of Statistics, Columbia University J. Marc Overhage, MD, PhD Chief Medical Informatics Officer, Siemens Health Services Paul Stang, PhD Senior Director, Epidemiology, Johnson & Johnson Pharmaceutical Research and Development Marc Suchard, MD, PhD Professor, University of California, Los Angeles 2/20/12 Scarnecchia - OMOP 10

Current Funded Collaborations Organiza<on Project Erasmus University Medical Center Detec<on of Long Term Adverse Drug Reac<ons in Electronic HealthCare Data using OSIM 2 and OMOP Laboratory Data with the Longitudinal Evalua<on of Observa<onal Profiles of Adverse Events Related to Drugs [LEOPARD] method. Indiana University Uppsala Monitoring Centre MassachuseOs General Hospital Auburn University Soon to be announced Cohort Design Method enhancements IC Temporal PaOern Discovery Method Improvements and In- Depth Analysis of Clinical Outcomes of Interest Valida<ng OMOP Results with Reproducible Detailed Data Inves<ga<on (An<bio<cs & Acute Renal Failure, Typical An<psycho<cs & GI Ulcer Hospitaliza<on, and Warfarin & Hip Fracture) Developing a Structured Process for Measuring and Interpre<ng Health Outcomes of Interest in the OMOP Common Data Model (star<ng with Acute Liver Injury) Extending OMOP across seven sites in Europe 2/20/12 Scarnecchia - OMOP 11

OMOP Data Community Designed to Test Different Data Types Research Laboratory Coordinating Center Five Central Databases GE Thomson Reuters Distributed Partners* Department of Veterans Affairs Miami-Humana Health Services Research Center Partners Healthcare Regenstrief Institute SDI 178 million persons with pa%ent- level data 5.4 billion drug exposures 5.8 billion procedures 2.3 billion clinical observa%ons * Funded through Q1 2011 2/20/12 Scarnecchia - OMOP 12

Open Source Collaborations Extending the OMOP Common Framework for CER related research Scalable Architecture for Federated Translational Inquiries Network (SAFTINet) Scalable National Network for Effectiveness Research (SCANNER ) 2/20/12 Scarnecchia - OMOP 13

OMOP Technology Secure research computing laboratory and network of data partners with access to observational data representing nearly 200 million patients Stable framework for organizing, characterizing, and analyzing disparate data sources across a network of healthcare and insurance providers Process and technology to access the quality of a data source for use in observational research Growing portfolio of tested and deployed analysis methods within the OMOP Research Lab and other data environments 2/20/12 Scarnecchia - OMOP 14

2/20/12 Scarnecchia - OMOP 15

OMOP Tool Common Data Model (CDM) & Standardized Terminologies Health Outcomes (Defini%ons) Applica<on Analy<cal model to organize disparate types of data into a common format Conceived for ac<ve medical product surveillance, but extensible for other use cases Op<mized for large- scale analy<cs Implemented using standardized terminologies An open- source library of 35 HOI defini<ons for use in observa<onal studies Methods Analy%cs 14 methods Epidemiology designs Sta<s<cal approaches adapted for longitudinal data Observa%onal Medical Dataset Simulator (OSIM) Observa%onal Source Characteris%cs Analysis Report (OSCAR ) OSIM generates simulated longitudinal pa<ent records Models complexi<es found in real- world healthcare data organized in the OMOP CDM format Generates summary sta<s<cs from observa<onal datasets Valida<on of transforma<on from raw data to OMOP common data model Comparison of overall database to specific subpopula<ons of interest (such as people exposed to a par<cular drug or people with a specific condi<on) Providing context for interpre<ng and analyzing findings of drug safety studies 2/20/12 Scarnecchia - OMOP 16

OMOP Tool Natural History Analysis (NATHAN) Regularized Iden%fica%on of Cohorts (RICO) Generalized Review of OSCAR Unified Checking (GROUCH) Applica<on An extension of OSCAR, where data characteris<cs can be produced for a par<cular subpopula<on of interest Exposed popula<on (e.g. pa<ents taking an<bio<cs) Cases (e.g. pa<ents with acute liver injury) Exposed cases (e.g. pa<ents taking an<bio<cs who develop acute liver injury) Addi<onal NATHAN summary sta<s<cs provide context for interpre<ng and analyzing findings of drug safety studies Use NATHAN to refine HOI algorithms in ac<ve surveillance Standardizes pa<ent cohort selec<on Pa<ents mee<ng the criteria can be automa<cally and rapidly selected from any database conforming to the OMOP CDM Data quality summary report Produces a report for each data source with warnings of implausible and suspicious data observed from the OSCAR summary Allows for data quality review of specific drugs (such as the ingredients that comprise the OMOP drugs of interest) or specific condi<ons 2/20/12 Scarnecchia - OMOP 17

*RxNorm *SNOMED- CT *LOINC h_p://omop.fnih.org/cdmandterminologies Developed with broad stakeholder input Designed to accommodate disparate types of data (claims and EHRs) Optimized to use case of standardized large-scale analytics Conceived for active medical product surveillance, but extensible for other use cases Applied successfully across OMOP data community Standards-based, conforming to ONC Meaningful Use Stage 2 recommendations * 2/20/12 Scarnecchia - OMOP 18

Proposed OMOP CDM v3 Person Provider Observa<on_period Payer_plan_period Loca<on Visit_occurrence Drug_exposure Care_site Organiza<on Drug_era Drug_cost Condi<on_occurrence Condi<on_era Procedure_occurrence Procedure_cost Observa<on Health Outcomes of Interest Drugs of Interest Interven8ons 2/20/12 Scarnecchia - OMOP Cohort Death Standardized Vocabulary Illustra<ve version: Refer to CDMv3 document for formal specifica<on 19

Standardizing condi<ons: Standardizing drugs: h_p://omop.fnih.org/vocabularies 2/20/12 Scarnecchia - OMOP 20

Disparate Sources Drug Codes Normalized to RxNorm Enabling Standardized Tools and Methods Thomson NDC Ingredient- based eras Drug class defini<on GE VA GPI VA Product GPPC NDF- RT RxNorm Clinical drug Branded drug Ingredient Form Strength HOI defini<ons Queries Quality control Characteriza<on tools Database benchmarking PHS Multum Indica<on based methods Comparison of analy<cal methods GPRD Mul<lex FDB 2/20/12 Scarnecchia - OMOP 21

Method Name DISPROPORTIONALITY ANALYSIS Dispropor<onality analysis (DP) IC Temporal PaOern Discovery (ICTPD) HSIU Cohort Method (HSIU) Longitudinal Gamma Poisson Shrinker (LGPS) & Longitudinal Evalua<on of Observa<onal Profiles of Adverse events Related to Drugs (LEOPARD) CASE- BASED METHODS Bayesian Mul<variate Self- Controlled Case Series (MSCCS) Mul<- set Case Control Es<ma<on (MSCCE) Bayesian Logis<c Regression (BLR) Case- control Surveillance (CCS) Case- crossover (CCO) EXPOSURE- BASED METHODS Observa<onal Screening (OS) High- dimensional Propensity Score (HDPS) Contributor Columbia / Merck Uppsala Monitoring Centre Indiana University / Regenstrief Ins<tute Erasmus University Medical Center RoOerdam Columbia University / UCLA Columbia University / GlaxoSmithKline Rutgers / Columbia University Lilly University of Utah ProSanos UBC / GlaxoSmithKline Harvard Medical School / Columbia Incident User Design (IUD) University of North Carolina SEQUENTIAL TESTING METHODS Maximized Sequen<al Probability Ra<o Test (MSPRT) Harvard Pilgrim / Group Health Condi<onal Sequen<al Sampling Procedure (CSSP) Harvard Pilgrim / Group Health 2/20/12 Scarnecchia - OMOP 22

Research Lab FTP Results CDM Source Results DB Libraries Community Website Lab team posts Methods to website library Distributed Partners retrieve Methods Distributed Partners Coordinating Center announces availability of new method Secure Collaboration Website Discussions and document exchange 2/20/12 Scarnecchia - OMOP 23

Initial Research Objectives Define and test a pool of analytical methods that can be used to explore the relationships between drugs and health-related conditions across multiple types of observational data (administrative claims, inpatient and outpatient EHRs). Develop and test methods to apply to a network of central and distributed data sources for drug safety and effectiveness questions Assess the performance of the analytical methods Based on the results of these analyses, determine how the results can shape the implementation of an active drug surveillance program. 2/20/12 Scarnecchia - OMOP 24

Varia%on across data sites: prevalence of all diseases in one site vs. the network 2/20/12 Scarnecchia - OMOP 25

Distribution of estimates across all drug-outcome pairs True - False - False + True + False nega%ves: Bisphosphonates GI Ulcer hospitaliza<on Tricyclic an<depressants Acute myocardial infarc<on An<bio<cs Acute liver injury Warfarin- Bleeding Each method has a different es<mated distribu<on impac<ng its opera<ng characteris<cs False posi%ves: Typical an<psycho<cs Acute renal failure Typical an<psycho<cs GI Ulcer Hospitaliza<on Beta blockers Hip fracture An<epilep<cs Acute renal failure An<bio<cs Acute renal failure An<bio<cs Aplas<c anemia Amphotericin B Acute liver failure Amphotericin B Aplas<c anemia CCO, CCS are posi<vely biased across pairs False posi<ves and false nega<ves are not consistent across methods 2/20/12 Scarnecchia - OMOP 26

OMOP Initial Summary A risk identification system can complement current practice by providing evidence to support a comprehensive safety assessment No one clear best method, as it depends on tolerance for false positives vs. false negatives In this experiment, active surveillance methods achieved: At 50% sensitivity, false positive rate ranges 16%-30% At 10% false positive rate, sensitivity ranges 9%-33% Need to be cautious in interpreting results from single method in single database Replication does not necessarily provide complete confidence Further empirical research needed to have more complete understanding of operating characteristics before widespread adoption (Current year 3) 2/20/12 Scarnecchia - OMOP 27

Ongoing Research Priorities: Advance methodological research to explore the performance of methods over time, within specific populations of interest, and across a broader array of medical products and health outcomes Refine and enhance OMOP s tools and capabilities to translate research into practice Sustain the shared resource (research lab) so the research community maintains an open forum for collaborative research Develop approaches to incorporating benefits including increased application of clinical data to help ascertain benefits Additional research with these methods moves us from risk identification to risk refinement with same empiricism 2/20/12 Scarnecchia - OMOP 28

Drug- outcome pairs + European replica<on Improve HOI defini<ons Explore false posi<ves Evaluate study design decisions (EDDIE) Observa%onal data Methods development Methods enhancements Mul8variate self- controlled case series Increased parameteriza<on Case- control, new user cohort designs Applica<on of exis<ng tools ICTPD, OS, LGPS, DP Expand CDM for addi<onal use cases Real- world performance: Thomson MarketScan GE + OMOP Distributed Partners + EU- ADR network Simulated data: signal Strength (RR) Type (<ming) OSIM2 popula<on size data type Claims vs. EHR Privately insured vs. Medicare vs. Medicaid 1/10/50m pa<ents 2/20/12 Scarnecchia - OMOP 29

Lessons From Establishing the Research Laboratory and Distributed Network Policy level Process level Engineering Decision making Communications Research Sites Contracts OMOP Lab Technology From the OMOP Experiments Observational Data Performance of Methods 2/20/12 Scarnecchia - OMOP 30

Creating Value Providing an Empiric Compass Identifying the boundaries and limits of these data and methods Providing empirical evidence on performance of observational data and methods Providing a learning laboratory Methodological Research Research on Observational Data & Analysis Frameworks Integral public health research endeavor of importance to the field and Regulatory Science 2/20/12 Scarnecchia - OMOP 31

Thomas P. Scarnecchia Vice President and Chief Technology Officer tpscarnecchia@digitalaurorainc.com 802 362 8111 Execu<ve Director, Observa<onal Medical Outcomes Partnership tscarnecchia@fnih.org hop://omop.fnih.org 2/20/12 Scarnecchia - OMOP 32