CITI PROGRAM WWW.CITIPROGRAM.ORG GCP TRAINING PROGRAM AND CITI PROGRAM LMS Paul Braunschweiger Ph.D. Professor, Radiation Oncology CoDirector, Ethics Component, University of Miami CTSI CoFounder & CEO, CITI Program at the University of Miami University of Miami Miller School of Medicine Enhancing Clinical Research Professionals Training & Qualifica8on Project
What is The CITI Program www.ci=program.org v CITI Program is subscrip8on based, online, research ethics educa8on program >2,100 par8cipa8ng ins8tu8ons worldwide n >72,000 learners per month complete a CITI Course September 2014, 124,064 people /170,110 courses n 58% are first 8me learners to CITI n 65-70% are students and fellows n GCP Course 64% clinical, 30% students v Subscriber Profile Universi8es, colleges, medical centers, community hospitals, socie8es, government, networks (e.g., N2, HANC), commercial IRBs, industry, and interna8onal organiza8ons v Courses in all areas of research ethics Good Clinical Prac8ce www.citiprogram.org
CITI Program GCP Course v v CITI ICH GCP Course is basic inves8gator GCP training CITI Basic Course in HSR + CITI GCP course provide a solid founda8on for a programma8c approach to clinical research ethics training
CITI Program GCP Course Stats v ~7,100 CITI GCP courses completed per month v 893 (43%) Organiza8ons use a CITI GCP Course v >65,300 CITI GCP courses completed 2014 YTD v All CTSIs use the CITI Program to meet Federal and ins8tu8onal HSR training mandates All CTSI have access to CITI GCP courses 56 CTSIs currently use the CITI Program GCP course
Top 20 CITI GCP Course Users 2014 YTD R # Organiza8on 1 1993 Quorum Review 2 1724 WIRB/CG 3 1219 UT Southwestern Medical Center 4 1087 Icahn School of Medicine at Mount Sinai 5 864 Greater Cincinna8 Academic and Regional Health Centers. 6 860 Thomas Jefferson University 7 859 The Society for Clinical Research Sites 8 858 University of California, Davis 9 740 University of California, San Francisco 10 708 Sarah Cannon Research Ins8tute R # Organiza8on 11 675 University of Southern California 12 669 Emory University 13 663 University of Pennsylvania 14 654 Schulman Associates IRB, Inc. 15 636 St. Michael's Hospital 16 608 University of North Carolina at Chapel Hill 17 586 Department of The Navy 18 572 Drexel University 19 558 University of Pilsburgh 20 545 University of Wisconsin - Madison
Top 20 CITI GCP Course Users 2014 YTD R # Organiza8on 1 1993 Quorum Review 2 1724 WIRB/Copernicus Group 3 1219 UT Southwestern Medical Center 4 1087 Icahn School of Medicine at Mount Sinai 5 864 Greater Cincinna8 Academic and Regional Health Centers. 6 860 Thomas Jefferson University 7 859 The Society for Clinical Research Sites 8 858 University of California, Davis 9 740 University of California, San Francisco 10 708 Sarah Cannon Research Ins8tute R # Organiza8on 11 675 University of Southern California 12 669 Emory University 13 663 University of Pennsylvania 14 654 Schulman Associates IRB, Inc. 15 636 St. Michael's Hospital 16 608 University of North Carolina at Chapel Hill 17 586 Department of The Navy 18 572 Drexel University 19 558 University of Pilsburgh 20 545 University of Wisconsin - Madison 12 of top 20 are CTSIs 3/20 Commercial IRBs, CRO Government Agency
Talking Points CITI Program Research Ethics Educa=on I. CITI GCP Training Program Characteris=cs II. CITI Program Learner Experience III. CITI Program Ins=tu=onal U=lity IV. CITI 2.0 Learning Management System Q&A
I. Training Program Characteris=cs A. Program format B. Course development C. Competency based learning assessment D. Module focus
I. Training Program Characteris=cs A. Program Format v CITI GCP courses are text based learning experiences delivered on the CITI 2.0 LMS v Balanced presenta8on including use of bullets and lists for easy reading and comprehension v Case studies, have recently been added to GCP modules v Quizzes v Links to regulatory and guidance documents as well as other resources
Example
Module Example Audits and inspec8ons
Example module Audits and inspec8ons
I. Training Program Characteris=cs B. CITI Program Course Development v Course development CITI Program Advisory Commilee (PAC) 12 members " Course/Module development proposal submiled " PAC reviews and approves development " Program Director recruits expert consultants to develop the course/module
I. Training Program Characteris=cs B. Course Development v Peer Review Process 2-3 experts recruited by Program Director Consensus is sought Copy editor Author approves changes Web designer posts materials Author and CITI staff approve Roll out
I. Training Program Characteris=cs B. Professional development v Professional enhancement Dr. Arango member of the CTTI GCP Working Group CITI ICH/GCP course meets the minimum TransCelerate criteria for GCP training for site personnel CITI content staff par8cipate in professional conferences PRIMR what s New at the CITI Program
I. Training Program Characteris=cs C. Competency- based learner assessment v Post module quizzes associated with each GCP module Quizzes considered a teaching exercise Modules are designed to require about 25-30 minutes for the average reader Quizzes require 4-5 minutes Links for feedback to the developers
I. Training Program Characteris=cs C. Competency- based learner assessment v The Quizzes Quizzes based on the L.O.s Correct responses and explanatory feedback are provided aqer submission of each quiz Ins8tu8ons set the bar, 75% to 100% correct Learners may return to the module to review materials and retake the quiz to alain a higher score
I. Training Program Characteris=cs Quiz Ques=ons
I. Training Program Characteris=cs Quiz feedback
I. Training Program Characteris=cs D. Module focus - 3 courses 1. GCP Course for Trials With Inves=ga=onal Drugs and Biologics (ICH focus) n Meets TransCelerate Group requirements n English, Spanish & Portuguese 2. CITI Good Clinical Prac=ce Course (US FDA focus) n English, Spanish, Portuguese, Chinese, Thai, Khmer, Vietnamese 3. GCP Course for Clinical Trials Involving Inves=ga=onal Medical Devices 4. Interna=onal Courses n Canada GCP, CITI Canada (meets TransCelerate criteria) n India GCP, CITI India
II. Learner Experience A. Usability B. Accessibility C. User Support
II. Learner Experience A. Usability
II. Learner Experience B. Availability and accessibility Available world wide. 24/7/365 Browser Frequency Browser Chrome MSIE Safari Firefox Android Opera Silk others Total Usage 37.8% 27.8% 17.7% 16.1% 0.3% 0.1% 0.1% 0.1% 100.0%
II. Learner Experience C. Learner support v Excellence in technical support through the CITI Help Desk Learner support via email. Chat line, phone calls Ins8tu8onal Coordinator support to: n Choose and set up curriculum n Modify curriculum\ add new courses to ins8tu8onal menu n Provide specialty reports
II. Learner Experience. C. Learner support - CME v CME and CEU available for all GCP courses v U of Miami Div. of CME Physicians, Cat 1 Nurses Psychologists
II. Learner Experience D. Learner Feedback to The Developers v Mul8ple opportuni8es for the learner v Links available to submit a comment: Aqer comple8ng a specific module Aqer comple8ng a course Via helpdesk, email, phone, and chat sessions
II. Learner Experience D. Learner Feedback to The Developers v Course Learner Sa8sfac8on Survey Voluntary, Anonymous, Lime Survey tool, RR=~30% Pre- course Survey n Experience in HSR n Expecta8ons Post- course Survey n Follow up on expecta8ons n Naviga8on issues n U8lity of the course to learner n Value of the course to the learner
Value Example: Rate the CITI GCP Course (9 months,2014; n=19,317; RR =~ 33%) 35% Frequency of Response 30% 25% 20% 15% 10% 5% 0% 1% 1% 2% 2% 10% 10% 23% 29% 23% 1 2 3 4 5 6 7 8 9 Poor 75% Outstanding
III. Ins=tu=onal U=lity A. Costs B. Ins8tu8onal Usability C. Customizability D. No8fica8ons University of Miami
III. Ins=tu=onal U=lity A. Cost v No Extra Cost for GCP if organiza8on is a CITI subscribing member organiza8on All CTSI organiza8ons are CITI subscribers v $3,000 annual subscrip8on, unlimited use basic package Includes: HSR, GCP, CoI, RCR, ACU v Course / module selec8ons and setup guided by CITI HD Tech
III. Ins=tu=onal U=lity B. Usability v CITI Integrates with local systems/databases for administra8ve/irb func8ons reducing administra8ve burden Incorporates training status into Electronic Protocol Submission and Tracking systems (e.g., Click Commerce) v SSO access available via Shibboleth (In Commons Ini)a)ve) v Manual or automa8c download of stored training data Customizable formats Available 24/7 v All organiza8ons have someone responsible for compliance training and CITI data review
III. Ins=tu=onal U=lity C. Customizability v All CITI courses are modular for customiza=on. Mandatory, elec8ve, and op8onal module selec8on op8ons All GCP course modules recommended as mandatory Seamless blending of organiza8on- specific modules to address: n Organiza8onal policies and procedures n Country- specific requirements (e.g., Canada, India) Custom training reports to the ins8tu8on n Customize data downloaded n Manual or automa8c downloads n SSO implementa8on
III. Ins=tu=onal U=lity D. No=fica=ons v CITI soqware sends training reports to: Learner n Automated and Manual Compliance / Training coordinators n Automated and Manual Ins8tu8onal training databases n Manual and automated downloads (automated queries, SSO) n Can be modified according to Ins8tu8onal policy/needs n 24/7/365
III. Ins=tu=onal U=lity D. No=fica=ons v Ins8tu8onal no8fica8ons Email updates to coordinators n New courses, updates, modifica8ons, new CITI procedures as they occur Conference exhibits and presenta8ons e.g., PRIMR What s New at the CITI Program Mass emails to subsets of CITI learners n Ins8tu8on subsets and course groups (e.g. GCP) Quarterly CITI Newsleler n Ins8tu8onal Coordinators n CITI Program Homepage
IV. IT Pla`orm A. IT Overview B. Capacity C. Program improvements D. Security E. System safeguards
IV. IT Pla`orm A. IT Overview v CITI IT development group Dir. IT Development 1 Senior Systems Analyst 2 Senior systems programmers 2 web designers Contractors (e.g, M&K Tech Group) 10 help desk techs v Custom Playorm Soqware CRADA with the VA Easily modified Compa8ble with many presenta8on modes
IV. IT Pla`orm A. IT Overview Availability & Reliability v Servers located at University of Miami Datacenter and Commercial Datacenter (off site) share hos8ng environments v Intensive and extensive University network security (Firewalls, Intrusion Detec8on and Preven8on Systems IDPs) v Courses are available 24/7/365 v High capacity bandwidth Internet 2 and LambdaRail v Redundancy in all components
Production Web Server 1 Production Web Server 2 Production Web Server 3 Production Web Server 4 Reporting / Survey Server 1 Reporting / Survey Server 2 Prod Switch 1 Prod Switch 2 Prod SQL DB Server 1 Prod SQL DB Server 2 Storage Area Network (current database) INTERNET CITI Server configura=on at U of Miami Staging Web Server 1 Develop Web Server 1 Failover Web Server 1 Failover Web Server 2 Failover Web Server 3 Staging SQL Server 1 (recent database copy) Development-SQL Server 1 (recent database copy) Failover SQL Server 1 (sync d database copy) Failover Web Server 4
IT Overview B. Capacity Capacity More than 1.5 million courses per year At least 124,000 Learners per month At least 174,000 courses per month (9-14) Capacity for > 2,000 concurrent users Upgrades pending to boost >~3,000 n Current usage varies, max. about 1,200 to date
IT Overview C. Code upgrades and improvements v Ongoing ac8vity Make code more efficient and secure v SoP for development, tes8ng, implementa8on, and go- live E.G., New CME code in Aug 2014 n Makes obtaining CEU/CME much easier for learner n Much more efficient, more secure, and easier to maintain v Current CITI ver 2.0, August 2013 v Currently planning ver. 3.0 August 2016
IT Pla`orm D. Security and Availability v Who has DB access? Role based access to CITI admin components Role based access for ins8tu8ons to their site specific data Periodic internal and external audits of code Problem, Change and Project management in accord with the ITIL standards Log all user events (page views, quiz comple8ons) Forensics to assess fraudulent use when required
IV. IT Pla`orm D. Security and Availability v Extensive monitoring soqware for all aspects of opera8ons v Redundant components in all elements of system architecture v Excess produc8on capacity to meet peak loads v Separate development, tes8ng, produc8on and disaster recovery environments v Offsite backups for data archiving
Summary CITI Program at the University of Miami v 14 years delivering web based research ethics training to the research community v Academic NFP program v 3 GCP courses & interna8onal device course v Text and cases v Wrilen, peer reviewed and updated by recognized experts in GCP v High learner sa8sfac8on with the CITI GCP courses
Summary CITI Program at the University of Miami v ALL CTSIs have access to CITI Program Lille or no addi8onal cost to NCATS v Easy for learners to access and use v CITI playorm designed with the needs of the learner and coordinators in mind v Outstanding HD customer service and high service level Queries and reports easy to obtain as needed Sufficiently robust to accommodate >2,000 concurrent learners Program is scalable Addi8onal capacity easily added as needed
CITI Program At the University of Miami www.ci8program.org GCP TRAINING PROGRAM AND CITI PROGRAM LMS ACKNOWLEDGEMENTS
CITI GCP Authors and Peer Reviewers Name Degree GCP Years Affiliation GCP Consultant Authors Deborah Rosenbaum MS 2004-2005 Clinical Research Consultant, Sarrison Clin. Res., Chair ACRP Academy Board of Trustees. 2014 Cynthia Gates J.D. CIP 2010-2011 Western IRB., Currently at UC Davis. GCP Peer Reviewers Karen Hansen BA 2004 2012 IRB Director, The Fred Hutchison Cancer Center Paul Braunschweiger Ph.D. 2004 2005 CITI Program at the University of Miami Sally Mann MS 2004 2011 IRB Director at University of Nebraska Medical Center Allison Handler MS 2004 2009 Duke University Sue Avery MS 2007 2009 Duke University Gary Chadwick Pharm. D. 2009 2013 University of Rochester Gwenn Oki MPH 2009 2011 City of Hope, Van Andel Institute David Forester JD 2009 2011 Western IRB Melody Lin Ph.D. 2009 2012 Deputy Director at OHRP, International Studies Ann Taylor BS 2011 Pfizer at the time Mary Jane Clark BS 2011 Pfizer at the time Chiu Lin Ph.D. 2012 2014 Lin & Associates, previously with FDA Holly Scott BA 2012 Currently with Biologics, formerly with the FDA Susan Miller MD 2012 2014 The Methodist Hospital, Houston Diane Paul MSRN 2012 2014 NCCN Jaime Arango Ed.D. 2012 2014 CITI Program at the University of Miami
Ques=ons and Answers Sugges=ons v List of GCP course module 8tles v New CITI Program courses under development CRC v CITI Program org. chart
CITI PROGRAM WWW.CITIPROGRAM.ORG GCP TRAINING PROGRAM AND CITI PROGRAM LMS Paul Braunschweiger Ph.D. Professor, Radiation Oncology CoDirector, Ethics Component, University of Miami CTSI CoFounder & CEO, CITI Program at the University of Miami University of Miami Miller School of Medicine Enhancing Clinical Research Professionals Training & Qualifica8on Project
COURSE MODULES
GCP Course for Trials With Inves=ga=onal Drugs (interna=onal/ ICH focus). TransCellerate approved 1. The CITI Good Clinical Prac8ce Course for Clinical Trials Involving Drugs and Biologics 2. Conduc8ng Inves8gator- Ini8ated Studies According to FDA Regula8ons and GCP 3. Inves8gator Obliga8ons in FDA- Regulated Research 4. Managing Inves8ga8onal Agents According to GCP Requirements 5. Informed Consent in Clinical Trials of Drugs and Biologics
GCP Course for Trials With Inves=ga=onal Drugs (interna=onal/ ICH focus). TransCellerate approved 6. Monitoring Clinical Trials of Drugs by Industry Sponsors 7. Audits and Inspec8ons of Clinical Trials of Drugs and Biologics 8. Overview of New Drug Development 9. Overview of ICH GCP 10. Detec8ng and Evalua8ng Adverse Events
CITI ORGANIZATIONAL CHART
CITI Program Org. Chart University of Miami Office of the Provost UM Innovation EAB Dir. IT Dev. Web design CITI CEO PAC v v Assoc. Dir. for Program/Chair PAC Dir. interna8onal programs
NEW CITI COURSES COMING SOON
New CITI Program Courses Currently Under Development v GCP refresher v CRC Course v Phase 1 trials modules for GCP and HSR courses v The 3 modules for Community Based Par8cipatory Research/Community Engagement v GLP* v HECM* * For individual learners
CITI PROGRAM INTERNATIONAL DEVICE COURSE
GCP Course for Clinical Trials Involving Inves=ga=onal Medical Devices Interna=onal focus. 1. The CITI Good Clinical Prac8ce Course for Clinical Trials Involving Inves8ga8onal Medical Devices 2. Conduc8ng Inves8gator- Ini8ated Clinical Trials of Devices 3. Inves8gator Obliga8ons in FDA- Regulated Clinical Research of Devices 4. Managing Inves8ga8onal Devices According to GCP Requirements 5. Overview of U.S. FDA Regula8ons for Medical Devices 6. Informed Consent in Clinical Trials of Devices
GCP Course for Clinical Trials Involving Inves=ga=onal Medical Devices Interna=onal focus. 7. Monitoring Clinical Trials of Devices 8. Audits and Inspec8ons of Clinical Trials of Devices 9. Repor8ng Requirements for Device Studies 10. Comple8ng the CITI GCP Course 11. Repor8ng Serious Adverse Events in Inves8ga8ons of Drugs and Biologics 12. ICH Comparison Between ICH GCP E6 and U.S. FDA Regula8ons 13. Comple8ng the CITI GCP Course
The CITI Good Clinical Prac=ce Course for Clinical Trials Involving Drugs and Devices FDA Focus 1. Overview of New Drug Development 2. Overview of ICH GCP 3. ICH Comparison Between ICH GCP E6 and U.S. FDA Regula8ons 4. Conduc8ng Inves8gator- Ini8ated Studies According to FDA Regula8ons and GCP 5. Inves8gator Obliga8ons in FDA- Regulated Clinical Research 6. Managing Inves8ga8onal Agents According to GCP Requirements
MORE SURVEY DATA
Pre- Course Survey Data How do you feel about comple8ng this GCP course on line? n=1,080 85% 10% 5% Great! This permits me greater freedom to allocate my time wisely. I really don t know, this is my first experience with web based training and education. Not too good! I don t like web based learning. I would much rather spend the time (~ up to 4 or 5 hrs) in a traditional class room environment with an instructor.
Pre- Course Survey Data Please tell us about your background in GCP n- 1080 50% 30% 19% 1% The topic in the course I am about to take is all new to me I have some understanding of the topic but, it is rather superficial Although I have a good understanding of the topic, it is complex. I expect that this course will be a good review for me. I consider myself to be an expert in this topic. Comple8ng this course will likely be a waste of my 8me
Please tell the CITI developers about your learning experience in the GCP course that you just completed. N=2,511 49% 21% 21% 8% 1% Most of the modules that I read, enhanced my knowledge of the course topic. I knew a great deal about the GCP. However, this course was a valuable review for me. It was all new to me, every module that I read, enhanced my knowledge of the GCP. Only a few of the modules enhanced my knowledge of GCP. I learned nothing from this GCP course. It was a complete waste of my 8me.