Best Practices for Running a Hyperfunctional Psychology Laboratory

Transcription

1 Best Practices for Running a Hyperfunctional Psychology Laboratory Greg J. Siegle, Ph.D. University of Pittsburgh School of Medicine These slides available at Presented work supported by MH082998

2 Why bother? You and others can trust your data It s easy to know when you step into a best-practices lab Some researchers get a reputation as careful Increase replicability Decrease debacles Example from my lab: The chilling chiller incident Example from the current fmri world

3 Stuff we ll discuss Study setup Data collection Storing data Analysis Managing the lab

4 Study Setup

5 Pre-emptive strike: Clinical Operations Manual INCLUDING template documents Common elements Study Responsibility Log who does what when Study worksheet stuff which has to happen and when, e.g., calibrations, audits Assessment Schedule Assessment Grid Procedural Checklists Regulatory Binder Template From

6 Regulatory Binders & Lab documents Basic clinical trial model 2 folders per patient 1 for identifiable info, 1 for all study documents. + master list. Excellent list of lab documents Binders/Folders for Protocol and amendments Data Subject Logs and Lists Patient Data 1 per participant Contact Logs and monitoring Reporting Corrospondence with outside organizations (e.g., FDA) IRB Documents Case Report Blank Forms Adverse events People Investigator Information Team Information Lab information Lab certifications, etc. Equipment Investigational product (e.g., drug) info Meeting documents Study meetings Study reports Publications

7 Study management database Stuff to include in addition to data Subject information Screening/Enrollment log Visit Schedule Log Tracking/Reporting information Adverse Event Log Protocol Deviation Log Data Cleaning log Accountability logs Device calibrations and accountabilities Note: SEPARATE database for Master subject log

8 Quality management plan what will you check, how will you check it? Data collection Data processing

9 Quality assurance guidelines Illustrations of procedures What bad data looks like

10 Create folders Study folders: at least data pupil heart behav. analysis matlab spss documents publications regulatory software

11 Folder contents (from Dr. Nicole Prause) Data Raw, important processed stages, data processing scripts such as.m file backup, compiled data, final data The data folder should contain enough information to quickly reconstruct important phases of data processing without storing too many large files on the computer indefinitely. Every data folder should include is a "notes.txt" file, where you note abnormalities for particular subjects and files to enable quick reconstruction of data sets. For example, if a person becomes ill and withdraws from the study, it will be much easier to find this noted in a single file than to start searching to understand why the last two test conditions are missing to make decisions about data inclusion/exclusion. Institutional Review Board Compliance Submissions, revisions, letters of approval, up-to-date informed consent Scripts Electronic questionnaires, up-to-date DMDX scripts, backup of stimuli if size reasonable Publication Poster presentations, papers being prepared, final drafts of accepted/published papers

12 Select protocols carefully Stay as close as possible to industry standards when possible (deviating as necessary ) E.g., the Society for Psychophysiological Research has published standards for EEG, ERP, Startle, Heart rate, HRV, EMG, disease transmission l/index.cfm Internet questionnaires: Skitka papers/others/2006/skitka200 6a.pdf ASTM (standards body)

13 Do an in-house ethics review

14 Data collection

15 Procedural checklists & records Every detail is golden: Have checklists and how-to guides Check the checklists All records shall be prepared, dated, and signed (full signature, hand written) by one person and independently checked, dated, and signed by a 2 nd person (GMP (Good Manufacturing practices) ) Electronic checklists? Possible Electronic records may be considered trustworthy and reliable and be used in leiu of paper records provided that the electronic records have proper secuirty controls (21 CFR Part 11 Subpart A Sec 11.1) Ensure authenticity & integrity of electronic records such that the person responsible for the electronic record cannot readily repudiate the record as not genuine (21 CFR Part 11 Subpart B Sec 11.10) Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec (a)) But I don t recommend it yet!

16 Trouble shooting guides Guideline for ERP data by Cecile Ladouceur and Naho Ichikawa

17 AFTER data collection Data cleaning Signing off

18 Video more is better Essential for clinical interviews to at least get audio. Video is better. Note: Need IRB Approval

19 Task design Validation Check timing / event logging w/ fmri we test at the scanner 1x phantom + 1x pilot before any protocol Check single subjects Write analysis scripts for single subjects BEFORE your first real subject Be a subject for your own protocols Test everything completely BEFORE your first pilot subject. Test everything completely BEFORE your first real subject.

20 Psychophys lab setup Neat reproducable lab setups Diagram in your Ops Manual to show how to do stuff exactly the same every time As many procedural diagrams as might be useful Dr. Nicole Prause s lab setup Care about disease transmission Bloodborne Pathogen control: Gloves as much as possible Don t abraid the skin more than you need to Disposable electrodes when possible Disinfect CIDEX if you have ventillation Control III + Suave shampoo if you don t Wear a labcoat that s actually what they re for

21 Checking stuff works before data collection Protocols before your protocols Check all communications between computers, peripherals, and data collection devices Make sure your stimuli show Have this in your checklists We have eprime routines to test getting scanner trigger, eye-tracker events mouse/button pushes

22 Storing Data

23 Data Security & Integrity Whitebox standards: Keep original data in unalterable form Have 2 nd copy for any necessary changes (e.g., remove a few trials, concatenate runs ) Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec (a)) Security 21 CFR part 11: Double password protection Standards They exist for most things: IRB E.g., consent forms separate from data

24 Databases Huge science - E.g., Have primary keys Don t change schemas Consistent long descriptive column names across tables Try things first in a local database Good rule of thumb: 20 columns per table more is weird design Lab standards Ids are in columns called id All tables have id 21 CFR Part 11 Keep an audit history of date created and by who, and dates changed/updated

25 Backups Ideally Daily data backups Weekly incremental computer backups Monthly full backups Keep a set of backups in a secure place outside your lab

26 Analysis & Quality Control

27 Document everything Lab notebooks are essential Extreme: Open Lab Notebook Notebook_Science All work posted immediately to the public eye Good tool: Page Commercial approaches Big list at: ntent.php?pid=126157&sid= My approach: Powerpoints per study Greg s Journal template on the PICAN server \\oacres3\rcn\pican\docs\gjsjourna l.pot Sharepoint blog? Database page for all changes with name, date, change description Analyses should be reproducible I like 1 matlab or SPSS file with all commands that produce all analyses for a given study. Documentation

28 Reasons for using ELNs/ virtual workspaces 1. They are an efficient way of managing large projects, multiple projects and multi-institution projects. 2. Provenance ensures that any accusation of fraud can easily be addressed. 3. Addresses the problem of missing information due to turnover in lab personnel (and students). 4. Can access research results from anywhere and therefore keep up with the ongoing work in the lab while traveling. 5. These systems are already being used in industry, therefore are studentsneed to be acquainted with them to be employable. 6. Meets requirements of granting agency mandates for data managment plans. 7. Facilitates depositing data into data repositories for reuse and repurposing.

29 Example journal page

30 Beyond Powerpoint Lab Bench People layer id=126157&sid=

31 Example commercial solution: (Not endorsed just summarized) From labarchives.com Intuitive Electronic Lab Notebook (ELN) organizes your laboratory data Preserve all your data securely, including all versions of all files Share information within your laboratory Keep abreast of developments in your lab even when traveling Collaborate with investigators by sharing selected data from your Electronic Laboratory Notebook Publish selected data to specific individuals or the public Protect your intellectual property Runs on all platforms, including Windows, Mac, Linux, ipad and Android devices Special classroom version of our Electronic Lab Notebook also available

32 Sample all-figures-in-paper script %% associations of power change with change in other things within and between groups ctrl=find((s.grp==1) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); cct=find((s.grp==2) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); tau=find((s.grp==3) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); cct_tau=[cct; tau]; fprintf(' \n'); fprintf('cct r(power_on change, rumination change)\n'); st.r_poweronchg_rsqchg_cct=r(poweronchg(cct),s.rsqchg(cct),0,1,1.5,-999); figure(7); clf; regplot(rescaleoutliers(poweronchg(cct)),rescaleoutliers(s.rsqchg(cct))); xlabel('trial Frequency Power Post CCT - Pre CCT'); ylabel('rumination (RSQ) Post CCT - Pre CCT'); figure(8); clf; regplot(rescaleoutliers(poweronchg(tau)),rescaleoutliers(s.rsqchg(tau))); xlabel('trial Frequency Power Post TAU - Pre TAU'); ylabel('rumination (RSQ) Post TAU - Pre TAU'); figure(9); clf; focindfromcct_change= *poweronchg *poweroffchg; regplot(rescaleoutliers(focindfromcct_change(cct)),rescaleoutliers(s.rsqchg(cct))); xlabel('unfocus Index Post CCT - Pre CCT'); ylabel('rumination (RSQ) Post CCT - Pre CCT');

33 Use best practices for preprocessing data Again with the Psychophysiology guidelines Visual inspection of artifacts When are artifacts ok to let in to data? How much should we say we re letting in? Contingency planning What if you change preprocessing midway through? I think you should reprocess everything What if you change preprocessing after-the-fact? Depending on how serious, note it.

34 Diagnosis and Clinical dispositions: Case conferences Reliability on ANYTHING subjective Double data entry See your Research Methods textbook Quality control

35 Check your data early and often Quality check psychophys data that day and fmri data within a week (while it s on the servers) Single subject analyses Group analyses with N=5

36 Maintaining a lab

37 Calibrations Regular monthly calibrations of all instruments Currently done for pupilometer Other stuff? MR center and BIRC have done calibrations, e.g., stability checks regularly. We don t request them. But we should for our own documentation.

38 Security Double-locked file cabinets Password protection for computers, files, etc. Note: 21 CFR (Code of Federal Regulations) Part 11 - Food and Drug Administration (FDA) guidelines on electronic records has security standards for data audits, system validations, audit trails, electronic signatures, and documentation FR/CFRSearch.cfm?CFRPart=11

39 Audits Every 6 months, all data within that 6 months Quality management help at: RC/trial_conduct/qualitymanagement.htm There are chart audit tools Regulatory file review tools Every year full audit should be easy

40 The human thing Laboratory mentality is important. Attend to it. Anecdotal evidence suggests happy inspired labs are often more functional. You will likely not be in touch with the emotional health of the lab. Have someone who is. Make their report to you on lab health a regular thing.

41 Hire for your weaknesses Good labs often have people who are (not all of these at once) Detail oriented Socially attuned Tech savvy Inspired

42 Sources 21 CFR (Code of Federal Regulations) Part 11 - Food and Drug Administration (FDA) guidelines on electronic records has security standards for data audits, system validations, audit trails, electronic signatures, and documentation arch.cfm?cfrpart=11 Esp. Subpart B electronic records Good Manufacturing Practices (GMP) ASTM (standards body) Robert L. Zimmerman Jr, 10 Best Practices for Good Laboratories. Nov Dec, 2010, November/December, Standardization News Clinical Trials Resource Center