Best Practices for Running a Hyperfunctional Psychology Laboratory Greg J. Siegle, Ph.D. University of Pittsburgh School of Medicine These slides available at http://www.pitt.edu/~gsiegle/sieglelaboratorybestpracticescolloquium.pdf Presented work supported by MH082998
Why bother? You and others can trust your data It s easy to know when you step into a best-practices lab Some researchers get a reputation as careful Increase replicability Decrease debacles Example from my lab: The chilling chiller incident Example from the current fmri world
Stuff we ll discuss Study setup Data collection Storing data Analysis Managing the lab
Study Setup
Pre-emptive strike: Clinical Operations Manual INCLUDING template documents Common elements Study Responsibility Log who does what when Study worksheet stuff which has to happen and when, e.g., calibrations, audits Assessment Schedule Assessment Grid Procedural Checklists Regulatory Binder Template From http://www.uth.tmc.edu/ctrc/studymanagement.html
Regulatory Binders & Lab documents Basic clinical trial model 2 folders per patient 1 for identifiable info, 1 for all study documents. + master list. Excellent list of lab documents http://www.uth.tmc.edu/ctrc/regulatory.html Binders/Folders for Protocol and amendments Data Subject Logs and Lists Patient Data 1 per participant Contact Logs and monitoring Reporting Corrospondence with outside organizations (e.g., FDA) IRB Documents Case Report Blank Forms Adverse events People Investigator Information Team Information Lab information Lab certifications, etc. Equipment Investigational product (e.g., drug) info Meeting documents Study meetings Study reports Publications
Study management database Stuff to include in addition to data Subject information Screening/Enrollment log Visit Schedule Log Tracking/Reporting information Adverse Event Log Protocol Deviation Log Data Cleaning log Accountability logs Device calibrations and accountabilities Note: SEPARATE database for Master subject log
Quality management plan what will you check, how will you check it? Data collection Data processing
Quality assurance guidelines Illustrations of procedures What bad data looks like
Create folders Study folders: at least data pupil heart behav. analysis matlab spss documents publications regulatory software
Folder contents (from Dr. Nicole Prause) Data Raw, important processed stages, data processing scripts such as.m file backup, compiled data, final data The data folder should contain enough information to quickly reconstruct important phases of data processing without storing too many large files on the computer indefinitely. Every data folder should include is a "notes.txt" file, where you note abnormalities for particular subjects and files to enable quick reconstruction of data sets. For example, if a person becomes ill and withdraws from the study, it will be much easier to find this noted in a single file than to start searching to understand why the last two test conditions are missing to make decisions about data inclusion/exclusion. Institutional Review Board Compliance Submissions, revisions, letters of approval, up-to-date informed consent Scripts Electronic questionnaires, up-to-date DMDX scripts, backup of stimuli if size reasonable Publication Poster presentations, papers being prepared, final drafts of accepted/published papers
Select protocols carefully Stay as close as possible to industry standards when possible (deviating as necessary ) E.g., the Society for Psychophysiological Research has published standards for EEG, ERP, Startle, Heart rate, HRV, EMG, disease transmission http://www.sprweb.org/journa l/index.cfm Internet questionnaires: Skitka www.uvm.edu/~pdodds/files/ papers/others/2006/skitka200 6a.pdf ASTM (standards body) www.astm.org
Do an in-house ethics review
Data collection
Procedural checklists & records Every detail is golden: Have checklists and how-to guides Check the checklists All records shall be prepared, dated, and signed (full signature, hand written) by one person and independently checked, dated, and signed by a 2 nd person (GMP (Good Manufacturing practices) 211.186) Electronic checklists? Possible Electronic records may be considered trustworthy and reliable and be used in leiu of paper records provided that the electronic records have proper secuirty controls (21 CFR Part 11 Subpart A Sec 11.1) Ensure authenticity & integrity of electronic records such that the person responsible for the electronic record cannot readily repudiate the record as not genuine (21 CFR Part 11 Subpart B Sec 11.10) Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec 11.10 (a)) But I don t recommend it yet!
Trouble shooting guides Guideline for ERP data by Cecile Ladouceur and Naho Ichikawa
AFTER data collection Data cleaning Signing off
Video more is better Essential for clinical interviews to at least get audio. Video is better. Note: Need IRB Approval
Task design Validation Check timing / event logging w/ fmri we test at the scanner 1x phantom + 1x pilot before any protocol Check single subjects Write analysis scripts for single subjects BEFORE your first real subject Be a subject for your own protocols Test everything completely BEFORE your first pilot subject. Test everything completely BEFORE your first real subject.
Psychophys lab setup Neat reproducable lab setups Diagram in your Ops Manual to show how to do stuff exactly the same every time As many procedural diagrams as might be useful Dr. Nicole Prause s lab setup http://www.span-lab.com/assets/images/photos/eegprep.jpg Care about disease transmission Bloodborne Pathogen control: Gloves as much as possible Don t abraid the skin more than you need to Disposable electrodes when possible Disinfect CIDEX if you have ventillation Control III + Suave shampoo if you don t Wear a labcoat that s actually what they re for
Checking stuff works before data collection Protocols before your protocols Check all communications between computers, peripherals, and data collection devices Make sure your stimuli show Have this in your checklists We have eprime routines to test getting scanner trigger, eye-tracker events mouse/button pushes
Storing Data
Data Security & Integrity Whitebox standards: Keep original data in unalterable form Have 2 nd copy for any necessary changes (e.g., remove a few trials, concatenate runs ) Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec 11.10 (a)) Security 21 CFR part 11: Double password protection Standards They exist for most things: http://www.astm.org/ IRB E.g., consent forms separate from data
Databases Huge science - http://c2.com/cgi/wiki?databasebestpractices E.g., Have primary keys Don t change schemas Consistent long descriptive column names across tables Try things first in a local database Good rule of thumb: 20 columns per table more is weird design Lab standards Ids are in columns called id All tables have id 21 CFR Part 11 Keep an audit history of date created and by who, and dates changed/updated
Backups Ideally Daily data backups Weekly incremental computer backups Monthly full backups Keep a set of backups in a secure place outside your lab
Analysis & Quality Control
Document everything Lab notebooks are essential Extreme: Open Lab Notebook http://en.wikipedia.org/wiki/open_ Notebook_Science All work posted immediately to the public eye Good tool: http://openwetware.org/wiki/main_ Page Commercial approaches Big list at: http://campusguides.lib.utah.edu/co ntent.php?pid=126157&sid=21316 70 My approach: Powerpoints per study Greg s Journal template on the PICAN server \\oacres3\rcn\pican\docs\gjsjourna l.pot Sharepoint blog? Database page for all changes with name, date, change description Analyses should be reproducible I like 1 matlab or SPSS file with all commands that produce all analyses for a given study. Documentation
Reasons for using ELNs/ virtual workspaces 1. They are an efficient way of managing large projects, multiple projects and multi-institution projects. 2. Provenance ensures that any accusation of fraud can easily be addressed. 3. Addresses the problem of missing information due to turnover in lab personnel (and students). 4. Can access research results from anywhere and therefore keep up with the ongoing work in the lab while traveling. 5. These systems are already being used in industry, therefore are studentsneed to be acquainted with them to be employable. 6. Meets requirements of granting agency mandates for data managment plans. 7. Facilitates depositing data into data repositories for reuse and repurposing. http://campusguides.lib.utah.edu/content.php?pid=126157&sid=2131670
Example journal page
Beyond Powerpoint Lab Bench People layer http://campusguides.lib.utah.edu/content.php?p id=126157&sid=2131670
Example commercial solution: (Not endorsed just summarized) From labarchives.com Intuitive Electronic Lab Notebook (ELN) organizes your laboratory data Preserve all your data securely, including all versions of all files Share information within your laboratory Keep abreast of developments in your lab even when traveling Collaborate with investigators by sharing selected data from your Electronic Laboratory Notebook Publish selected data to specific individuals or the public Protect your intellectual property Runs on all platforms, including Windows, Mac, Linux, ipad and Android devices Special classroom version of our Electronic Lab Notebook also available
Sample all-figures-in-paper script %% associations of power change with change in other things within and between groups ctrl=find((s.grp==1) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); cct=find((s.grp==2) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); tau=find((s.grp==3) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); cct_tau=[cct; tau]; fprintf('----------------------------------\n'); fprintf('cct r(power_on change, rumination change)\n'); st.r_poweronchg_rsqchg_cct=r(poweronchg(cct),s.rsqchg(cct),0,1,1.5,-999); figure(7); clf; regplot(rescaleoutliers(poweronchg(cct)),rescaleoutliers(s.rsqchg(cct))); xlabel('trial Frequency Power Post CCT - Pre CCT'); ylabel('rumination (RSQ) Post CCT - Pre CCT'); figure(8); clf; regplot(rescaleoutliers(poweronchg(tau)),rescaleoutliers(s.rsqchg(tau))); xlabel('trial Frequency Power Post TAU - Pre TAU'); ylabel('rumination (RSQ) Post TAU - Pre TAU'); figure(9); clf; focindfromcct_change=-9.94-151.94.*poweronchg+109.13.*poweroffchg; regplot(rescaleoutliers(focindfromcct_change(cct)),rescaleoutliers(s.rsqchg(cct))); xlabel('unfocus Index Post CCT - Pre CCT'); ylabel('rumination (RSQ) Post CCT - Pre CCT');
Use best practices for preprocessing data Again with the Psychophysiology guidelines http://www.sprweb.org/journal/index.cfm Visual inspection of artifacts When are artifacts ok to let in to data? How much should we say we re letting in? Contingency planning What if you change preprocessing midway through? I think you should reprocess everything What if you change preprocessing after-the-fact? Depending on how serious, note it.
Diagnosis and Clinical dispositions: Case conferences Reliability on ANYTHING subjective Double data entry See your Research Methods textbook Quality control
Check your data early and often Quality check psychophys data that day and fmri data within a week (while it s on the servers) Single subject analyses Group analyses with N=5
Maintaining a lab
Calibrations Regular monthly calibrations of all instruments Currently done for pupilometer Other stuff? MR center and BIRC have done calibrations, e.g., stability checks regularly. We don t request them. But we should for our own documentation.
Security Double-locked file cabinets Password protection for computers, files, etc. Note: 21 CFR (Code of Federal Regulations) Part 11 - Food and Drug Administration (FDA) guidelines on electronic records has security standards for data audits, system validations, audit trails, electronic signatures, and documentation http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfc FR/CFRSearch.cfm?CFRPart=11
Audits Every 6 months, all data within that 6 months Quality management help at: http://www.uthouston.edu/ct RC/trial_conduct/qualitymanagement.htm There are chart audit tools Regulatory file review tools Every year full audit should be easy
The human thing Laboratory mentality is important. Attend to it. Anecdotal evidence suggests happy inspired labs are often more functional. You will likely not be in touch with the emotional health of the lab. Have someone who is. Make their report to you on lab health a regular thing.
Hire for your weaknesses Good labs often have people who are (not all of these at once) Detail oriented Socially attuned Tech savvy Inspired
Sources 21 CFR (Code of Federal Regulations) Part 11 - Food and Drug Administration (FDA) guidelines on electronic records has security standards for data audits, system validations, audit trails, electronic signatures, and documentation http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrse arch.cfm?cfrpart=11 Esp. Subpart B electronic records Good Manufacturing Practices (GMP) ASTM (standards body) www.astm.org Robert L. Zimmerman Jr, 10 Best Practices for Good Laboratories. Nov Dec, 2010, November/December, Standardization News Clinical Trials Resource Center http://www.uthouston.edu/ctrc/trial_conduct/