Methods for Creating Reliable and Reproducible Analyses

Transcription

1 Methods for Creating Reliable and Reproducible Analyses Jennifer Blackford, PhD Assistant Professor of Psychiatry VKC Stats & Methodology Core Training Coordinator

2 Goals Introduce a new idea Persuade you that reproducible analyses are necessary Convince you that achieving reproducible analyses is possible Provide the necessary tools for you to begin creating reliable and reproducible analyses

3 Outline 1. Reliability in research methods 2. Reliability in analyses 3. Dangers of point and click software 4. What are reproducible results? 5. Barriers to reproducible results 6. Methods for creating reproducible analyses 7. Practical implementation

4 Reliability in Research Methods

5 What is reliability? re li a ble [ri-lahy-uh-buh l] adjective that may be relied on; dependable in achievement, accuracy, honesty, etc.: reliable information. From Random House Dictionary

6 How do we ensure reliability of our research methods? Examples?

7 How reliable is reliable enough? a) 70% b) 80% c) 90% d) 100%

8 What if the method is not reliable? a) use it anyway b) find a different method c) revise method to increase reliability d) develop a new method

9 Reliability in (Statistical) Analyses

10 Scientific Method Steps 1. Generate hypothesis 2. Perform experiment and collect data 3. Analyze data 4. Interpret data and draw conclusions 5. Publish results

11 What does analysis include?

12 Statistical Analysis Data entry Data cleaning Data entry errors Missing data Recoding Data preparation Checking data distributions Handling outliers Transforming variables Creating new variables Performing statistical analyses Recording results Creating graphs

13 Neuroimaging Analysis Image file pre-processing Convert image files to standard format Correct head motion within and across trials Align structural and functional image Transform images into a standard space Perform statistical analyses Input experimental design Assign event-timing information to image files Run statistical analyses Record Results Create Graphs

14 Should we expect the same reliability (or reproducibility) from our analytic methods as we do from our other research methods? a) No, analyses aren t really methods b) No, analytic methods are too complex c) Maybe, but everything is fine, no need to change methods d) Yes, all parts of the scientific process should be reliable

15 What are the common practices? Enter data in Excel Use Excel for data cleaning, etc Generate descriptive statistics (counts, means, etc) in Excel Import data into SPSS for analysis Use point-and-click options to run statistical analyses Type data back into Excel to create graphs

16 Now let s evaluate reliability. Enter data in Excel Use Excel for data cleaning, etc Generate descriptive statistics (counts, means, etc) in Excel Import data into SPSS for analysis Use point-and-click options to run statistical analyses Type data back into Excel to create graphs

17 The Dangers of Point-and-Click Software

18 Excel Data entry inconsistencies and errors No record of what steps were performed Sorting dangers Data export issues There is a bug in how Windows processing Excel files which results in missing data being exported. Cryer (2001). Problems using Microsoft Excel for Statistics.

19 Spreadsheet from Hell Dan Byrne

20 Excel Statistical Issues Missing data are handled inconsistently and sometimes incorrectly Excel uses poor algorithms to find the standard deviation Excel does not treat tied observations correctly when ranking Regression is especially problematic-- computations are often erroneous due to multiple incorrect procedures Cryer (2001). Problems using Microsoft Excel for Statistics.

21 Other Point-and-Click There are also dangers with other programs that offer user-friendly pointand-click capabilities, such as SPSS and SAS. Neuroimaging analysis programs, like Brain Voyager and SPM have the same issues too.

22 Has this ever happened to you..

23 Vignette #1 You sit down to finish writing your manuscript. You realize that you need to clarify one result by running an additional analysis. First, you re-run the primary analysis but the primary results don t match what you have in your paper.

24 Vignette #2 You get an request from the graduate student of a colleague at another university. She has conducted an experiment similar to one you published five years ago. She has a question and would like for you to run an analysis on a subset of subjects. When you go to the data folder, you find thirty program files and can t remember which one has the analysis. Where is that final dataset????

25 Would you like to increase the reliability of your analytic methods? a) No need, my current methods are sufficient. I m finding significant results already. b) Maybe, but I think it will be hard and I don t have the energy for something new. c) Yes! Show me how, I ll do anything.

26 What are reproducible results?

27 Reproducible Results Some consider the use of statistical methods that can not be replicated as a breach of research ethics. There is a new movement in statistics called reproducible results or literate statistical programming.

28 Claerbout s Principle An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures. Note: The same Principle obviously applies to tables, standard errors, and so on. deleeuw, J (2001). Reproducible Results: The Botton Line.

29 Literate Statistical Practice Literate Statistical Practice (LSP, Rossini, 2001) describes an approach for creating self-documenting statistical results. It applies literate programming (Knuth, 1992) and related techniques in a natural fashion to the practice of statistics. In particular, documentation, specification, and descriptions of results are written concurrently with writing and evaluation of statistical programs. Rossini & Leisch (2003). Literate Statistical Practice.

30 Barriers to Reproducible Results

31 Barriers What are some barriers to reproducible results? 1. Awareness We didn t even know there was a problem. 2. Knowledge 3. Change 4. Effort What are the right methods? You mean I wasn t doing the right thing? It s too hard. I don t have time. I don t have the energy to learn something new.

32 1. Reliable Results Benefits The results you produce today will be the same results you will produce tomorrow. 2. Time Savings/Efficiency In the (not so) long run, you will save much time. 3. Flexibility When you don t point-and-click you gain many new analytic options. 4. Institutional (Lab) Memory

33 Baby Steps We ve already addressed the first barrier Awareness Now, we ll address the second barrier Knowledge

34 Methods for Creating Reproducible Analyses

35 Caveats 1. This will take discipline. On the front end, it does take more time to create a system or structure for your analysis. It is much faster to use Excel to do one quick analysis. 2. You might have to learn something new. This can be a major barrier. Learning takes a commitment of time and effort.

36 Overarching Principle: Organization Organization for everything is key. This principle is as true for your research study as for your data analysis. electronic folder structure file naming conventions analytic programs Things can get out of control.*

37 Data Entry Enter your data into a stable structure. Good examples of stable structures a relational database (like Microsoft Access) VKC provides database development VUMC REDCap (Research Electronic Data Capture) SPSS or SAS tables

38 Data Cleaning and Preparation The second step is to create methods or procedures for documenting every step

39 How-To Guides

40 How-To Guides By Fred Davidson, 1996

41

42 What About Data Analysis?

43 Now for the scary word. programming

44 Which statement best describes you? a) I have never programmed before. b) I have programmed a little.10 years ago. c) I have some basic programming skills (and feel comfortable learning more) d) I am a code junkie!

45 A code example

46 Final Product The final product is a single file that includes all of the code necessary to reproduce all of the analyses. The code file should be heavily commented so that another reader can follow the workflow. Note: Ultimately, the file will also print out text and graphics.

47 Specifically Data import Data cleaning/formatting/manipulation You must have an audit trail of every change made to the data. Descriptive statistics Inferential/predictive statistics Include all of the workflow steps, including analyses that didn t work or that you don t keep so that you ll remember what you did later

48 Practical Implementation An illustration using SPSS

49 SPSS Starting simply, SPSS provides a relatively easy way to create a document that keeps track of all of your statistical steps. You can begin with point-and-click, saving the underlying code, while progressing to programming. SPSS journaling (edit->options->session journal, record syntax in journal) can track your syntax selections

50 SPSS Example

51 Protect Yourself Track your changes version control software (like Subversion) tracks of each iteration of document or program RCS software: Screen capture software can capture your mouse movements: SnagIt ( Keep prior operating systems With Microsoft Virtual PC you can have other operating systems on your computer pc/default.mspx Jim Steiger

52 Protect Yourself Back up, back up, back up Disk space is cheap, your time is not Amazon S3 storage for the internet Jungle Disk Acronis True Image Home Note: first, go to Computer Management and make certain that the Volume Shadow Copy service starts automatically before you install the software make several copies of the Emergency Boot Disk, because you can boot into a Windows system that will not load windows and restore the whole hard drive (or just the system settings) from OUTSIDE WINDOWS. Jim Steiger

53 Future Steps SAS Enterprise Guide With programs like R and Sweave/LaTex, you can incorporate the entire text of your report with results and graphics Upcoming Seminar Terri Scott May 16, 2008.