Data archiving and reproducible research for ecology and evolu6on. March 23 rd 2010 Ian Dworkin

Transcription

1 Data archiving and reproducible research for ecology and evolu6on. March 23 rd 2010 Ian Dworkin

2 Outline of the ques6ons for the workshop 1. Why should I share my data? 2. When should I share my data? 3. When should I not share my data? 4. How should I share my data? 5. Where should I share my data? 6. Final thoughts on data sharing, and the basics of reproducible research.

3 Why should I share my data Science is an open process. Increases exposure and cita6ons of your work. Because journals (are beginning to ) demand it. Because funding agencies (are beginning to) demand it. No longer need to deal with repeated requests for your data. Permanent storage of data

4 Why should I share my data: Science as an open Process Science as an endeavor is an open process, that includes sharing materials, methods, the analy6c tools and the raw data to both replicate your analysis, as well as providing the seeds for future work. The important thing in science is not so much to obtain new facts as to discover new ways of thinking about them Sir William Bragg.

5 Why should I share my data: Science as an open Process Ecology and Evolu6on are disciplines which to a very great degree require the use of past data both for basic and applied research (clima6c shixs in popula6on, conserva6on, etc ). Even in basic research some of the most important papers in the field have been colla6ng and synthesizing other work in the field.

6 An example of the importance of data Times Cited: 595 This paper used only summary sta6s6cs on es6mates of selec6on gradients from the papers. Think how much more they could have done if they had all the raw data!!!

7 Why should I share my data? Unfortunately, data has not been shared as freely as required by science in many instances.

8 Data sharing in prac6ce Wicherts (2004) requested data from the corresponding authors of 141 ar6cles recently published in American Psychological Associa6on (APA) journals. All authors had signed a commitment to share their data upon request "6 months later, a/er wri2ng more than 400 e- mails and sending some corresponding authors detailed descrip2ons of our study aims, approvals of our ethical commi?ee, signed assurances not to share data with others, and even our full resumes- we ended up with a meager 38 posi2ve reac2ons and the actual data sets from 64 studies (25.7% of the total number of 249 data sets). This means that 73% of the authors did not share their data." h_ps://

9 Obstacles to data reuse In a survey of 1240 gene6cists 47% had been denied at least one request for data or materials in the preceding 3 yrs 28% reported that they had been unable to confirm published research because of data withholding The most common reasons cited for withholding: Too much effort to produce the data (80%) Protec6ng the ability of a junior colleague to publish (64%) Protec6ng their own ability to publish (57%). h_ps://

10 Published works in evolu6onary biology 27 papers from 5 different journals. 41% had supplemental materials. But only 7% included raw orphan data. Excep6ons included simula6on results and sequence alignments. S6ll, 67% of papers using alignments did not provide them. Genbank submission was generally honored. 78% analyzed data not deposited in any repository. 48% were based at least in part on data from other publica6ons. Evolu2onary biologists use published data more frequently than they are providing it themselves! h_ps://

11 Why should I share my data Science is an open process. Increases exposure and cita=ons of your work. Because journals (are beginning to ) demand it. Because funding agencies (are beginning to) demand it. No longer need to deal with repeated requests for your data. Permanent storage of data

12 Sharing data increases the exposure and may increase cita6on rate of your studies..but the argument against oxen comes from younger or less established inves6gators who are afraid of being scooped. They need to understand that the danger isn t being scooped; it s being ignored. In fact, however, those who share data are cited much more frequently than those who do not, and so it is in everyone s interest to share data." ~ Gary King, in "Interview with Gary King," Scien6fic Data Sharing Project, December 13, 2010.

13 Sharing data increases the exposure and may increase cita6on rate of your studies. While more work needs to be done to validate this observa6on, it is a poten6ally nice added bonus!!!

14 Why should I share my data Science is an open process. Increases exposure and cita6ons of your work. Because journals (are beginning to ) demand it. Because funding agencies (are beginning to) demand it. No longer need to deal with repeated requests for your data. Permanent storage of data

15 Journal are requiring data sharing.

16 American Society of Naturalists American Naturalist Ecological Society of America Ecology, Ecological Letters, Ecological Monographs, etc. European Society for Evolutionary Biology Journal of Evolutionary Biology Society for Integrative and Comparative Biology Integrative and Comparative Biology Society for Molecular Biology and Evolution Molecular Biology and Evolution Society for the Study of Evolution Evolution Society for Systematic Biology Systematic Biology Commercial journals Molecular Ecology Molecular Phylogenetics and Evolution h_ps://

17 Why should I share my data Science is an open process. Increases exposure and cita6ons of your work. Because journals (are beginning to ) demand it. Because funding agencies (are beginning to) demand it. No longer need to deal with repeated requests for your data. Permanent storage of data

18 Because funding agencies (are beginning to) demand it. NSF has now ins6tuted a Data management plan requirement. NIH is also in the process of formalizing such requirements.

19 Why should I share my data (in a public database) Science is an open process. Increases exposure and cita6ons of your work. Because journals (are beginning to ) demand it. Because funding agencies (are beginning to) demand it. No longer need to deal with repeated requests for your data. Permanent storage of data

20 No longer need to deal with repeated requests for your data if the data is available in a public repository. I have always worked very hard to supply my data when requested. If it is requested recently axer publica6on usually this is trivial. However when the request comes several years later, when I have changed computers, opera6ng systems, etc. It can be more difficult. If it is publically available, then once it is submi_ed, it is done!

21 Why should I share my data (in a public database) Science is an open process. Increases exposure and cita6ons of your work. Because journals (are beginning to ) demand it. Because funding agencies (are beginning to) demand it. No longer need to deal with repeated requests for your data. Permanent storage of data.

22 Permanent storage of data Computers breakdown, hard drives fail. Par6cular file formats (especially proprietary ones) may no longer be supported. Inves6gators may leave science, and data is lost.. If data is permanently stored in a simple flat file (.txt or.csv) and stored in a public database (that is backed up), this will never be an issue, and the data will be available for genera6ons to come.

23 Advantages to you, the author Access to your colleagues data Visibility and citability for your own data Another way for your work to have high impact Integra6on Combinability with other data adds value to your own Long- term preserva6on Including data format migra6on Ad hoc data sharing can be burdensome Deposi6on to mul6ple specialized repositories Fulfilling individual requests for data takes effort h_ps://

24 For more on these ra6onales Vision, T Open Data and the Social Contract of scien6fic publishing. Bioscience. 60(5):330 Whitlock, MC Data archiving in ecology and evolu6on: best prac6ces. TREE 26(2): Also see the Panton Principles for open data in science. (see extra material at the end of the presenta6on).

25 Outline of the ques6ons for the workshop 1. Why should I share my data? 2. When should I share my data? 3. When should I not share my data? 4. How should I share my data? 5. Where should I share my data? 6. Final thoughts on data sharing, and the basics of reproducible research.

26 When should I share my data? What data? The general advice (and requirements) is to share the data used in a manuscript to coincide with the publica6on of that manuscript. Databases such as DRYAD allow for an embargo for up to a year, to provide you (the author) more opportunity in using that data for addi6onal research, without concern of compe66on.

27 What data should I share. The general requirement is to share all of the raw data required to reproduce the analyses that were performed in your paper. There is no requirement to share unpublished data (even if it is related). However several databases allow for the publica6on of such data (at the authors discre6on) in par6cular for long term data sets.

28 Outline of the ques6ons for the workshop 1. Why should I share my data? 2. When should I share my data? 3. When should I not share my data? 4. How should I share my data? 5. Where should I share my data? 6. Final thoughts on data sharing, and the basics of reproducible research.

29 When should I not share my data? An oxen heard concern is.. What if I did something wrong in the analysis This is not a valid reason for not sharing your data. Indeed this is a great reason to share it, so that your results can be verified, and mistakes corrected. Plus this can lead to addi6onal cita6ons!

30 When should I not share my data? There are legi6mate reasons not to share data in some situa6ons. For example, if you have GIS data that may point to organisms that are of par6cular conserva6on concern. Of course you could just remove the GIS data

31 Outline of the ques6ons for the workshop 1. Why should I share my data? 2. When should I share my data? 3. When should I not share my data? 4. How should I share my data? 5. Where should I share my data? 6. Final thoughts on data sharing, and the basics of reproducible research.

32 Where should I archive my data? There are a few op6ons: Your website. Journal Supplementary informa6on. Public or Community Data repositories.

33 Where should I archive my data? Personal websites (even on a University server) are not ideal, as it may be difficult to permanently maintain access to the url, in par6cular axer a researcher departs. Supplementary material sec6ons at journals are problema6c for TWO reasons. OXen there is a paywall, so users without subscrip6ons (personal or ins6tu6onal) can not access the data. Journals are NOT doing an adequate job of maintaining the supplementary materials. See Anderson et al BMC Bioinforma6cs 7:260

34 Where should I archive my data? There are now numerous general (dataone) and data specific repositories available. Most are designed for long term storage, with backups. Here are a few examples.

35 Can this vision be achieved by specialized databases? There are a growing number of specialized databases to which deposi6on is expected (Genbank, Treebase) And others are emerging (Morphbank, PDB, etc) A world in which every datatype had its own required database, each with its own submission system Would be a huge burden on authors Would inevitably leave some data orphaned Might never be financially possible h_ps://

36 What is the alterna6ve? A catch- all digital library for data that are Heterogeneous Idiosyncra6cally structured h_ps://

37 Outline of the ques6ons for the workshop 1. Why should I share my data? 2. When should I share my data? 3. When should I not share my data? 4. How should I share my data? 5. Where should I share my data? 6. Final thoughts on data sharing, and the basics of reproducible research.

38 How should I share my data? Read: Whitlock, MC Data archiving in ecology and Evolu6on: best prac6ces. TREE 26(2):61-65

39 How should I share my data? Make sure that you provide raw data, and if it is appropriate transformed data (i.e. if your transforma6on is more complex than a simple log transforma6on ). Re- run the analysis that you published with the data you are about to submit. Can you replicate your results? If you can not, no one else will be able to either!!!!!!*** Make sure you have it in a very simple format such as a "flat" file such as tab/space (.txt), or comma seperated values (.csv).* These files can be opened in MOST soxware.** - The problem with formats for soxware such as Excel, SAS, jmp is two fold: - They are generally proprietary so not everyone will have access to them. - Many of these formats do not have a long shelf life. For instance the earliest versions of Excel, SAS and jmp file formats are not supported by most (if any) current programs. h_p://repositories.lib.utexas.edu/recommended_file_formats

40 How should I share my data? It is best to start considering how to share the meta- data associated with the data (variable names, organisms, when it was collected, anything else needed to replicate results not included in the paper) while doing the analysis and manuscript prepara6on.

41 How should I share my data? Clearly you need to choose an appropriate archive for your discipline and the data type. Many par6cular data types have very specific requirements for submission (NCBI GEO). Others such as DRYAD are more flexible, but they are s6ll manually curated before being accepted. Generally a short clear readme file associated with the data will be enough to clear up any specific ques6ons.

42 How should I share my data? Make sure the meaning of column headers is clear. Also consider units, loca6on, indicators for missing data, codes for categorical data. (Whitlock 2010). There is an ecological metadata language (EML) to aid in greater use.

43 Reproducible research However the data is only one part of the equa6on. Most analyses have become increasingly complicated and computa6onal. Thus having just the raw data may not be enough to reproduce your results. Point and click based analysis soxware, while convenient, hinders reproducibility of science.

44 The problem with GUI analysis tools.

45 Reproducible research If at all possible use open analysis tools (such as R, python, etc..) or even proprietary languages (SAS, matlab, mathema6ca) but provide the scripts as metadata used for the analysis in the paper. Remember to run it before submivng it to the database, to make sure you can replicate the results of your paper.*

46 But what if I only have access to GUI based analysis tools for my work. Then keep a detailed analysis notebook of how you performed the analysis (bu_ons selected, op6ons clicked, etc ). Provide this with the data. Tom suggested that you also look to see if the soxware has a log or macro language that may have the underlying script that you can archive.

47 Extra stuff (ques6ons)

48 Unpublished data in Dryad Datasets of special historical, educa6onal and scien6fic significance Par6cularly long- term data collec6ons that are greater than the sum of their parts. NESCent s Dis6nguished Visi6ng Scholar program. Prepublished data from NESCent scien6sts. Otherwise, unpublished data is not accepted. h_ps://

49 Issues of intellectual property Some data can be copyrighted (e.g. images) Most data cannot, although law varies by jurisdic6on Most experts downplay the role of legal agreements in regula6ng data reuse within a Science Commons Instead, scien6fic norms (for reuse, and for a_ribu6on) are thought to be more appropriate When immediate data sharing might be truly deleterious, an embargo is an op6on. h_ps://

50 h_p://hdl.handle.net/10255/dryad.23 Iden6fier is a handle Handle belongs to Dryad Specific item ID h_ps://

51 DRYAD Paper cita6on Sidlauskas B (2007) Tes6ng for Unequal Rates of Morphological Diversifica6on in the Absence of a Detailed Phylogeny: A Case Study From Characiform Fishes. Evolu2on 61 (2), , doi: /j x Data cita6on Sidlauskas B (2007) Rela6ve warps. hdl:10255/dryad.23 h_ps://

52 DRYAD: Submission process h_ps://

53 Panton Principles (h_p://pantonprinciples.org/) Science is based on building on, reusing and openly cri6cising the published body of scien6fic knowledge. For science to effec6vely func6on, and for society to reap the full benefits from scien6fic endeavours, it is crucial that science data be made open. By open data in science we mean that it is freely available on the public internet permivng any user to download, copy, analyse, re- process, pass them to soxware or use them for any other purpose without financial, legal, or technical barriers other than those inseparable from gaining access to the internet itself. To this end data related to published science should be explicitly placed in the public domain.

54 h_p://pantonprinciples.org/ Formally, we recommend adop6ng and ac6ng on the following principles: 1. Where data or collec6ons of data are published it is cri6cal that they be published with a clear and explicit statement of the wishes and expecta6ons of the publishers with respect to re- use and re- purposing of individual data elements, the whole data collec6on, and subsets of the collec6on. This statement should be precise, irrevocable, and based on an appropriate and recognized legal statement in the form of a waiver or license. When publishing data make an explicit and robust statement of your wishes. 2. Many widely recognized licenses are not intended for, and are not appropriate for, data or collec6ons of data. A variety of waivers and licenses that are designed for and appropriate for the treatment of data are described here. Crea6ve Commons licenses (apart from CCZero), GFDL, GPL, BSD, etc are NOT appropriate for data and their use is STRONGLY discouraged. Use a recognized waiver or license that is appropriate for data. 3. The use of licenses which limit commercial re- use or limit the produc6on of deriva6ve works by excluding use for par6cular purposes or by specific persons or organiza6ons is STRONGLY discouraged. These licenses make it impossible to effec6vely integrate and re- purpose datasets and prevent commercial ac6vi6es that could be used to support data preserva6on. If you want your data to be effec6vely used and added to by others it should be open as defined by the Open Knowledge/Data Defini6on in par6cular non- commercial and other restric6ve clauses should not be used. 4. Furthermore, in science it is STRONGLY recommended that data, especially where publicly funded, be explicitly placed in the public domain via the use of the Public Domain Dedica6on and Licence or Crea6ve Commons Zero Waiver. This is in keeping with the public funding of much scien6fic research and the general ethos of sharing and re- use within the scien6fic community. Explicit dedica6on of data underlying published science into the public domain via PDDL or CCZero is strongly recommended and ensures compliance with both the Science Commons Protocol for Implemen6ng Open Access Data and the Open Knowledge/Data Defini6on.

55 Comments during the workshop A number of par6cipants raised the point of data ownership, which is jointly owned by the scien6sts (grad students, post- docs and PI), but does not belong to any one individual. The other issue was on whether the university owns the research, and whether there is copyright issues in submivng data. However, it does not seem that this is a concern. I will look for more informa6on on these issues, so people can come speak to me in the future.