Data Management: Best Practices. Michelle Craft Research IT Coordinator

Size: px

Start display at page:

Download "Data Management: Best Practices. Michelle Craft Research IT Coordinator mcraft@discovery.wisc.edu http://cct-resources.discovery.wisc."

Virginia Holt
8 years ago
Views:

1 Data Management: Best Practices Michelle Craft Research IT Coordinator

2 Stanford University Libraries, Data Management Services library.stanford.edu/research/data-managementservices/data-best-practices assists Stanford researchers with the organization, management, and curation of research data to enhance its preservation and access now and into the future

3 Carpentry Workshops Software Carpentry software-carpentry.org/ non-profit organization whose members teach researchers basic software skills (command line, python, version control, etc.) run workshops all over the world, provide open access teaching materials, and run an instructor training program Data Carpentry datacarpentry.org/ new offshoot/partner of Software Carpentry designed to teach basic concepts, skills and tools for working more effectively with data

4 Best Practices Overview File Management Descriptive Names Metadata Sensitive Data Data Versions File Content Rows vs Columns Spreadsheet Mistakes File Formats Working with Data Find useful tools Quality control, data manipulation Software and Analysis Versions

5 Examples Les Optimal Bess Practice

6 Use descriptive and informative Special Characters Spaces # $ % ^ & * ( ) ` ; < >?, [ ] { } ' LONG filenames Sequential numbers without zeros 1,10,11, 2 01,02, 10,11 file names Essential information Project or experiment or acronym Researcher name/initials Type of data Version number Sortable dates (YYYYMMDD) readme.txt to explain the naming scheme Happy is not happy. Think like a computer. Stanford University Libraries, Data Management Services Data Best Practices

7 Create metadata for every experiment or analysis you run Stanford University Libraries, Data Management Services Data Best Practices

8 Handle sensitive data in an appropriate manner Secure Storage File system Backup Transmission (sometimes not or usb drives) Modify sensitive data for public release Software for sensitive information REDCap, Qualtrics Follow your protocol Stanford University Libraries, Data Management Services Data Best Practices

9 Track different versions of your documents Modify the name Version number (v1, v2) Status (draft, final) Change information (cropped, normalized) One person works on a file at a time Files are accessed from one location (e.g. Box) To Be Continued Stanford University Libraries, Data Management Services Data Best Practices

10 File Content You can have data without information, but you cannot have information without data. -Daniel Keys Moran

11 (Independent) Variables in Columns, Observations (Dependent Variables) in Rows Each cell is an observation that must have all the relevant information connected to stand on its own. There are more available rows than columns. Most programs process row by row. Think like a computer. Lessons for Excel

12 Common Spreadsheet Mistakes Not filling in zeros Using bad null values Using formatting to: convey information make the data sheet look pretty (e.g. putting in unnecessary blank rows) Placing comments or units in cells More than one piece of information in a cell Field name problems Data and date ambiguity Each cell is an observation! Lessons for Excel

13 Choose file formats that will ensure long-term access Proprietary, closed specifications WPS, MELLEL Encrypted Compressed In common usage by the research community XML, CSV If proprietary, use open specifications digitalpreservation.gov/ formats/ Stanford University Libraries, Data Management Services Data Best Practices

14 Working With Data Men have become the tools of their tools. -Henry David Thoreau You can t always get what you want, but if you try sometimes well you just might find, you get what you need. -Mick Jagger, Keith Richards

15 Find helpful tools for analyzing your data Metadata tools Annotare, ISA Creator, OMERO, OntoMaton, RightField Data analysis packages and libraries R (Bioconductor, dplyr, tidyr), Matlab, SciPy, SAS, SPSS, Stata, GAMS, Mathematica, Excel Data visualization Matplotlib, ggplot2 DIY Python, Java, Matlab, R Stanford University Libraries, Data Management Services Data Best Practices

16 Quality Control and Data Manipulation with Excel Sorting (Sort & Filter -> Filter) Conditional Formatting Check Cell Formats Pivot Tables Think like a computer. Lessons for Excel

17 Track different versions of your documents Track software, tools, and documents line by line Track everything needed to ensure reproducibility Cloud: Google Drive, Overleaf, Git/GitHub, Box Not Cloud: Word s Track Changes Stanford University Libraries, Data Management Services Data Best Practices

18 More?

Aspose.Cells Product Family

Aspose.Cells Product Family time and effort by using our efficient and robust components instead of developing your own. lets you open, create, save and convert files from within your application without Microsoft Excel, confident