BIG DATA: DATA EVERYWHERE

Transcription

1 Line Pouchard, PhD Purdue Libraries, Research Data 03/10/2015 BIG DATA INTEREST GROUP Issues in Big Data Cura/on

2 BIG DATA: DATA EVERYWHERE

3 DEFINITIONS OF DATA CURATION Data curation is a term used to indicate management activities required to maintain research data long-term such that it is available for reuse and preservation (Wikipedia) The active and ongoing management of data through its life cycle of interest and usefulness to scholarship, science, and education. Data curation activities enable data discovery and retrieval, maintain its quality, add value, and provide for reuse over time, and this new field includes authentication, archiving, management, preservation, retrieval, and representation (GSLIS)

4 BIG DATA LIFECYCLE Assure

5 ISSUES IN BIG DATA CURATION Storage Data preparation & clean up Quality Discoverability Selection for preservation Privacy and ethics Reproducibility

6 QUESTIONS INFORMING CURATION ACTIVITIES Plan Acquire Prepare Volume What is an es1mate of volume & growth rate? Variety Are the data sensi1ve? What provisions are made to accommodate sensi1ve data? What is the most suited storage (databases, NoSQL, cloud)? What are the data formats and steps needed to integrate them? How do we prepare datasets for analysis? (remove blanks, duplicates, splifng columns, adding/removing headers)? What transforma1ons are needed to aggregate data? Do we need to create a pipeline? Velocity Is bandwidth sufficient to accommodate input rates? Veracity What are the data sources? What allows us to trust them? Will datasets be aggregated into series? Will metadata apply to individual datasets or to series? Who collects the data? Do they have the tools and skills to ensure con1nuity? What type of naming format is needed to keep track of incoming and derived datasets? Are the wrangling steps sufficiently documented to foster trust in the analysis?

7 QUESTIONS INFORMING CURATION ACTIVITIES Analyse Preserve Discover Volume Are adequate compute power and analysis methods available? Should raw data be preserved? What storage space is needed in the long- term? What part of the data (derived, raw, somware code) will be made accessible to searches? Variety Are the various analy1cal methods compa1ble with the different datasets? Velocity At what 1me point does the analy1cal feedback need to inform decisions? Are there different legal considera1ons for each data source? Are there conflicts with privacy and confiden1ality? When does data become obsolete? What search methods best suit this data keyword- based, geo- spa1al searches, metadata- based, seman1c searches? What degree of search latency is tolerable? Veracity What kind of access to scripts, somware, and procedures is needed to ensure transparency and reproducibility? What are the trade- offs if only derived products and no raw data are preserved? Providing well- documented data in open access allows scru1ny. How is veracity supported with sensi1ve and private data?

8 COLLABORATIONS Collaborations on multi-disciplinary proposals and projects Levels of collaboration Developing customized Data Management Plans Organizing your data Describing your data Sharing your data Publishing your datasets Preserving your data Education on best practices

9 A BIG DATA PROJECT AT PURDUE Dr. Yung- Hsiang Lu, PI

10 CURATION ISSUES IN CAM2 PROJECT Data access and re-use - policies of video streams and CCTV - Sparse legal framework except UK - Few policies available Data ownership Data storage Data organization - naming scheme - metadata Protect metadata storage where the intellectual property lies Data information literacy skills for Big Data

11 CONCLUSION Big Data looks very different than small data in maintenance and storage Curation primarily focuses on different areas than small data Planning from the beginning is crucial: without planning, curation will fall short Collaborations are more important Facilitating access is where efforts need to focus, not storing the data.

12 THE END

13 A REPRESENTATION OF THE FOUR Vs hup:// veracity

14 SOLUTION WHAT DID WE GET? Approximately 2.25 PB of IBM GPFS Hardware provided by a pair of Data Direct Networks SFA12k arrays, one in each of MATH and FREH datacenters 160 Gb/sec to each datacenter 5x Dell R620 servers in each datacenter