Data sharing and Big Data in the physical sciences. 2 October 2015

Transcription

1 Data sharing and Big Data in the physical sciences 2 October 2015

2 Content Digital curation: Data and metadata Why consider the physical sciences? Astronomy: Video Physics: LHC for example. Video The Research Data Alliance (RDA) ITNPD4: Applications of Big Data 2

3 Digital curation (from JISC Digital Curation Centre) Digital curation involves maintaining, preserving and adding value to digital research data throughout its lifecycle. The active management of research data reduces threats to their long-term research value and mitigates the risk of digital obsolescence. Meanwhile, curated data in trusted digital repositories may be shared among the wider UK research community. As well as reducing duplication of effort in research data creation, curation enhances the long-term value of existing data by making it available for further high quality research. See ITNPD4: Applications of Big Data 3

4 Data and metadata Metadata is Data about data That is, it is descriptions of data. May be structural metadata Data about the format of some other (large set of) data Structure of files, structures of containers of data and/or descriptive metadata E.g. where and when was this record created May be subclassed into Guide metadata: helps people to find particular data Business metadata: further information about the dataset content The descriptive metadata required is very data type and task dependent ITNPD4: Applications of Big Data 4

5 Data without metadata is generally not usable at all. Structural metadata is required to enable syntactically correct reading of the data Descriptive metadata is required to enable semantically correct processing of data Metadata standards See and despair! Metadata usage Everywhere e.g. photographs, music, scientific datasets, Long-term usage of data requires explicit metadata. ITNPD4: Applications of Big Data 5

6 Why the physical sciences? Concerned with external reality Lots of scientists working on related problems Working on the same systems Interested in similar problems (Possibly) not concerned with secrecy Depending whether they are in a University or at a military establishment! Long history of sharing data and papers Which was the root of the WWW itself Very early adopters of big data and data sharing technologies And of data curation and metadata as well ITNPD4: Applications of Big Data 6

7 Astronomy Long history of sharing datasets. See slides from documents/webpage/pga_ pdf ITNPD4: Applications of Big Data 7

8 Physics Berners-Lee created the WWW (essentially a web application for the Internet) In order to allow physicists at CERN and elsewhere to share papers And later, datasets as well Physicists are used to dealing with large data volumes And helped to develop many of the computational techniques associated with Big Data. ITNPD4: Applications of Big Data 8

9 Big data and the Large Hadron Collider (LHC) The LHC produces millions of collisions every second in each detector, generating approximately one petabyte of data per second. None of today s computing systems are capable of recording such rates, so sophisticated selection systems are used for a first fast electronic pre-selection, only passing one out of 10,000 events. Tens of thousands of processor cores then select 1% of the remaining events for analysis. Even after such drastic data reduction, the four big experiments, ALICE, ATLAS, CMS and LHCb, together need to store over 25 petabytes per year. The LHC data are aggregated in the CERN Data Centre, which performs initial data reconstruction is performed, and a copy is archived to long-term tape storage. Another copy is sent to several large data centres around the world. ITNPD4: Applications of Big Data 9

10 LHC and big data About 20 to 25PBytes/year (after throwing away 99% of what s recorded) 150 datacenters worldwide Structured data: ROOT structured files Compressed binary files: efficient storage Self-documenting: helps with changing requirements Open-source: not proprietary at all Backwards-compatible: data will be re-analysed years hence Including metadata ITNPD4: Applications of Big Data 10

11 Research Data Alliance (RDA) Cross-disciplinary international organisation The Research Data Alliance (RDA) builds the social and technical bridges that enable open sharing of data. The RDA vision is researchers and innovators openly sharing data across technologies, disciplines, and countries to address the grand challenges of society. Breadth: see ITNPD4: Applications of Big Data 11