How To Build An Open Source Data Infrastructure

Size: px

Start display at page:

Download "How To Build An Open Source Data Infrastructure"

Suzan Wells
5 years ago
Views:

1 EUDAT Collaborative Data Infrastructure Towards the convergence of Compute, Data, Knowledge and Scientific Instruments Giuseppe Fiameni CINECA EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-infrastructures. Contract No

2 EUDAT is... a pan-european initiative building a sustainable cross-disciplinary and cross-national data infrastructure providing a set of shared services for accessing and preserving research data supporting multiple research communities by working closely with them to deliver technical services as part of the EUDAT Collaborative Data Infrastructure (CDI) consortium of high performance computing (HPC) / data centres, libraries, scientific communities, data scientists

providing a set of shared services for accessing and preserving research data supporting multiple research

3 Delivering an integrated suite of common data services covering the full research data lifecycle & addressing both long tail and big data EUDAT s contribution to Open Science

4 B2 SERVICE SUITE

5 Community-Driven BIOMEDICAL & MEDICAL SCIENCES MATERIALS & ANALYTICAL FACILITIES MAPPER PHYSICAL SCIENCES & ENGINEERING ICT Networking Session 6

6 Open Science & EUDAT OPEN all researchers, communities and infrastructures are encouraged to use / join the Collaborative Data Infrastructure GLOBAL pan European infrastructure with global collaborations US, Japan, COLLABORATIVE designed and driven by research communities and end users CREATIVE enabling scientists from any research discipline to find, access and process data enabling them to carry out research effectively CLOSER TO SOCIETY- services and technology offers are available to all

driven by research communities and end users CREATIVE enabling scientists from any research discipline to find, access and

E-Infrastructure Collaboration DFT, data fabric, PID, metadata, practical policy Four interoperability pilots fostering the coupling of data Policy and & guidelines cloud resources.

7 E-Infrastructure Collaboration DFT, data fabric, PID, metadata, practical policy Four interoperability pilots fostering the coupling of data Policy and & guidelines cloud resources. Large Data communities management involved, plans BBMRI, Service integration Open ICOS, EISCAT-3D, ELIXIR. AIRE DMP LERU LIBER RDA EGI Policy & networking Output adoption Test beds Cloud Catalogue Contribution to regular PRACE calls by providing medium/long-term storage capacity GEANT and services. 10 potential pilots emerging from the last DECI Call (13 th ) Cross-infra services & ops Common protocols, APIs HPC/HTC/Clouds DECI calls PRACE Helix Nebula Data Cloud ICT Networking Session 9

AIRE DMP LERU LIBER RDA EGI Policy & networking Output adoption Test beds Cloud Catalogue Contribution to regular PRACE calls by providing medium/long-term storage

For more info: https://b2drop.eudat.eu https://eudat.

eu/services/userdoc/b2share https://eudat.

8 For more info:

9 Services description EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-infrastructures. Contract No

10 Sync and Exchange Research Data B2DROP EUDAT s Personal Cloud Storage Service B2DROP is a secure and trusted data exchange service for researchers and scientists to keep their research data synchronized and up-to-date and to exchange with others. b2drop.eudat.eu

for researchers and scientists to keep their research data

11 An ideal solution for researchers and scientists to: Store and exchange data with colleagues and team members, including research data not finalized for publishing share data with fine-grained access controls synchronize multiple versions of data across different devices ICT Networking Session Features: 20GB storage per user Living objects, so no PIDs Versioning and offline use Desktop synchronisation 13

controls synchronize multiple versions of data across different devices ICT 2015 - Networking Session

12 Store and Share Research Data B2SHARE B2SHARE is a user-friendly, reliable and trustworthy way for researchers, scientific communities and scientists to store and share small-scale research data from diverse contexts. b2share.eudat.eu

scientific communities and scientists to store and share

A winning solution for researchers, scientists and communities to: store data safely at a trusted and certified data centre preserve data to guarantee long-term

13 A winning solution for researchers, scientists and communities to: store data safely at a trusted and certified data centre preserve data to guarantee long-term persistence control access and share data with colleagues and the world ICT Networking Session Features: metadata management permanent PIDs Open Access support 15

persistence control access and share data with colleagues and the world ICT 2015 -

Replicate Research Data Safely B2SAFE B2SAFE is a robust, safe and highly available service which allows community and departmental repositories

14 Replicate Research Data Safely B2SAFE B2SAFE is a robust, safe and highly available service which allows community and departmental repositories to implement data management policies on research data across multiple administrative domains in a trustworthy manner. eudat.eu/b2safe

repositories to implement data management policies on research data

15 The ideal solution for communities with no facility for archival to: replicate research data into secure data stores archive and preserve research data in the long-term bring data close to powerful compute resources co-locate data with different communities benefit from economies of scale ICT Networking Session Features: large-scale storage robust and highly available permanent PIDs 17

powerful compute resources co-locate data with different communities benefit from economies of

16 Get Data to Computation B2STAGE B2STAGE is a reliable, efficient, light-weight and easy-to-use service to transfer research data sets between EUDAT storage resources and high-performance computing (HPC) workspaces eudat.eu/b2stage

research data sets between EUDAT storage resources and

Facilitating communities to: move large amounts of data between data stores and highperformance compute resources re-ingest computational results back into EUDAT deposit

17 Facilitating communities to: move large amounts of data between data stores and highperformance compute resources re-ingest computational results back into EUDAT deposit large data sets onto EUDAT resources for long-term preservation ICT Networking Session Features: high-speed transfer reliable and light-weight manages permanent PIDs 19

deposit large data sets onto EUDAT resources for long-term preservation ICT 2015 -

18 Find Research Data B2FIND B2FIND is a simple, user-friendly metadata catalogue of research data collections stored in EUDAT data centres and other repositories. b2find.eudat.eu

19 A metadata catalogue service to: seek data objects and collections using powerful metadata searches catalogue community data by means of selected metadata browse through multi-disciplinary data collections filtered by content, provenance and temporal keywords ICT Networking Session Features: simple to use standards-based comprehensive catalogue 21

multi-disciplinary data collections filtered by content, provenance and temporal

EUDAT - Open Data Services for Research

EUDAT - Open Data Services for Research Per Öster 05.03.2015 CSC at a Glance Founded in 1971 as a technical support unit for Univac 1108 Connected Finland to the Internet in 1988 Reorganized as a company,